2024-04-24T07:56:44.795 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T07:56:44.796 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211 --name rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi --verbose --owner scheduled_rishabh@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} -- /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/orig.config.yaml 2024-04-24T07:56:44.859 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T07:56:44.952 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211 branch: rishabh-main-apr17-a654945 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: null first_in_suite: false job_id: '7671211' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-main-apr17-a654945 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-main-apr17-a654945 sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7954 sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 sleep_before_teardown: 0 subset: 364/512 suite: fs suite_branch: rishabh-main-apr17-a654945 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH4UL0UGN77fOzq4DniUeKxxJ73gvr/+PwPkWcELJH4dBsnvsDVEuwhq201b3ENKd0KO+2H208H1050bayk7euM= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZzf8luyseeVcaZiGowMJccyPvqtJlHQb3q3ff/S05DEjC/HQHGz1uWFhCxDIrsszFBCNYdVxNPALAQjqXK4rg= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKl74kDCk+AtyDYOUxaHR1cEiDpsNpuAESOVvS/cWRalGBHLjMWI3kUIAgGfchrR4oozK3ox6O683eTo0RnqhY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_07:32:23 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T07:56:44.952 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa; will attempt to use it 2024-04-24T07:56:44.953 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks 2024-04-24T07:56:44.953 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T07:56:44.955 INFO:teuthology.task.internal:Checking packages... 2024-04-24T07:56:44.978 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1f81fb397ae98da7563d451a78c61574c8f4e6e0' 2024-04-24T07:56:44.978 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T07:56:44.979 INFO:teuthology.packaging:ref: None 2024-04-24T07:56:44.979 INFO:teuthology.packaging:tag: None 2024-04-24T07:56:44.979 INFO:teuthology.packaging:branch: rishabh-main-apr17-a654945 2024-04-24T07:56:44.979 INFO:teuthology.packaging:sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:56:44.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-main-apr17-a654945 2024-04-24T07:56:45.231 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3116.g1f81fb39 2024-04-24T07:56:45.233 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T07:56:45.240 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T07:56:45.240 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T07:56:45.532 INFO:teuthology.task.internal:Saving configuration 2024-04-24T07:56:45.553 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T07:56:45.559 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T07:56:45.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:18.284071', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH4UL0UGN77fOzq4DniUeKxxJ73gvr/+PwPkWcELJH4dBsnvsDVEuwhq201b3ENKd0KO+2H208H1050bayk7euM='} 2024-04-24T07:56:45.602 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:18.286677', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZzf8luyseeVcaZiGowMJccyPvqtJlHQb3q3ff/S05DEjC/HQHGz1uWFhCxDIrsszFBCNYdVxNPALAQjqXK4rg='} 2024-04-24T07:56:45.625 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:18.285492', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKl74kDCk+AtyDYOUxaHR1cEiDpsNpuAESOVvS/cWRalGBHLjMWI3kUIAgGfchrR4oozK3ox6O683eTo0RnqhY='} 2024-04-24T07:56:45.625 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T07:56:45.631 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-24T07:56:45.631 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-24T07:56:45.631 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-24T07:56:45.631 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T07:56:45.808 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6c6297f160>, signals=[15]) 2024-04-24T07:56:45.808 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T07:56:45.815 INFO:teuthology.task.internal:Opening connections... 2024-04-24T07:56:45.815 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T07:56:45.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:56:45.893 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-04-24T07:56:45.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:56:45.967 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-04-24T07:56:45.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:56:46.043 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T07:56:46.050 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-04-24T07:56:46.068 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-04-24T07:56:46.068 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-04-24T07:56:46.125 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-04-24T07:56:46.125 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-04-24T07:56:46.125 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-04-24T07:56:46.125 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:56:46.126 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:56:46.127 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:56:46.127 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:56:46.128 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-04-24T07:56:46.153 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-04-24T07:56:46.171 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-04-24T07:56:46.171 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:56:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:56:46.229 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:56:46.229 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:56:46.229 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-04-24T07:56:46.253 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-04-24T07:56:46.270 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-04-24T07:56:46.270 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:56:46.327 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-04-24T07:56:46.328 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:56:46.328 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:56:46.328 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:56:46.328 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:56:46.328 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:56:46.328 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:56:46.328 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-04-24T07:56:46.353 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T07:56:46.364 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T07:56:46.466 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T07:56:46.466 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:56:46.469 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:56:46.472 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:56:46.489 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T07:56:46.495 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T07:56:46.495 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:56:46.526 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:56:46.529 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:56:46.645 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T07:56:46.721 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T07:56:46.736 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-24T07:56:46.737 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-24T07:56:46.737 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-24T07:56:46.737 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T07:56:46.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:56:46.754 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-24T07:56:46.933 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:46.933 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-24T07:56:46.933 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-04-24T07:56:46.951 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-437.el9.x86_64 2024-04-24T07:56:46.951 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:46.951 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-24T07:56:46.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum upgrade -y linux-firmware 2024-04-24T07:56:50.240 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-04-24T07:57:04.022 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-04-24T07:57:09.198 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-04-24T07:57:11.082 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-24T07:57:13.841 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-04-24T07:57:18.866 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-24T07:57:20.834 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-04-24T07:57:20.835 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-04-24T07:57:20.835 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-04-24T07:57:20.930 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-24T07:57:20.930 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:57:21.164 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-24T07:57:21.164 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-24T07:57:21.167 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi146.front.sepia.ceph.com to complete... 2024-04-24T07:57:33.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24 07:57:33 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-24T07:57:33.545 INFO:teuthology.orchestra.run.smithi146.stderr:FINISHED --2024-04-24 07:57:33-- 2024-04-24T07:57:33.545 INFO:teuthology.orchestra.run.smithi146.stderr:Total wall clock time: 12s 2024-04-24T07:57:33.545 INFO:teuthology.orchestra.run.smithi146.stderr:Downloaded: 1 files, 341M in 12s (28.4 MB/s) 2024-04-24T07:57:33.548 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-24T07:57:33.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-24T07:57:35.695 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying... ######################################## 2024-04-24T07:57:35.844 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing... ######################################## 2024-04-24T07:57:35.956 INFO:teuthology.orchestra.run.smithi146.stdout:Updating / installing... 2024-04-24T07:57:43.879 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-24T07:58:22.687 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-24T07:58:22.687 DEBUG:teuthology.orchestra.run.smithi146:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-24T07:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-24T07:58:22.772 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-24T07:58:22.773 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-24T07:58:22.774 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-24T07:58:22.775 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-24T07:58:22.776 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-24T07:58:22.777 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-24T07:58:22.778 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-24T07:58:22.779 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-24T07:58:22.780 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-24T07:58:22.781 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-24T07:58:22.782 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-24T07:58:22.783 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-24T07:58:22.784 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-24T07:58:22.785 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-24T07:58:22.786 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-24T07:58:22.787 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-24T07:58:22.788 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-24T07:58:22.789 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-24T07:58:22.790 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-24T07:58:22.791 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-24T07:58:22.792 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-24T07:58:22.793 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-24T07:58:22.794 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-24T07:58:22.795 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-24T07:58:22.796 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-24T07:58:22.797 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-24T07:58:22.798 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-24T07:58:22.799 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-24T07:58:22.800 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-24T07:58:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-24T07:58:22.802 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-24T07:58:22.803 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-24T07:58:22.804 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-24T07:58:22.805 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-24T07:58:22.806 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-24T07:58:22.807 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-24T07:58:22.808 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-24T07:58:22.809 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-24T07:58:22.810 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-24T07:58:22.811 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-24T07:58:22.812 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-24T07:58:22.813 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-24T07:58:22.814 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-24T07:58:22.815 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-24T07:58:22.816 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-24T07:58:22.817 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-24T07:58:22.818 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-24T07:58:22.819 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-24T07:58:22.820 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-24T07:58:22.821 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-24T07:58:22.822 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-24T07:58:22.823 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-24T07:58:22.824 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-24T07:58:22.825 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-24T07:58:22.826 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-24T07:58:22.827 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-24T07:58:22.828 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-24T07:58:22.829 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-24T07:58:22.830 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-24T07:58:22.831 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-24T07:58:22.832 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-24T07:58:22.833 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-24T07:58:22.834 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-24T07:58:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-24T07:58:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-24T07:58:22.837 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-24T07:58:22.838 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-24T07:58:22.839 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-24T07:58:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-24T07:58:22.841 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-24T07:58:22.842 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-24T07:58:22.843 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-24T07:58:22.844 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-24T07:58:22.845 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-24T07:58:22.846 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-24T07:58:22.847 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-24T07:58:22.848 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-24T07:58:22.849 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-24T07:58:22.850 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-24T07:58:22.851 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-24T07:58:22.852 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-24T07:58:22.853 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-24T07:58:22.854 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-24T07:58:22.855 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-24T07:58:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-24T07:58:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-24T07:58:22.858 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-24T07:58:22.859 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-24T07:58:22.860 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-24T07:58:22.861 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-24T07:58:22.862 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-24T07:58:22.863 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-24T07:58:22.864 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-24T07:58:22.865 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-24T07:58:22.866 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-24T07:58:22.867 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-24T07:58:22.868 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-24T07:58:22.869 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-24T07:58:22.870 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-24T07:58:22.871 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-24T07:58:22.872 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-24T07:58:22.873 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-24T07:58:22.874 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-24T07:58:22.875 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-24T07:58:22.876 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-24T07:58:22.877 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-24T07:58:22.878 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-24T07:58:22.879 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-24T07:58:22.880 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-24T07:58:22.881 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-24T07:58:22.882 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-24T07:58:22.883 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-24T07:58:22.884 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-24T07:58:22.885 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-24T07:58:22.886 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-24T07:58:22.887 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-24T07:58:22.888 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-24T07:58:22.889 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-24T07:58:22.890 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-24T07:58:22.891 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-24T07:58:22.892 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-24T07:58:22.893 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-24T07:58:22.894 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-24T07:58:22.897 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-24T07:58:22.897 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-24T07:58:22.897 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-24T07:58:22.897 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-24T07:58:22.897 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-24T07:58:22.897 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-24T07:58:22.897 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-24T07:58:22.898 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-24T07:58:22.899 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-24T07:58:22.900 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-24T07:58:22.901 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-24T07:58:22.902 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-24T07:58:22.903 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-24T07:58:22.904 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-24T07:58:22.905 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-24T07:58:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-24T07:58:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-24T07:58:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-24T07:58:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-24T07:58:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-24T07:58:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-24T07:58:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-24T07:58:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:58:22.912 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:22.912 DEBUG:teuthology.orchestra.run.smithi146:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-24T07:58:22.934 INFO:teuthology.orchestra.run.smithi146.stdout:postinstall scriptlet (using /bin/sh): 2024-04-24T07:58:22.934 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:58:22.934 INFO:teuthology.orchestra.run.smithi146.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:58:22.934 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:for file in vmlinuz System.map config; do 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout: fi 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:done 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-24T07:58:22.935 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:58:22.936 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:22.936 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-04-24T07:58:22.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-04-24T07:58:23.007 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-04-24T07:58:23.007 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-04-24T07:58:23.007 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-04-24T07:58:23.008 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-04-24T07:58:23.009 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T07:58:23.009 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-24T07:58:23.009 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-24T07:58:23.009 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-24T07:58:23.009 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-04-24T07:58:23.009 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T07:58:23.009 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-04-24T07:58:23.010 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-24T07:58:23.010 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:23.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-24T07:58:23.511 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-04-24T07:58:24.865 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-24T07:58:24.893 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-04-24T07:58:24.895 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-04-24T07:58:24.910 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.PD6ab6zrvf 2024-04-24T07:58:24.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PD6ab6zrvf 2024-04-24T07:58:25.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.PD6ab6zrvf 2024-04-24T07:58:25.097 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.PD6ab6zrvf is 6KB 2024-04-24T07:58:25.146 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.PD6ab6zrvf 2024-04-24T07:58:25.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-04-24T07:58:25.230 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-24T07:58:25.230 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-24T07:58:25.230 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-24T07:58:25.230 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-24T07:58:25.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-24T07:58:25.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-04-24T07:58:55.390 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-24T07:58:55.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-24T07:58:55.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:59:30.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-04-24T07:59:39.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-24T07:59:39.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T08:00:39.083 DEBUG:teuthology.orchestra.remote:timed out 2024-04-24T08:00:51.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-24T08:00:51.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T08:00:51.453 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-04-24T08:00:52.102 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-24T08:00:52.102 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-24T08:00:52.103 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-04-24T08:00:52.153 INFO:teuthology.orchestra.run.smithi146.stdout:6.9.0-rc3-g806ec36b2112 2024-04-24T08:00:52.154 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-24T08:00:52.154 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T08:00:52.154 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-24T08:00:53.155 DEBUG:teuthology.parallel:result is None 2024-04-24T08:00:53.155 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T08:00:53.163 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T08:00:53.163 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T08:00:53.171 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T08:00:53.173 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T08:00:53.221 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T08:00:53.346 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T08:00:53.353 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T08:00:53.353 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T08:00:53.362 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T08:00:53.365 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T08:00:53.428 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T08:00:53.437 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T08:00:53.438 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T08:00:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:00:53.455 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T08:00:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:00:53.473 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T08:00:53.480 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T08:00:53.497 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T08:00:53.526 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T08:00:53.548 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T08:00:53.561 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T08:00:53.566 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T08:00:53.596 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T08:00:53.650 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T08:00:53.654 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T08:00:53.662 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T08:00:53.662 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T08:00:53.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T08:00:53.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T08:00:53.744 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T08:00:53.754 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T08:00:53.754 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T08:00:53.761 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T08:00:53.764 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T08:00:53.808 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T08:00:53.891 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T08:00:53.989 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T08:00:54.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T08:00:54.161 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:00:54.162 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T08:00:54.246 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T08:00:54.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T08:00:54.336 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T08:00:54.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T08:00:54.460 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:00:54.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T08:00:54.525 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T08:00:54.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T08:00:54.630 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T08:00:54.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T08:00:54.715 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:00:54.715 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T08:00:54.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-04-24T08:00:54.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-04-24T08:00:54.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-04-24T08:00:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:00:54.852 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:00:54.890 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:00:55.329 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T08:00:55.337 INFO:teuthology.task.internal:Starting timer... 2024-04-24T08:00:55.337 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T08:00:55.358 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T08:00:55.367 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-24T08:00:55.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-04-24T08:00:55.509 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-04-24T08:00:55.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-04-24T08:00:55.586 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-04-24T08:00:55.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-04-24T08:00:55.640 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-04-24T08:00:55.642 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T08:00:55.642 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-04-24T08:00:55.699 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-04-24T08:00:55.700 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-04-24T08:00:55.725 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-04-24T08:00:55.725 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-04-24T08:00:55.763 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-04-24T08:00:55.763 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi146.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-04-24T08:00:55.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:00:55.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:00:55.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:00:55.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:00:55.865 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:00:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:00:55.890 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T08:00:55.890 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-04-24T08:00:55.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-04-24T08:00:55.987 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-04-24T08:00:56.015 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T08:00:56.027 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T08:00:56.033 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T08:00:56.181 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T08:00:56.197 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T08:00:56.198 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi146.front.sepia.ceph.com,smithi153.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T08:06:00.059 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi146.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-04-24T08:06:00.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-24T08:06:00.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T08:06:00.169 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-04-24T08:06:00.321 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-04-24T08:06:00.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-24T08:06:00.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T08:06:00.395 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-24T08:06:00.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-24T08:06:00.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-24T08:06:00.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T08:06:00.506 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-24T08:06:00.550 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-24T08:06:00.550 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T08:06:00.561 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T08:06:00.561 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T08:06:00.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:06:00.569 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T08:06:00.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:06:00.571 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T08:06:00.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:06:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T08:06:00.612 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T08:06:00.618 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T08:06:00.632 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T08:06:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-04-24T08:06:00.645 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-04-24T08:06:00.655 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T08:06:00.656 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-04-24T08:06:00.658 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T08:06:00.666 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-04-24T08:06:00.671 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T08:06:00.679 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T08:06:00.692 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T08:06:00.702 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T08:06:00.728 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:06:00.731 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:06:00.731 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-24T08:06:00.731 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.732 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.732 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.732 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.745 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:06:00.748 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:06:00.749 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-24T08:06:00.749 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.749 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.749 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.749 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:00.771 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-04-24T08:06:00.804 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-04-24T08:06:00.843 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T08:06:00.884 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T08:06:01.030 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:06:01.039 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:06:01.039 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-04-24T08:06:01.040 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:01.040 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:01.040 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:01.040 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:06:01.041 INFO:teuthology.run_tasks:Running task install... 2024-04-24T08:06:01.050 DEBUG:teuthology.task.install:project ceph 2024-04-24T08:06:01.050 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'}} 2024-04-24T08:06:01.050 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'} 2024-04-24T08:06:01.050 INFO:teuthology.task.install:Using flavor: default 2024-04-24T08:06:01.056 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T08:06:01.056 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-24T08:06:01.056 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T08:06:01.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:06:01.059 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T08:06:01.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:06:01.060 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T08:06:01.061 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:06:01.291 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T08:06:01.291 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T08:06:01.297 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T08:06:01.297 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T08:06:01.309 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T08:06:01.309 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T08:06:01.416 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T08:06:01.416 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:06:01.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T08:06:01.480 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T08:06:01.480 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:06:01.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T08:06:01.506 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T08:06:01.506 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T08:06:01.507 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T08:06:01.507 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T08:06:01.572 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T08:06:01.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T08:06:01.647 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T08:06:01.707 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T08:06:01.707 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:06:01.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T08:06:01.710 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-04-24T08:06:01.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-04-24T08:06:01.737 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T08:06:01.737 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T08:06:01.747 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T08:06:01.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T08:06:01.825 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-04-24T08:06:01.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-04-24T08:06:01.883 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T08:06:01.937 INFO:teuthology.orchestra.run.smithi153.stdout:48 files removed 2024-04-24T08:06:01.949 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-04-24T08:06:01.950 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-04-24T08:06:01.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T08:06:02.143 INFO:teuthology.orchestra.run.smithi146.stdout:48 files removed 2024-04-24T08:06:02.176 INFO:teuthology.orchestra.run.smithi177.stdout:48 files removed 2024-04-24T08:06:02.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T08:06:02.204 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T08:06:02.541 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 264 kB/s | 83 kB 00:00 2024-04-24T08:06:02.876 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-04-24T08:06:02.958 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-24T08:06:03.083 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-04-24T08:06:03.276 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-04-24T08:06:03.293 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-24T08:06:03.676 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 23 kB/s | 12 kB 00:00 2024-04-24T08:06:03.878 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 3.5 kB/s | 1.9 kB 00:00 2024-04-24T08:06:03.884 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-24T08:06:04.151 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-24T08:06:05.183 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 8.5 MB/s | 8.0 MB 00:00 2024-04-24T08:06:05.661 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-04-24T08:06:07.163 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-24T08:06:09.870 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-24T08:06:11.788 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-04-24T08:06:12.910 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 916 kB/s | 8.0 MB 00:08 2024-04-24T08:06:14.566 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-04-24T08:06:14.861 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-24T08:06:19.320 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.2 MB 00:00 2024-04-24T08:06:19.554 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-24T08:06:19.634 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 736 kB/s | 6.2 MB 00:08 2024-04-24T08:06:21.279 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 16 kB 00:00 2024-04-24T08:06:21.577 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-24T08:06:21.664 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:06:21.664 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:06:21.675 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T08:06:21.676 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T08:06:21.677 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T08:06:21.678 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T08:06:21.679 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T08:06:21.681 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T08:06:21.682 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T08:06:21.742 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-24T08:06:21.757 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T08:06:21.758 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T08:06:21.759 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T08:06:21.760 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T08:06:21.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T08:06:21.762 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T08:06:21.763 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T08:06:21.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T08:06:21.765 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T08:06:21.766 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T08:06:21.767 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T08:06:21.768 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-24T08:06:21.769 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-04-24T08:06:21.770 INFO:teuthology.orchestra.run.smithi153.stdout:Install 148 Packages 2024-04-24T08:06:21.770 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 4 Packages 2024-04-24T08:06:21.770 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:06:21.770 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 222 M 2024-04-24T08:06:21.770 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-24T08:06:22.389 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-04-24T08:06:22.539 INFO:teuthology.orchestra.run.smithi153.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-24T08:06:22.898 INFO:teuthology.orchestra.run.smithi153.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 2.4 MB/s | 890 kB 00:00 2024-04-24T08:06:22.990 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-24T08:06:22.998 INFO:teuthology.orchestra.run.smithi153.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 1.4 MB/s | 145 kB 00:00 2024-04-24T08:06:23.073 INFO:teuthology.orchestra.run.smithi153.stdout:(4/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-04-24T08:06:23.198 INFO:teuthology.orchestra.run.smithi153.stdout:(5/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 12 MB/s | 1.5 MB 00:00 2024-04-24T08:06:23.557 INFO:teuthology.orchestra.run.smithi153.stdout:(6/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 13 MB/s | 4.7 MB 00:00 2024-04-24T08:06:23.674 INFO:teuthology.orchestra.run.smithi153.stdout:(7/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 3.5 MB/s | 2.4 MB 00:00 2024-04-24T08:06:24.901 INFO:teuthology.orchestra.run.smithi153.stdout:(8/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x86 13 MB/s | 17 MB 00:01 2024-04-24T08:06:24.993 INFO:teuthology.orchestra.run.smithi153.stdout:(9/152): ceph-selinux-19.0.0-3116.g1f81fb39.el9 276 kB/s | 25 kB 00:00 2024-04-24T08:06:25.368 INFO:teuthology.orchestra.run.smithi153.stdout:(10/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el 6.3 MB/s | 11 MB 00:01 2024-04-24T08:06:25.568 INFO:teuthology.orchestra.run.smithi153.stdout:(11/152): ceph-common-19.0.0-3116.g1f81fb39.el9 6.5 MB/s | 21 MB 00:03 2024-04-24T08:06:25.593 INFO:teuthology.orchestra.run.smithi153.stdout:(12/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 998 kB/s | 225 kB 00:00 2024-04-24T08:06:25.627 INFO:teuthology.orchestra.run.smithi153.stdout:(13/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 463 kB/s | 27 kB 00:00 2024-04-24T08:06:25.652 INFO:teuthology.orchestra.run.smithi153.stdout:(14/152): libcephfs-devel-19.0.0-3116.g1f81fb39 542 kB/s | 32 kB 00:00 2024-04-24T08:06:25.736 INFO:teuthology.orchestra.run.smithi153.stdout:(15/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 1.9 MB/s | 163 kB 00:00 2024-04-24T08:06:25.778 INFO:teuthology.orchestra.run.smithi153.stdout:(16/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 4.8 MB/s | 746 kB 00:00 2024-04-24T08:06:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:(17/152): librados-devel-19.0.0-3116.g1f81fb39. 1.4 MB/s | 127 kB 00:00 2024-04-24T08:06:25.879 INFO:teuthology.orchestra.run.smithi153.stdout:(18/152): libradosstriper1-19.0.0-3116.g1f81fb3 4.9 MB/s | 501 kB 00:00 2024-04-24T08:06:25.963 INFO:teuthology.orchestra.run.smithi153.stdout:(19/152): python3-ceph-argparse-19.0.0-3116.g1f 541 kB/s | 45 kB 00:00 2024-04-24T08:06:26.046 INFO:teuthology.orchestra.run.smithi153.stdout:(20/152): python3-ceph-common-19.0.0-3116.g1f81 1.6 MB/s | 135 kB 00:00 2024-04-24T08:06:26.130 INFO:teuthology.orchestra.run.smithi153.stdout:(21/152): python3-cephfs-19.0.0-3116.g1f81fb39. 1.9 MB/s | 161 kB 00:00 2024-04-24T08:06:26.247 INFO:teuthology.orchestra.run.smithi153.stdout:(22/152): python3-rados-19.0.0-3116.g1f81fb39.e 2.7 MB/s | 320 kB 00:00 2024-04-24T08:06:26.364 INFO:teuthology.orchestra.run.smithi153.stdout:(23/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 2.5 MB/s | 298 kB 00:00 2024-04-24T08:06:26.464 INFO:teuthology.orchestra.run.smithi153.stdout:(24/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 994 kB/s | 99 kB 00:00 2024-04-24T08:06:26.564 INFO:teuthology.orchestra.run.smithi153.stdout:(25/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 851 kB/s | 85 kB 00:00 2024-04-24T08:06:26.940 INFO:teuthology.orchestra.run.smithi153.stdout:(26/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 4.8 MB/s | 5.4 MB 00:01 2024-04-24T08:06:27.031 INFO:teuthology.orchestra.run.smithi153.stdout:(27/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 6.6 MB/s | 3.1 MB 00:00 2024-04-24T08:06:27.057 INFO:teuthology.orchestra.run.smithi153.stdout:(28/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 1.4 MB/s | 170 kB 00:00 2024-04-24T08:06:27.124 INFO:teuthology.orchestra.run.smithi153.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3116.g 372 kB/s | 34 kB 00:00 2024-04-24T08:06:27.149 INFO:teuthology.orchestra.run.smithi153.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 1.5 MB/s | 142 kB 00:00 2024-04-24T08:06:27.539 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-24T08:06:27.641 INFO:teuthology.orchestra.run.smithi153.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 6.8 MB/s | 3.5 MB 00:00 2024-04-24T08:06:27.733 INFO:teuthology.orchestra.run.smithi153.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3116.g1f 2.7 MB/s | 249 kB 00:00 2024-04-24T08:06:28.117 INFO:teuthology.orchestra.run.smithi153.stdout:(33/152): ceph-mgr-diskprediction-local-19.0.0- 7.7 MB/s | 7.4 MB 00:00 2024-04-24T08:06:28.142 INFO:teuthology.orchestra.run.smithi153.stdout:(34/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 121 kB/s | 49 kB 00:00 2024-04-24T08:06:28.559 INFO:teuthology.orchestra.run.smithi153.stdout:(35/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 14 MB/s | 49 MB 00:03 2024-04-24T08:06:28.584 INFO:teuthology.orchestra.run.smithi153.stdout:(36/152): ceph-prometheus-alerts-19.0.0-3116.g1 35 kB/s | 16 kB 00:00 2024-04-24T08:06:28.609 INFO:teuthology.orchestra.run.smithi153.stdout:(37/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 584 kB/s | 273 kB 00:00 2024-04-24T08:06:28.643 INFO:teuthology.orchestra.run.smithi153.stdout:(38/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 311 kB/s | 18 kB 00:00 2024-04-24T08:06:28.677 INFO:teuthology.orchestra.run.smithi153.stdout:(39/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-24T08:06:28.727 INFO:teuthology.orchestra.run.smithi153.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 687 kB/s | 80 kB 00:00 2024-04-24T08:06:28.810 INFO:teuthology.orchestra.run.smithi153.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-04-24T08:06:28.836 INFO:teuthology.orchestra.run.smithi153.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-04-24T08:06:28.869 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-24T08:06:28.878 INFO:teuthology.orchestra.run.smithi153.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-04-24T08:06:28.903 INFO:teuthology.orchestra.run.smithi153.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-24T08:06:28.928 INFO:teuthology.orchestra.run.smithi153.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-24T08:06:28.962 INFO:teuthology.orchestra.run.smithi153.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-24T08:06:28.987 INFO:teuthology.orchestra.run.smithi153.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-24T08:06:29.029 INFO:teuthology.orchestra.run.smithi153.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-04-24T08:06:29.714 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:06:29.715 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:06:29.726 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T08:06:29.726 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T08:06:29.727 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T08:06:29.728 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T08:06:29.730 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T08:06:29.732 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T08:06:29.733 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T08:06:29.779 INFO:teuthology.orchestra.run.smithi153.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 142 kB/s | 107 kB 00:00 2024-04-24T08:06:29.793 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:06:29.815 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T08:06:29.816 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T08:06:29.817 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T08:06:29.818 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T08:06:29.819 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T08:06:29.820 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T08:06:29.821 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T08:06:29.822 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T08:06:29.823 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T08:06:29.824 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T08:06:29.825 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout:Install 148 Packages 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 4 Packages 2024-04-24T08:06:29.826 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:06:29.827 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 222 M 2024-04-24T08:06:29.827 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-04-24T08:06:30.355 INFO:teuthology.orchestra.run.smithi153.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 37 kB/s | 21 kB 00:00 2024-04-24T08:06:30.463 INFO:teuthology.orchestra.run.smithi153.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 459 kB/s | 689 kB 00:01 2024-04-24T08:06:30.763 INFO:teuthology.orchestra.run.smithi153.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 531 kB/s | 943 kB 00:01 2024-04-24T08:06:31.064 INFO:teuthology.orchestra.run.smithi153.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-04-24T08:06:31.230 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:06:31.231 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:06:31.251 INFO:teuthology.orchestra.run.smithi146.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T08:06:31.252 INFO:teuthology.orchestra.run.smithi146.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T08:06:31.253 INFO:teuthology.orchestra.run.smithi146.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T08:06:31.255 INFO:teuthology.orchestra.run.smithi146.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T08:06:31.257 INFO:teuthology.orchestra.run.smithi146.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T08:06:31.260 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T08:06:31.261 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T08:06:31.307 INFO:teuthology.orchestra.run.smithi177.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-24T08:06:31.331 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-04-24T08:06:31.339 INFO:teuthology.orchestra.run.smithi153.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 319 kB/s | 313 kB 00:00 2024-04-24T08:06:31.348 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================= 2024-04-24T08:06:31.348 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-04-24T08:06:31.348 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================= 2024-04-24T08:06:31.348 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-04-24T08:06:31.348 INFO:teuthology.orchestra.run.smithi146.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T08:06:31.348 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T08:06:31.348 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T08:06:31.349 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T08:06:31.350 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-04-24T08:06:31.351 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T08:06:31.352 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:06:31.353 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T08:06:31.354 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T08:06:31.355 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T08:06:31.356 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T08:06:31.357 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T08:06:31.358 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-04-24T08:06:31.359 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================= 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout:Install 148 Packages 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 4 Packages 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 222 M 2024-04-24T08:06:31.360 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-04-24T08:06:31.581 INFO:teuthology.orchestra.run.smithi153.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 18 kB 00:00 2024-04-24T08:06:31.665 INFO:teuthology.orchestra.run.smithi177.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 2.4 MB/s | 890 kB 00:00 2024-04-24T08:06:31.724 INFO:teuthology.orchestra.run.smithi177.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 2.4 MB/s | 145 kB 00:00 2024-04-24T08:06:31.873 INFO:teuthology.orchestra.run.smithi153.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-04-24T08:06:32.141 INFO:teuthology.orchestra.run.smithi177.stdout:(4/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 5.7 MB/s | 2.4 MB 00:00 2024-04-24T08:06:32.332 INFO:teuthology.orchestra.run.smithi177.stdout:(5/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-04-24T08:06:32.406 INFO:teuthology.orchestra.run.smithi153.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 201 kB/s | 160 kB 00:00 2024-04-24T08:06:32.490 INFO:teuthology.orchestra.run.smithi153.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-04-24T08:06:32.529 INFO:teuthology.orchestra.run.smithi146.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-04-24T08:06:32.883 INFO:teuthology.orchestra.run.smithi177.stdout:(6/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 8.5 MB/s | 4.7 MB 00:00 2024-04-24T08:06:33.171 INFO:teuthology.orchestra.run.smithi146.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 1.4 MB/s | 890 kB 00:00 2024-04-24T08:06:33.208 INFO:teuthology.orchestra.run.smithi177.stdout:(7/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-04-24T08:06:33.513 INFO:teuthology.orchestra.run.smithi146.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 425 kB/s | 145 kB 00:00 2024-04-24T08:06:33.792 INFO:teuthology.orchestra.run.smithi177.stdout:(8/152): ceph-common-19.0.0-3116.g1f81fb39.el9. 7.6 MB/s | 21 MB 00:02 2024-04-24T08:06:33.823 INFO:teuthology.orchestra.run.smithi153.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 468 kB/s | 662 kB 00:01 2024-04-24T08:06:33.850 INFO:teuthology.orchestra.run.smithi177.stdout:(9/152): ceph-selinux-19.0.0-3116.g1f81fb39.el9 433 kB/s | 25 kB 00:00 2024-04-24T08:06:33.865 INFO:teuthology.orchestra.run.smithi153.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 423 kB/s | 582 kB 00:01 2024-04-24T08:06:34.369 INFO:teuthology.orchestra.run.smithi146.stdout:(4/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-04-24T08:06:34.415 INFO:teuthology.orchestra.run.smithi153.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 43 kB/s | 25 kB 00:00 2024-04-24T08:06:34.482 INFO:teuthology.orchestra.run.smithi153.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 77 kB/s | 47 kB 00:00 2024-04-24T08:06:35.005 INFO:teuthology.orchestra.run.smithi146.stdout:(5/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 1.6 MB/s | 2.4 MB 00:01 2024-04-24T08:06:35.189 INFO:teuthology.orchestra.run.smithi146.stdout:(6/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-04-24T08:06:35.242 INFO:teuthology.orchestra.run.smithi177.stdout:(10/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x8 7.1 MB/s | 17 MB 00:02 2024-04-24T08:06:35.325 INFO:teuthology.orchestra.run.smithi177.stdout:(11/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 2.6 MB/s | 225 kB 00:00 2024-04-24T08:06:35.349 INFO:teuthology.orchestra.run.smithi153.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 217 kB/s | 188 kB 00:00 2024-04-24T08:06:35.400 INFO:teuthology.orchestra.run.smithi177.stdout:(12/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 361 kB/s | 27 kB 00:00 2024-04-24T08:06:35.407 INFO:teuthology.orchestra.run.smithi153.stdout:(64/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 659 kB/s | 3.0 MB 00:04 2024-04-24T08:06:35.433 INFO:teuthology.orchestra.run.smithi153.stdout:(65/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 287 kB/s | 292 kB 00:01 2024-04-24T08:06:35.476 INFO:teuthology.orchestra.run.smithi177.stdout:(13/152): libcephfs-devel-19.0.0-3116.g1f81fb39 423 kB/s | 32 kB 00:00 2024-04-24T08:06:35.634 INFO:teuthology.orchestra.run.smithi177.stdout:(14/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 4.6 MB/s | 746 kB 00:00 2024-04-24T08:06:35.735 INFO:teuthology.orchestra.run.smithi177.stdout:(15/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 1.6 MB/s | 163 kB 00:00 2024-04-24T08:06:35.818 INFO:teuthology.orchestra.run.smithi177.stdout:(16/152): librados-devel-19.0.0-3116.g1f81fb39. 1.5 MB/s | 127 kB 00:00 2024-04-24T08:06:35.969 INFO:teuthology.orchestra.run.smithi177.stdout:(17/152): libradosstriper1-19.0.0-3116.g1f81fb3 3.3 MB/s | 501 kB 00:00 2024-04-24T08:06:35.983 INFO:teuthology.orchestra.run.smithi153.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 56 kB/s | 35 kB 00:00 2024-04-24T08:06:36.869 INFO:teuthology.orchestra.run.smithi177.stdout:(18/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el 2.9 MB/s | 11 MB 00:03 2024-04-24T08:06:36.994 INFO:teuthology.orchestra.run.smithi177.stdout:(19/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 5.2 MB/s | 5.4 MB 00:01 2024-04-24T08:06:37.020 INFO:teuthology.orchestra.run.smithi177.stdout:(20/152): python3-ceph-argparse-19.0.0-3116.g1f 299 kB/s | 45 kB 00:00 2024-04-24T08:06:37.070 INFO:teuthology.orchestra.run.smithi177.stdout:(21/152): python3-ceph-common-19.0.0-3116.g1f81 1.7 MB/s | 135 kB 00:00 2024-04-24T08:06:37.121 INFO:teuthology.orchestra.run.smithi177.stdout:(22/152): python3-cephfs-19.0.0-3116.g1f81fb39. 1.6 MB/s | 161 kB 00:00 2024-04-24T08:06:37.163 INFO:teuthology.orchestra.run.smithi177.stdout:(23/152): python3-rados-19.0.0-3116.g1f81fb39.e 3.4 MB/s | 320 kB 00:00 2024-04-24T08:06:37.530 INFO:teuthology.orchestra.run.smithi146.stdout:(7/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-04-24T08:06:37.588 INFO:teuthology.orchestra.run.smithi177.stdout:(24/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 13 MB/s | 49 MB 00:03 2024-04-24T08:06:37.613 INFO:teuthology.orchestra.run.smithi177.stdout:(25/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 220 kB/s | 99 kB 00:00 2024-04-24T08:06:37.638 INFO:teuthology.orchestra.run.smithi177.stdout:(26/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-24T08:06:37.664 INFO:teuthology.orchestra.run.smithi177.stdout:(27/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 550 kB/s | 298 kB 00:00 2024-04-24T08:06:37.698 INFO:teuthology.orchestra.run.smithi177.stdout:(28/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 2.8 MB/s | 170 kB 00:00 2024-04-24T08:06:37.739 INFO:teuthology.orchestra.run.smithi177.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3116.g 454 kB/s | 34 kB 00:00 2024-04-24T08:06:37.765 INFO:teuthology.orchestra.run.smithi177.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 2.1 MB/s | 142 kB 00:00 2024-04-24T08:06:38.040 INFO:teuthology.orchestra.run.smithi177.stdout:(31/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 7.2 MB/s | 3.1 MB 00:00 2024-04-24T08:06:38.140 INFO:teuthology.orchestra.run.smithi177.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3116.g1f 2.4 MB/s | 249 kB 00:00 2024-04-24T08:06:38.224 INFO:teuthology.orchestra.run.smithi177.stdout:(33/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 594 kB/s | 49 kB 00:00 2024-04-24T08:06:38.368 INFO:teuthology.orchestra.run.smithi177.stdout:(34/152): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-04-24T08:06:38.391 INFO:teuthology.orchestra.run.smithi177.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3116.g1 98 kB/s | 16 kB 00:00 2024-04-24T08:06:38.417 INFO:teuthology.orchestra.run.smithi177.stdout:(36/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 5.4 MB/s | 273 kB 00:00 2024-04-24T08:06:38.458 INFO:teuthology.orchestra.run.smithi177.stdout:(37/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 438 kB/s | 18 kB 00:00 2024-04-24T08:06:38.492 INFO:teuthology.orchestra.run.smithi177.stdout:(38/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 7.4 MB/s | 762 kB 00:00 2024-04-24T08:06:38.589 INFO:teuthology.orchestra.run.smithi146.stdout:(8/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x86 4.9 MB/s | 17 MB 00:03 2024-04-24T08:06:38.639 INFO:teuthology.orchestra.run.smithi146.stdout:(9/152): ceph-selinux-19.0.0-3116.g1f81fb39.el9 515 kB/s | 25 kB 00:00 2024-04-24T08:06:38.642 INFO:teuthology.orchestra.run.smithi177.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 438 kB/s | 80 kB 00:00 2024-04-24T08:06:38.667 INFO:teuthology.orchestra.run.smithi153.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 606 kB/s | 1.6 MB 00:02 2024-04-24T08:06:38.734 INFO:teuthology.orchestra.run.smithi177.stdout:(40/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 3.6 MB/s | 3.5 MB 00:00 2024-04-24T08:06:38.759 INFO:teuthology.orchestra.run.smithi177.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-24T08:06:38.793 INFO:teuthology.orchestra.run.smithi177.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-24T08:06:38.851 INFO:teuthology.orchestra.run.smithi177.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-24T08:06:38.910 INFO:teuthology.orchestra.run.smithi177.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-24T08:06:38.943 INFO:teuthology.orchestra.run.smithi177.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 6.7 MB/s | 1.2 MB 00:00 2024-04-24T08:06:38.969 INFO:teuthology.orchestra.run.smithi177.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-24T08:06:39.003 INFO:teuthology.orchestra.run.smithi177.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-24T08:06:39.028 INFO:teuthology.orchestra.run.smithi177.stdout:(48/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 862 kB/s | 253 kB 00:00 2024-04-24T08:06:39.462 INFO:teuthology.orchestra.run.smithi177.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 247 kB/s | 107 kB 00:00 2024-04-24T08:06:39.612 INFO:teuthology.orchestra.run.smithi177.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 689 kB 00:00 2024-04-24T08:06:39.729 INFO:teuthology.orchestra.run.smithi177.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 1.3 MB/s | 943 kB 00:00 2024-04-24T08:06:39.754 INFO:teuthology.orchestra.run.smithi177.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 74 kB/s | 21 kB 00:00 2024-04-24T08:06:40.021 INFO:teuthology.orchestra.run.smithi177.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 112 kB/s | 33 kB 00:00 2024-04-24T08:06:40.056 INFO:teuthology.orchestra.run.smithi146.stdout:(10/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el 4.2 MB/s | 11 MB 00:02 2024-04-24T08:06:40.155 INFO:teuthology.orchestra.run.smithi177.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 578 kB/s | 313 kB 00:00 2024-04-24T08:06:40.305 INFO:teuthology.orchestra.run.smithi177.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 64 kB/s | 18 kB 00:00 2024-04-24T08:06:40.331 INFO:teuthology.orchestra.run.smithi146.stdout:(11/152): ceph-common-19.0.0-3116.g1f81fb39.el9 2.6 MB/s | 21 MB 00:08 2024-04-24T08:06:40.338 INFO:teuthology.orchestra.run.smithi177.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 61 kB/s | 11 kB 00:00 2024-04-24T08:06:40.365 INFO:teuthology.orchestra.run.smithi146.stdout:(12/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 731 kB/s | 225 kB 00:00 2024-04-24T08:06:40.431 INFO:teuthology.orchestra.run.smithi146.stdout:(13/152): libcephfs-devel-19.0.0-3116.g1f81fb39 482 kB/s | 32 kB 00:00 2024-04-24T08:06:40.475 INFO:teuthology.orchestra.run.smithi153.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 504 kB/s | 911 kB 00:01 2024-04-24T08:06:40.565 INFO:teuthology.orchestra.run.smithi146.stdout:(14/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 5.5 MB/s | 746 kB 00:00 2024-04-24T08:06:40.607 INFO:teuthology.orchestra.run.smithi146.stdout:(15/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 98 kB/s | 27 kB 00:00 2024-04-24T08:06:40.641 INFO:teuthology.orchestra.run.smithi146.stdout:(16/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 2.1 MB/s | 163 kB 00:00 2024-04-24T08:06:40.688 INFO:teuthology.orchestra.run.smithi177.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 129 kB/s | 45 kB 00:00 2024-04-24T08:06:40.691 INFO:teuthology.orchestra.run.smithi146.stdout:(17/152): librados-devel-19.0.0-3116.g1f81fb39. 1.5 MB/s | 127 kB 00:00 2024-04-24T08:06:40.714 INFO:teuthology.orchestra.run.smithi177.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 392 kB/s | 160 kB 00:00 2024-04-24T08:06:40.766 INFO:teuthology.orchestra.run.smithi146.stdout:(18/152): libradosstriper1-19.0.0-3116.g1f81fb3 3.9 MB/s | 501 kB 00:00 2024-04-24T08:06:40.816 INFO:teuthology.orchestra.run.smithi146.stdout:(19/152): python3-ceph-argparse-19.0.0-3116.g1f 923 kB/s | 45 kB 00:00 2024-04-24T08:06:40.884 INFO:teuthology.orchestra.run.smithi146.stdout:(20/152): python3-ceph-common-19.0.0-3116.g1f81 2.0 MB/s | 135 kB 00:00 2024-04-24T08:06:41.200 INFO:teuthology.orchestra.run.smithi146.stdout:(21/152): python3-cephfs-19.0.0-3116.g1f81fb39. 511 kB/s | 161 kB 00:00 2024-04-24T08:06:41.259 INFO:teuthology.orchestra.run.smithi153.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 170 kB/s | 133 kB 00:00 2024-04-24T08:06:41.289 INFO:teuthology.orchestra.run.smithi177.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 1.0 MB/s | 582 kB 00:00 2024-04-24T08:06:41.300 INFO:teuthology.orchestra.run.smithi146.stdout:(22/152): python3-rados-19.0.0-3116.g1f81fb39.e 3.2 MB/s | 320 kB 00:00 2024-04-24T08:06:41.348 INFO:teuthology.orchestra.run.smithi177.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 1.0 MB/s | 662 kB 00:00 2024-04-24T08:06:41.384 INFO:teuthology.orchestra.run.smithi146.stdout:(23/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 3.5 MB/s | 298 kB 00:00 2024-04-24T08:06:41.548 INFO:teuthology.orchestra.run.smithi177.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 98 kB/s | 25 kB 00:00 2024-04-24T08:06:41.568 INFO:teuthology.orchestra.run.smithi146.stdout:(24/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 6.1 MB/s | 5.4 MB 00:00 2024-04-24T08:06:41.575 INFO:teuthology.orchestra.run.smithi153.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 737 kB/s | 4.4 MB 00:06 2024-04-24T08:06:41.618 INFO:teuthology.orchestra.run.smithi146.stdout:(25/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-24T08:06:41.690 INFO:teuthology.orchestra.run.smithi177.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 139 kB/s | 47 kB 00:00 2024-04-24T08:06:41.716 INFO:teuthology.orchestra.run.smithi146.stdout:(26/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 300 kB/s | 99 kB 00:00 2024-04-24T08:06:41.794 INFO:teuthology.orchestra.run.smithi146.stdout:(27/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 2.2 MB/s | 170 kB 00:00 2024-04-24T08:06:41.844 INFO:teuthology.orchestra.run.smithi146.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3116.g 692 kB/s | 34 kB 00:00 2024-04-24T08:06:41.902 INFO:teuthology.orchestra.run.smithi146.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 2.4 MB/s | 142 kB 00:00 2024-04-24T08:06:42.023 INFO:teuthology.orchestra.run.smithi177.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 1.3 MB/s | 3.0 MB 00:02 2024-04-24T08:06:42.049 INFO:teuthology.orchestra.run.smithi177.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 584 kB/s | 292 kB 00:00 2024-04-24T08:06:42.107 INFO:teuthology.orchestra.run.smithi177.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 452 kB/s | 188 kB 00:00 2024-04-24T08:06:42.178 INFO:teuthology.orchestra.run.smithi146.stdout:(30/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 5.5 MB/s | 3.1 MB 00:00 2024-04-24T08:06:42.192 INFO:teuthology.orchestra.run.smithi153.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 715 kB/s | 4.7 MB 00:06 2024-04-24T08:06:42.307 INFO:teuthology.orchestra.run.smithi177.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 124 kB/s | 35 kB 00:00 2024-04-24T08:06:42.528 INFO:teuthology.orchestra.run.smithi146.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 5.7 MB/s | 3.5 MB 00:00 2024-04-24T08:06:42.628 INFO:teuthology.orchestra.run.smithi146.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3116.g1f 2.5 MB/s | 249 kB 00:00 2024-04-24T08:06:42.679 INFO:teuthology.orchestra.run.smithi146.stdout:(33/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 1.0 MB/s | 49 kB 00:00 2024-04-24T08:06:42.837 INFO:teuthology.orchestra.run.smithi146.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3116.g1 104 kB/s | 16 kB 00:00 2024-04-24T08:06:42.921 INFO:teuthology.orchestra.run.smithi146.stdout:(35/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 3.2 MB/s | 273 kB 00:00 2024-04-24T08:06:42.934 INFO:teuthology.orchestra.run.smithi153.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 118 kB/s | 87 kB 00:00 2024-04-24T08:06:43.071 INFO:teuthology.orchestra.run.smithi146.stdout:(36/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-24T08:06:43.130 INFO:teuthology.orchestra.run.smithi146.stdout:(37/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 317 kB/s | 18 kB 00:00 2024-04-24T08:06:43.314 INFO:teuthology.orchestra.run.smithi146.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 6.5 MB/s | 7.4 MB 00:01 2024-04-24T08:06:43.339 INFO:teuthology.orchestra.run.smithi146.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 387 kB/s | 80 kB 00:00 2024-04-24T08:06:43.406 INFO:teuthology.orchestra.run.smithi146.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-24T08:06:43.480 INFO:teuthology.orchestra.run.smithi146.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.4 MB/s | 253 kB 00:00 2024-04-24T08:06:43.526 INFO:teuthology.orchestra.run.smithi153.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 530 kB/s | 1.0 MB 00:01 2024-04-24T08:06:43.998 INFO:teuthology.orchestra.run.smithi146.stdout:(42/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 9.2 MB/s | 49 MB 00:05 2024-04-24T08:06:44.040 INFO:teuthology.orchestra.run.smithi146.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-24T08:06:44.090 INFO:teuthology.orchestra.run.smithi146.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 2.0 MB/s | 1.2 MB 00:00 2024-04-24T08:06:44.124 INFO:teuthology.orchestra.run.smithi146.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-24T08:06:44.149 INFO:teuthology.orchestra.run.smithi146.stdout:(46/152): python3-ply-3.11-14.el9.noarch.rpm 712 kB/s | 106 kB 00:00 2024-04-24T08:06:44.174 INFO:teuthology.orchestra.run.smithi146.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-24T08:06:44.208 INFO:teuthology.orchestra.run.smithi146.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-24T08:06:44.358 INFO:teuthology.orchestra.run.smithi146.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 716 kB/s | 107 kB 00:00 2024-04-24T08:06:44.426 INFO:teuthology.orchestra.run.smithi153.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 271 kB/s | 244 kB 00:00 2024-04-24T08:06:44.433 INFO:teuthology.orchestra.run.smithi146.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 290 kB/s | 21 kB 00:00 2024-04-24T08:06:44.467 INFO:teuthology.orchestra.run.smithi146.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-04-24T08:06:44.482 INFO:teuthology.orchestra.run.smithi177.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 1.9 MB/s | 4.7 MB 00:02 2024-04-24T08:06:44.509 INFO:teuthology.orchestra.run.smithi146.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 2.8 MB/s | 943 kB 00:00 2024-04-24T08:06:44.534 INFO:teuthology.orchestra.run.smithi146.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 494 kB/s | 33 kB 00:00 2024-04-24T08:06:44.567 INFO:teuthology.orchestra.run.smithi146.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-04-24T08:06:44.593 INFO:teuthology.orchestra.run.smithi146.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-24T08:06:44.618 INFO:teuthology.orchestra.run.smithi146.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-04-24T08:06:44.652 INFO:teuthology.orchestra.run.smithi146.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-24T08:06:44.677 INFO:teuthology.orchestra.run.smithi146.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 782 kB/s | 45 kB 00:00 2024-04-24T08:06:44.753 INFO:teuthology.orchestra.run.smithi146.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-24T08:06:44.786 INFO:teuthology.orchestra.run.smithi146.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 582 kB 00:00 2024-04-24T08:06:44.803 INFO:teuthology.orchestra.run.smithi146.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 514 kB/s | 25 kB 00:00 2024-04-24T08:06:44.824 INFO:teuthology.orchestra.run.smithi177.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 646 kB/s | 1.6 MB 00:02 2024-04-24T08:06:44.845 INFO:teuthology.orchestra.run.smithi146.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 822 kB/s | 47 kB 00:00 2024-04-24T08:06:44.920 INFO:teuthology.orchestra.run.smithi146.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.3 MB/s | 3.0 MB 00:00 2024-04-24T08:06:44.954 INFO:teuthology.orchestra.run.smithi146.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-24T08:06:44.987 INFO:teuthology.orchestra.run.smithi146.stdout:(65/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-04-24T08:06:45.004 INFO:teuthology.orchestra.run.smithi146.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 425 kB/s | 35 kB 00:00 2024-04-24T08:06:45.150 INFO:teuthology.orchestra.run.smithi177.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 1.3 MB/s | 911 kB 00:00 2024-04-24T08:06:45.213 INFO:teuthology.orchestra.run.smithi146.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 7.7 MB/s | 1.6 MB 00:00 2024-04-24T08:06:45.241 INFO:teuthology.orchestra.run.smithi177.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 320 kB/s | 133 kB 00:00 2024-04-24T08:06:45.318 INFO:teuthology.orchestra.run.smithi153.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 278 kB/s | 248 kB 00:00 2024-04-24T08:06:45.355 INFO:teuthology.orchestra.run.smithi146.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 6.3 MB/s | 911 kB 00:00 2024-04-24T08:06:45.413 INFO:teuthology.orchestra.run.smithi146.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-24T08:06:45.539 INFO:teuthology.orchestra.run.smithi146.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2024-04-24T08:06:45.639 INFO:teuthology.orchestra.run.smithi146.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 6.9 MB/s | 4.7 MB 00:00 2024-04-24T08:06:45.706 INFO:teuthology.orchestra.run.smithi146.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-24T08:06:45.748 INFO:teuthology.orchestra.run.smithi146.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-04-24T08:06:45.815 INFO:teuthology.orchestra.run.smithi146.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-24T08:06:45.882 INFO:teuthology.orchestra.run.smithi146.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-04-24T08:06:45.901 INFO:teuthology.orchestra.run.smithi153.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 44 kB/s | 26 kB 00:00 2024-04-24T08:06:45.917 INFO:teuthology.orchestra.run.smithi177.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.5 MB/s | 1.0 MB 00:00 2024-04-24T08:06:45.940 INFO:teuthology.orchestra.run.smithi146.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 445 kB/s | 26 kB 00:00 2024-04-24T08:06:45.999 INFO:teuthology.orchestra.run.smithi146.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 333 kB/s | 19 kB 00:00 2024-04-24T08:06:46.092 INFO:teuthology.orchestra.run.smithi177.stdout:(72/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:03 2024-04-24T08:06:46.166 INFO:teuthology.orchestra.run.smithi146.stdout:(78/152): postgresql-server-13.14-1.el9.x86_64. 7.7 MB/s | 5.8 MB 00:00 2024-04-24T08:06:46.199 INFO:teuthology.orchestra.run.smithi146.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 864 kB/s | 172 kB 00:00 2024-04-24T08:06:46.224 INFO:teuthology.orchestra.run.smithi146.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 604 kB/s | 35 kB 00:00 2024-04-24T08:06:46.259 INFO:teuthology.orchestra.run.smithi177.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 256 kB/s | 87 kB 00:00 2024-04-24T08:06:46.283 INFO:teuthology.orchestra.run.smithi146.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-24T08:06:46.350 INFO:teuthology.orchestra.run.smithi146.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-24T08:06:46.409 INFO:teuthology.orchestra.run.smithi146.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-24T08:06:46.443 INFO:teuthology.orchestra.run.smithi153.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-04-24T08:06:46.542 INFO:teuthology.orchestra.run.smithi146.stdout:(84/152): python3-babel-2.9.1-2.el9.noarch.rpm 7.1 MB/s | 6.0 MB 00:00 2024-04-24T08:06:46.592 INFO:teuthology.orchestra.run.smithi146.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 231 kB/s | 42 kB 00:00 2024-04-24T08:06:46.626 INFO:teuthology.orchestra.run.smithi146.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-04-24T08:06:46.660 INFO:teuthology.orchestra.run.smithi146.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-24T08:06:46.693 INFO:teuthology.orchestra.run.smithi146.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-24T08:06:46.719 INFO:teuthology.orchestra.run.smithi146.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 880 kB/s | 51 kB 00:00 2024-04-24T08:06:46.752 INFO:teuthology.orchestra.run.smithi146.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 927 kB/s | 54 kB 00:00 2024-04-24T08:06:46.834 INFO:teuthology.orchestra.run.smithi177.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 425 kB/s | 244 kB 00:00 2024-04-24T08:06:46.869 INFO:teuthology.orchestra.run.smithi146.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.6 MB/s | 5.1 MB 00:00 2024-04-24T08:06:46.894 INFO:teuthology.orchestra.run.smithi146.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 295 kB/s | 42 kB 00:00 2024-04-24T08:06:46.928 INFO:teuthology.orchestra.run.smithi146.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 767 kB/s | 44 kB 00:00 2024-04-24T08:06:46.962 INFO:teuthology.orchestra.run.smithi146.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.3 MB/s | 221 kB 00:00 2024-04-24T08:06:47.008 INFO:teuthology.orchestra.run.smithi146.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 456 kB/s | 36 kB 00:00 2024-04-24T08:06:47.037 INFO:teuthology.orchestra.run.smithi146.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 637 kB/s | 47 kB 00:00 2024-04-24T08:06:47.079 INFO:teuthology.orchestra.run.smithi146.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.3 MB/s | 305 kB 00:00 2024-04-24T08:06:47.104 INFO:teuthology.orchestra.run.smithi146.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 874 kB/s | 58 kB 00:00 2024-04-24T08:06:47.138 INFO:teuthology.orchestra.run.smithi146.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-24T08:06:47.163 INFO:teuthology.orchestra.run.smithi146.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-24T08:06:47.197 INFO:teuthology.orchestra.run.smithi146.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 780 kB/s | 45 kB 00:00 2024-04-24T08:06:47.243 INFO:teuthology.orchestra.run.smithi177.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 2.8 MB/s | 5.8 MB 00:02 2024-04-24T08:06:47.243 INFO:teuthology.orchestra.run.smithi153.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 215 kB/s | 172 kB 00:00 2024-04-24T08:06:47.264 INFO:teuthology.orchestra.run.smithi146.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 224 kB/s | 22 kB 00:00 2024-04-24T08:06:47.326 INFO:teuthology.orchestra.run.smithi177.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-04-24T08:06:47.351 INFO:teuthology.orchestra.run.smithi177.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 479 kB/s | 248 kB 00:00 2024-04-24T08:06:47.465 INFO:teuthology.orchestra.run.smithi146.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-04-24T08:06:47.510 INFO:teuthology.orchestra.run.smithi177.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 96 kB/s | 26 kB 00:00 2024-04-24T08:06:47.594 INFO:teuthology.orchestra.run.smithi177.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 72 kB/s | 19 kB 00:00 2024-04-24T08:06:47.786 INFO:teuthology.orchestra.run.smithi177.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 397 kB/s | 172 kB 00:00 2024-04-24T08:06:47.811 INFO:teuthology.orchestra.run.smithi177.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 116 kB/s | 35 kB 00:00 2024-04-24T08:06:47.860 INFO:teuthology.orchestra.run.smithi153.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 56 kB/s | 35 kB 00:00 2024-04-24T08:06:47.965 INFO:teuthology.orchestra.run.smithi146.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 783 kB/s | 548 kB 00:00 2024-04-24T08:06:48.065 INFO:teuthology.orchestra.run.smithi146.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 197 kB/s | 19 kB 00:00 2024-04-24T08:06:48.099 INFO:teuthology.orchestra.run.smithi146.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 488 kB/s | 309 kB 00:00 2024-04-24T08:06:48.216 INFO:teuthology.orchestra.run.smithi146.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-24T08:06:48.286 INFO:teuthology.orchestra.run.smithi177.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 501 kB/s | 250 kB 00:00 2024-04-24T08:06:48.395 INFO:teuthology.orchestra.run.smithi177.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 381 kB/s | 222 kB 00:00 2024-04-24T08:06:48.508 INFO:teuthology.orchestra.run.smithi146.stdout:(108/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.1 MB/s | 16 MB 00:01 2024-04-24T08:06:48.533 INFO:teuthology.orchestra.run.smithi146.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 154 kB/s | 49 kB 00:00 2024-04-24T08:06:48.628 INFO:teuthology.orchestra.run.smithi177.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 226 kB/s | 77 kB 00:00 2024-04-24T08:06:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:(110/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-04-24T08:06:48.675 INFO:teuthology.orchestra.run.smithi146.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 406 kB/s | 67 kB 00:00 2024-04-24T08:06:48.770 INFO:teuthology.orchestra.run.smithi177.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 113 kB/s | 42 kB 00:00 2024-04-24T08:06:48.792 INFO:teuthology.orchestra.run.smithi146.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-04-24T08:06:48.837 INFO:teuthology.orchestra.run.smithi177.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-04-24T08:06:48.968 INFO:teuthology.orchestra.run.smithi146.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 1.8 MB/s | 548 kB 00:00 2024-04-24T08:06:48.993 INFO:teuthology.orchestra.run.smithi146.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 218 kB/s | 43 kB 00:00 2024-04-24T08:06:49.085 INFO:teuthology.orchestra.run.smithi146.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 277 kB/s | 32 kB 00:00 2024-04-24T08:06:49.110 INFO:teuthology.orchestra.run.smithi146.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-04-24T08:06:49.135 INFO:teuthology.orchestra.run.smithi146.stdout:(117/152): luarocks-3.9.2-1.el9.noarch.rpm 251 kB/s | 151 kB 00:00 2024-04-24T08:06:49.178 INFO:teuthology.orchestra.run.smithi153.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 747 kB/s | 5.8 MB 00:07 2024-04-24T08:06:49.202 INFO:teuthology.orchestra.run.smithi146.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-24T08:06:49.252 INFO:teuthology.orchestra.run.smithi146.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-04-24T08:06:49.287 INFO:teuthology.orchestra.run.smithi177.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 307 kB/s | 159 kB 00:00 2024-04-24T08:06:49.311 INFO:teuthology.orchestra.run.smithi146.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 327 kB/s | 35 kB 00:00 2024-04-24T08:06:49.313 INFO:teuthology.orchestra.run.smithi177.stdout:(88/152): python3-protobuf-3.14.0-13.el9.noarch 394 kB/s | 269 kB 00:00 2024-04-24T08:06:49.371 INFO:teuthology.orchestra.run.smithi177.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 522 kB/s | 279 kB 00:00 2024-04-24T08:06:49.378 INFO:teuthology.orchestra.run.smithi146.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-04-24T08:06:49.412 INFO:teuthology.orchestra.run.smithi146.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.1 MB/s | 303 kB 00:00 2024-04-24T08:06:49.496 INFO:teuthology.orchestra.run.smithi146.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-24T08:06:49.521 INFO:teuthology.orchestra.run.smithi146.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-04-24T08:06:49.546 INFO:teuthology.orchestra.run.smithi146.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 865 kB/s | 144 kB 00:00 2024-04-24T08:06:49.621 INFO:teuthology.orchestra.run.smithi146.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 142 kB/s | 18 kB 00:00 2024-04-24T08:06:49.630 INFO:teuthology.orchestra.run.smithi177.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-04-24T08:06:49.646 INFO:teuthology.orchestra.run.smithi146.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-04-24T08:06:49.655 INFO:teuthology.orchestra.run.smithi177.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 157 kB/s | 54 kB 00:00 2024-04-24T08:06:49.671 INFO:teuthology.orchestra.run.smithi146.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 155 kB/s | 19 kB 00:00 2024-04-24T08:06:49.730 INFO:teuthology.orchestra.run.smithi146.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-04-24T08:06:49.780 INFO:teuthology.orchestra.run.smithi146.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2024-04-24T08:06:49.840 INFO:teuthology.orchestra.run.smithi146.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 722 kB/s | 79 kB 00:00 2024-04-24T08:06:49.881 INFO:teuthology.orchestra.run.smithi146.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 579 kB/s | 58 kB 00:00 2024-04-24T08:06:49.888 INFO:teuthology.orchestra.run.smithi177.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 161 kB/s | 42 kB 00:00 2024-04-24T08:06:49.939 INFO:teuthology.orchestra.run.smithi146.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 3.5 MB/s | 1.0 MB 00:00 2024-04-24T08:06:49.973 INFO:teuthology.orchestra.run.smithi146.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-24T08:06:49.997 INFO:teuthology.orchestra.run.smithi177.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 129 kB/s | 44 kB 00:00 2024-04-24T08:06:50.007 INFO:teuthology.orchestra.run.smithi146.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-04-24T08:06:50.057 INFO:teuthology.orchestra.run.smithi146.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 771 kB/s | 90 kB 00:00 2024-04-24T08:06:50.082 INFO:teuthology.orchestra.run.smithi146.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 261 kB/s | 28 kB 00:00 2024-04-24T08:06:50.116 INFO:teuthology.orchestra.run.smithi146.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-04-24T08:06:50.119 INFO:teuthology.orchestra.run.smithi153.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 266 kB/s | 250 kB 00:00 2024-04-24T08:06:50.174 INFO:teuthology.orchestra.run.smithi146.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-04-24T08:06:50.199 INFO:teuthology.orchestra.run.smithi146.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-04-24T08:06:50.225 INFO:teuthology.orchestra.run.smithi146.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 330 kB/s | 36 kB 00:00 2024-04-24T08:06:50.256 INFO:teuthology.orchestra.run.smithi177.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 139 kB/s | 36 kB 00:00 2024-04-24T08:06:50.284 INFO:teuthology.orchestra.run.smithi146.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 471 kB/s | 51 kB 00:00 2024-04-24T08:06:50.317 INFO:teuthology.orchestra.run.smithi146.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-24T08:06:50.342 INFO:teuthology.orchestra.run.smithi146.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 768 kB/s | 90 kB 00:00 2024-04-24T08:06:50.381 INFO:teuthology.orchestra.run.smithi177.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 448 kB/s | 221 kB 00:00 2024-04-24T08:06:50.426 INFO:teuthology.orchestra.run.smithi146.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-04-24T08:06:50.476 INFO:teuthology.orchestra.run.smithi146.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 126 kB/s | 20 kB 00:00 2024-04-24T08:06:50.510 INFO:teuthology.orchestra.run.smithi146.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-24T08:06:50.598 INFO:teuthology.orchestra.run.smithi177.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 139 kB/s | 47 kB 00:00 2024-04-24T08:06:50.669 INFO:teuthology.orchestra.run.smithi146.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 6.6 MB/s | 1.6 MB 00:00 2024-04-24T08:06:50.835 INFO:teuthology.orchestra.run.smithi146.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.2 MB/s | 1.3 MB 00:00 2024-04-24T08:06:50.940 INFO:teuthology.orchestra.run.smithi177.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 169 kB/s | 58 kB 00:00 2024-04-24T08:06:50.944 INFO:teuthology.orchestra.run.smithi146.stdout:(150/152): librados2-19.0.0-3116.g1f81fb39.el9. 7.3 MB/s | 3.4 MB 00:00 2024-04-24T08:06:50.961 INFO:teuthology.orchestra.run.smithi153.stdout:(82/152): python3-babel-2.9.1-2.el9.noarch.rpm 761 kB/s | 6.0 MB 00:08 2024-04-24T08:06:51.027 INFO:teuthology.orchestra.run.smithi146.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-04-24T08:06:51.236 INFO:teuthology.orchestra.run.smithi153.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 199 kB/s | 222 kB 00:01 2024-04-24T08:06:51.274 INFO:teuthology.orchestra.run.smithi177.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 342 kB/s | 305 kB 00:00 2024-04-24T08:06:51.299 INFO:teuthology.orchestra.run.smithi177.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 261 kB/s | 94 kB 00:00 2024-04-24T08:06:51.616 INFO:teuthology.orchestra.run.smithi177.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-04-24T08:06:51.633 INFO:teuthology.orchestra.run.smithi177.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 134 kB/s | 45 kB 00:00 2024-04-24T08:06:51.678 INFO:teuthology.orchestra.run.smithi146.stdout:(152/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 2.7 MB/s | 3.1 MB 00:01 2024-04-24T08:06:51.684 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-04-24T08:06:51.684 INFO:teuthology.orchestra.run.smithi146.stdout:Total 11 MB/s | 222 MB 00:20 2024-04-24T08:06:51.794 INFO:teuthology.orchestra.run.smithi153.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 93 kB/s | 77 kB 00:00 2024-04-24T08:06:51.903 INFO:teuthology.orchestra.run.smithi153.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 63 kB/s | 42 kB 00:00 2024-04-24T08:06:52.200 INFO:teuthology.orchestra.run.smithi177.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-04-24T08:06:52.475 INFO:teuthology.orchestra.run.smithi177.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.2 MB/s | 16 MB 00:03 2024-04-24T08:06:52.525 INFO:teuthology.orchestra.run.smithi177.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-24T08:06:52.559 INFO:teuthology.orchestra.run.smithi177.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 587 kB/s | 19 kB 00:00 2024-04-24T08:06:52.592 INFO:teuthology.orchestra.run.smithi177.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-04-24T08:06:52.634 INFO:teuthology.orchestra.run.smithi177.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 624 kB/s | 26 kB 00:00 2024-04-24T08:06:52.710 INFO:teuthology.orchestra.run.smithi177.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2024-04-24T08:06:52.735 INFO:teuthology.orchestra.run.smithi177.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-04-24T08:06:52.760 INFO:teuthology.orchestra.run.smithi177.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-24T08:06:52.761 INFO:teuthology.orchestra.run.smithi153.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 185 kB/s | 159 kB 00:00 2024-04-24T08:06:52.786 INFO:teuthology.orchestra.run.smithi177.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-24T08:06:52.803 INFO:teuthology.orchestra.run.smithi153.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 267 kB/s | 269 kB 00:01 2024-04-24T08:06:52.828 INFO:teuthology.orchestra.run.smithi177.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-24T08:06:52.853 INFO:teuthology.orchestra.run.smithi177.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-24T08:06:52.878 INFO:teuthology.orchestra.run.smithi177.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 864 kB/s | 43 kB 00:00 2024-04-24T08:06:52.903 INFO:teuthology.orchestra.run.smithi177.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-04-24T08:06:52.929 INFO:teuthology.orchestra.run.smithi177.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-24T08:06:52.954 INFO:teuthology.orchestra.run.smithi177.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-24T08:06:52.979 INFO:teuthology.orchestra.run.smithi177.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-24T08:06:53.005 INFO:teuthology.orchestra.run.smithi177.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-04-24T08:06:53.030 INFO:teuthology.orchestra.run.smithi177.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-04-24T08:06:53.055 INFO:teuthology.orchestra.run.smithi177.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-24T08:06:53.114 INFO:teuthology.orchestra.run.smithi177.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 24 MB/s | 2.0 MB 00:00 2024-04-24T08:06:53.139 INFO:teuthology.orchestra.run.smithi177.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-24T08:06:53.164 INFO:teuthology.orchestra.run.smithi177.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-24T08:06:53.189 INFO:teuthology.orchestra.run.smithi177.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-24T08:06:53.215 INFO:teuthology.orchestra.run.smithi177.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-24T08:06:53.240 INFO:teuthology.orchestra.run.smithi177.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-04-24T08:06:53.266 INFO:teuthology.orchestra.run.smithi177.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-04-24T08:06:53.308 INFO:teuthology.orchestra.run.smithi177.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-04-24T08:06:53.350 INFO:teuthology.orchestra.run.smithi177.stdout:(130/152): protobuf-compiler-3.14.0-13.el9.x86_ 503 kB/s | 863 kB 00:01 2024-04-24T08:06:53.375 INFO:teuthology.orchestra.run.smithi177.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 425 kB/s | 46 kB 00:00 2024-04-24T08:06:53.392 INFO:teuthology.orchestra.run.smithi177.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 956 kB/s | 79 kB 00:00 2024-04-24T08:06:53.434 INFO:teuthology.orchestra.run.smithi177.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 392 kB/s | 16 kB 00:00 2024-04-24T08:06:53.445 INFO:teuthology.orchestra.run.smithi153.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 79 kB/s | 51 kB 00:00 2024-04-24T08:06:53.459 INFO:teuthology.orchestra.run.smithi177.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-24T08:06:53.477 INFO:teuthology.orchestra.run.smithi177.stdout:(135/152): python3-natsort-7.1.1-5.el9.noarch.r 457 kB/s | 58 kB 00:00 2024-04-24T08:06:53.501 INFO:teuthology.orchestra.run.smithi177.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-24T08:06:53.518 INFO:teuthology.orchestra.run.smithi177.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2024-04-24T08:06:53.535 INFO:teuthology.orchestra.run.smithi177.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2024-04-24T08:06:53.560 INFO:teuthology.orchestra.run.smithi177.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-24T08:06:53.577 INFO:teuthology.orchestra.run.smithi177.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 860 kB/s | 36 kB 00:00 2024-04-24T08:06:53.611 INFO:teuthology.orchestra.run.smithi177.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-04-24T08:06:53.636 INFO:teuthology.orchestra.run.smithi177.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-04-24T08:06:53.662 INFO:teuthology.orchestra.run.smithi177.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-24T08:06:53.687 INFO:teuthology.orchestra.run.smithi177.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-24T08:06:53.712 INFO:teuthology.orchestra.run.smithi177.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 396 kB/s | 20 kB 00:00 2024-04-24T08:06:53.720 INFO:teuthology.orchestra.run.smithi153.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 291 kB/s | 279 kB 00:00 2024-04-24T08:06:53.729 INFO:teuthology.orchestra.run.smithi177.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-04-24T08:06:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-04-24T08:06:53.821 INFO:teuthology.orchestra.run.smithi177.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-24T08:06:54.063 INFO:teuthology.orchestra.run.smithi177.stdout:(149/152): librados2-19.0.0-3116.g1f81fb39.el9. 10 MB/s | 3.4 MB 00:00 2024-04-24T08:06:54.079 INFO:teuthology.orchestra.run.smithi153.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 85 kB/s | 54 kB 00:00 2024-04-24T08:06:54.097 INFO:teuthology.orchestra.run.smithi177.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 4.9 MB/s | 1.3 MB 00:00 2024-04-24T08:06:54.364 INFO:teuthology.orchestra.run.smithi177.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.0 MB/s | 2.4 MB 00:00 2024-04-24T08:06:54.444 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-04-24T08:06:54.447 INFO:teuthology.orchestra.run.smithi177.stdout:(152/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-04-24T08:06:54.450 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-04-24T08:06:54.450 INFO:teuthology.orchestra.run.smithi177.stdout:Total 9.0 MB/s | 222 MB 00:24 2024-04-24T08:06:54.690 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-04-24T08:06:54.690 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-04-24T08:06:54.696 INFO:teuthology.orchestra.run.smithi153.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 68 kB/s | 42 kB 00:00 2024-04-24T08:06:55.329 INFO:teuthology.orchestra.run.smithi153.stdout:(92/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 699 kB/s | 5.1 MB 00:07 2024-04-24T08:06:55.363 INFO:teuthology.orchestra.run.smithi153.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 66 kB/s | 44 kB 00:00 2024-04-24T08:06:55.523 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-04-24T08:06:55.667 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-04-24T08:06:55.667 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-04-24T08:06:55.972 INFO:teuthology.orchestra.run.smithi153.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 59 kB/s | 36 kB 00:00 2024-04-24T08:06:56.247 INFO:teuthology.orchestra.run.smithi153.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 241 kB/s | 221 kB 00:00 2024-04-24T08:06:56.597 INFO:teuthology.orchestra.run.smithi153.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 76 kB/s | 47 kB 00:00 2024-04-24T08:06:57.272 INFO:teuthology.orchestra.run.smithi153.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 86 kB/s | 58 kB 00:00 2024-04-24T08:06:57.298 INFO:teuthology.orchestra.run.smithi153.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 290 kB/s | 305 kB 00:01 2024-04-24T08:06:57.442 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-04-24T08:06:57.443 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-04-24T08:06:57.488 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-04-24T08:06:57.488 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-04-24T08:06:57.965 INFO:teuthology.orchestra.run.smithi153.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 96 kB/s | 64 kB 00:00 2024-04-24T08:06:57.990 INFO:teuthology.orchestra.run.smithi153.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 131 kB/s | 94 kB 00:00 2024-04-24T08:06:58.090 INFO:teuthology.orchestra.run.smithi153.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 224 kB/s | 22 kB 00:00 2024-04-24T08:06:58.240 INFO:teuthology.orchestra.run.smithi153.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-04-24T08:06:58.582 INFO:teuthology.orchestra.run.smithi153.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-04-24T08:06:58.766 INFO:teuthology.orchestra.run.smithi153.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-24T08:06:58.866 INFO:teuthology.orchestra.run.smithi153.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-04-24T08:06:59.033 INFO:teuthology.orchestra.run.smithi153.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 686 kB/s | 309 kB 00:00 2024-04-24T08:06:59.141 INFO:teuthology.orchestra.run.smithi153.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-24T08:06:59.259 INFO:teuthology.orchestra.run.smithi153.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-24T08:06:59.284 INFO:teuthology.orchestra.run.smithi153.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 342 kB/s | 49 kB 00:00 2024-04-24T08:06:59.351 INFO:teuthology.orchestra.run.smithi153.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-04-24T08:06:59.384 INFO:teuthology.orchestra.run.smithi153.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-24T08:06:59.460 INFO:teuthology.orchestra.run.smithi153.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-04-24T08:06:59.560 INFO:teuthology.orchestra.run.smithi153.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-04-24T08:06:59.585 INFO:teuthology.orchestra.run.smithi153.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-04-24T08:06:59.652 INFO:teuthology.orchestra.run.smithi153.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-04-24T08:06:59.677 INFO:teuthology.orchestra.run.smithi153.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-24T08:06:59.752 INFO:teuthology.orchestra.run.smithi153.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-24T08:06:59.778 INFO:teuthology.orchestra.run.smithi153.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-24T08:06:59.854 INFO:teuthology.orchestra.run.smithi153.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-24T08:06:59.879 INFO:teuthology.orchestra.run.smithi153.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-24T08:06:59.954 INFO:teuthology.orchestra.run.smithi153.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-24T08:07:00.055 INFO:teuthology.orchestra.run.smithi153.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-24T08:07:00.105 INFO:teuthology.orchestra.run.smithi153.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-04-24T08:07:00.138 INFO:teuthology.orchestra.run.smithi153.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-24T08:07:00.189 INFO:teuthology.orchestra.run.smithi153.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-24T08:07:00.222 INFO:teuthology.orchestra.run.smithi153.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-04-24T08:07:00.272 INFO:teuthology.orchestra.run.smithi153.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 233 kB/s | 19 kB 00:00 2024-04-24T08:07:00.306 INFO:teuthology.orchestra.run.smithi153.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-04-24T08:07:00.390 INFO:teuthology.orchestra.run.smithi153.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-24T08:07:00.415 INFO:teuthology.orchestra.run.smithi153.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-24T08:07:00.482 INFO:teuthology.orchestra.run.smithi153.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 856 kB/s | 79 kB 00:00 2024-04-24T08:07:00.507 INFO:teuthology.orchestra.run.smithi153.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-04-24T08:07:00.583 INFO:teuthology.orchestra.run.smithi153.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-24T08:07:00.608 INFO:teuthology.orchestra.run.smithi153.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-24T08:07:00.691 INFO:teuthology.orchestra.run.smithi153.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 830 kB/s | 90 kB 00:00 2024-04-24T08:07:00.717 INFO:teuthology.orchestra.run.smithi153.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-04-24T08:07:00.784 INFO:teuthology.orchestra.run.smithi153.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-04-24T08:07:00.817 INFO:teuthology.orchestra.run.smithi153.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-24T08:07:00.876 INFO:teuthology.orchestra.run.smithi153.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 644 kB/s | 59 kB 00:00 2024-04-24T08:07:00.909 INFO:teuthology.orchestra.run.smithi153.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-04-24T08:07:00.968 INFO:teuthology.orchestra.run.smithi153.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-24T08:07:01.029 INFO:teuthology.orchestra.run.smithi153.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-24T08:07:01.069 INFO:teuthology.orchestra.run.smithi153.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-04-24T08:07:01.127 INFO:teuthology.orchestra.run.smithi153.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.3 MB/s | 427 kB 00:00 2024-04-24T08:07:01.161 INFO:teuthology.orchestra.run.smithi153.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-04-24T08:07:01.228 INFO:teuthology.orchestra.run.smithi153.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-24T08:07:01.278 INFO:teuthology.orchestra.run.smithi153.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-24T08:07:01.720 INFO:teuthology.orchestra.run.smithi153.stdout:(148/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 7.1 MB/s | 3.1 MB 00:00 2024-04-24T08:07:01.770 INFO:teuthology.orchestra.run.smithi153.stdout:(149/152): librados2-19.0.0-3116.g1f81fb39.el9. 6.3 MB/s | 3.4 MB 00:00 2024-04-24T08:07:01.896 INFO:teuthology.orchestra.run.smithi153.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 1.3 MB 00:00 2024-04-24T08:07:01.979 INFO:teuthology.orchestra.run.smithi153.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-04-24T08:07:03.579 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-04-24T08:07:03.676 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T08:07:03.714 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T08:07:03.777 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T08:07:03.812 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T08:07:03.837 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T08:07:03.880 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T08:07:04.156 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-04-24T08:07:04.207 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T08:07:04.310 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T08:07:04.313 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:04.399 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T08:07:04.434 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:04.473 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:04.522 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T08:07:04.563 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:04.588 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T08:07:04.594 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:07:04.628 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T08:07:04.745 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T08:07:04.763 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T08:07:04.799 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T08:07:04.828 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:04.939 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:04.962 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:07:05.008 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:07:05.030 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:07:05.062 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:07:05.209 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T08:07:05.221 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:07:05.254 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:07:05.350 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:05.360 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T08:07:05.449 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T08:07:05.478 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T08:07:05.513 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T08:07:05.518 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:05.543 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:05.577 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T08:07:05.612 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T08:07:05.655 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T08:07:05.667 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:05.688 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T08:07:05.700 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:07:05.716 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T08:07:05.784 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T08:07:05.838 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T08:07:05.869 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T08:07:05.912 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T08:07:06.055 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T08:07:06.122 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T08:07:06.147 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:06.182 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T08:07:06.215 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T08:07:06.258 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T08:07:06.287 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T08:07:06.320 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:06.356 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:07:06.365 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T08:07:06.410 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T08:07:06.443 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:07:06.474 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:07:06.483 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T08:07:06.542 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:07:06.620 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T08:07:06.671 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T08:07:06.872 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:07:06.932 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:07:07.143 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T08:07:07.195 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T08:07:07.270 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T08:07:07.274 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T08:07:07.304 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T08:07:07.325 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T08:07:07.347 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T08:07:07.374 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T08:07:07.385 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T08:07:07.426 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T08:07:07.479 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T08:07:07.528 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T08:07:07.579 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T08:07:07.594 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T08:07:07.630 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T08:07:07.641 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T08:07:07.681 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T08:07:07.683 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T08:07:07.749 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T08:07:07.780 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T08:07:07.803 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T08:07:07.817 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T08:07:07.864 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T08:07:07.874 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T08:07:07.899 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T08:07:07.941 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T08:07:07.943 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T08:07:07.986 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T08:07:08.007 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T08:07:08.121 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T08:07:08.126 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T08:07:08.163 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T08:07:08.174 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T08:07:08.206 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T08:07:08.216 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T08:07:08.245 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T08:07:08.253 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T08:07:08.338 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T08:07:08.375 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T08:07:08.379 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T08:07:08.419 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T08:07:08.441 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T08:07:08.444 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T08:07:08.469 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T08:07:08.535 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T08:07:08.553 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T08:07:08.662 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T08:07:08.702 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T08:07:08.730 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T08:07:08.730 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T08:07:08.764 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T08:07:08.803 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T08:07:08.833 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T08:07:08.834 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T08:07:08.865 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T08:07:08.899 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T08:07:08.933 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T08:07:08.974 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T08:07:09.000 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T08:07:09.087 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T08:07:09.139 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T08:07:09.170 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T08:07:09.218 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T08:07:09.315 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T08:07:09.363 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T08:07:09.555 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T08:07:09.666 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T08:07:09.709 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T08:07:09.755 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T08:07:09.791 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T08:07:09.855 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T08:07:10.085 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T08:07:10.136 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T08:07:10.169 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T08:07:10.195 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T08:07:10.271 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T08:07:10.281 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T08:07:10.327 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T08:07:10.392 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T08:07:10.416 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T08:07:10.447 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T08:07:10.517 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T08:07:10.579 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T08:07:10.623 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T08:07:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T08:07:10.653 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T08:07:10.730 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T08:07:10.743 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T08:07:10.813 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T08:07:10.899 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T08:07:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T08:07:11.270 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T08:07:11.333 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T08:07:11.387 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T08:07:11.400 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T08:07:11.481 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T08:07:11.517 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T08:07:11.655 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T08:07:11.712 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T08:07:11.780 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T08:07:11.864 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T08:07:11.920 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T08:07:12.133 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T08:07:12.342 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T08:07:12.407 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T08:07:12.470 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T08:07:12.542 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T08:07:12.624 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T08:07:12.731 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T08:07:12.733 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T08:07:12.782 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T08:07:12.807 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T08:07:12.836 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T08:07:12.836 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T08:07:12.868 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T08:07:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T08:07:12.959 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T08:07:13.006 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T08:07:13.150 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T08:07:13.179 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T08:07:13.214 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T08:07:13.223 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:07:13.253 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T08:07:13.312 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:07:13.314 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T08:07:13.344 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T08:07:13.381 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T08:07:13.464 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T08:07:13.551 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T08:07:13.860 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T08:07:13.893 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:07:13.963 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:07:13.986 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T08:07:14.546 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T08:07:14.607 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T08:07:14.640 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T08:07:14.645 INFO:teuthology.orchestra.run.smithi153.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 795 kB/s | 16 MB 00:20 2024-04-24T08:07:14.649 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-24T08:07:14.649 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.2 MB/s | 222 MB 00:52 2024-04-24T08:07:14.767 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T08:07:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T08:07:14.941 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T08:07:14.992 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T08:07:15.228 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T08:07:15.279 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T08:07:15.408 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T08:07:15.700 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-24T08:07:15.845 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-24T08:07:15.845 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-24T08:07:16.192 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:16.232 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:16.242 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T08:07:16.396 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T08:07:16.769 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:16.811 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:07:16.894 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:07:16.895 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T08:07:16.895 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T08:07:16.895 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:16.929 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:07:17.615 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-24T08:07:17.615 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-24T08:07:18.006 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T08:07:18.103 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T08:07:18.150 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T08:07:18.197 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T08:07:18.589 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T08:07:18.874 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:07:19.005 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:07:19.031 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T08:07:19.098 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T08:07:19.641 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T08:07:19.668 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:07:19.793 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:07:19.820 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T08:07:22.557 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:22.599 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:23.252 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:23.303 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:07:23.502 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:07:23.503 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T08:07:23.503 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T08:07:23.503 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:23.545 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:07:23.643 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-24T08:07:23.731 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T08:07:23.770 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T08:07:23.824 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T08:07:23.857 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T08:07:23.883 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T08:07:23.926 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T08:07:24.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T08:07:24.367 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:24.489 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:24.518 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:24.609 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:24.647 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:07:24.798 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T08:07:24.852 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T08:07:24.888 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:07:25.085 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:07:25.115 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:07:25.137 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:07:25.341 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:07:25.373 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:07:25.489 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T08:07:25.586 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T08:07:25.622 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T08:07:25.682 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T08:07:25.748 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T08:07:25.781 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T08:07:25.824 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T08:07:25.865 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T08:07:25.910 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T08:07:25.980 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T08:07:26.032 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T08:07:26.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T08:07:26.106 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T08:07:26.183 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T08:07:26.217 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T08:07:26.260 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T08:07:26.289 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T08:07:26.359 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T08:07:26.438 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T08:07:26.513 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T08:07:26.656 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T08:07:26.706 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T08:07:27.239 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T08:07:27.319 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T08:07:27.347 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T08:07:27.390 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T08:07:27.428 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T08:07:27.470 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T08:07:27.629 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T08:07:27.689 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T08:07:27.744 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T08:07:27.817 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T08:07:27.848 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T08:07:27.884 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T08:07:27.932 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T08:07:27.967 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T08:07:28.019 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T08:07:28.062 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T08:07:28.197 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T08:07:28.249 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T08:07:28.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T08:07:28.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T08:07:28.416 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T08:07:28.454 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T08:07:28.494 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-04-24T08:07:28.499 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:28.519 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T08:07:28.544 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T08:07:28.619 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T08:07:28.668 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T08:07:28.683 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:28.756 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T08:07:28.802 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T08:07:28.829 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T08:07:28.864 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T08:07:28.903 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T08:07:28.934 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T08:07:28.964 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T08:07:28.999 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T08:07:29.032 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T08:07:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T08:07:29.108 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T08:07:29.196 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T08:07:29.247 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T08:07:29.278 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T08:07:29.325 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T08:07:29.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T08:07:29.480 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T08:07:30.138 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:07:30.163 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:07:30.163 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:30.163 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T08:07:30.163 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:07:30.163 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:07:30.163 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:30.216 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T08:07:30.268 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T08:07:30.302 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T08:07:30.380 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:07:30.401 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T08:07:30.402 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:07:30.402 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:30.402 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T08:07:30.402 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:07:30.402 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:07:30.402 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:30.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T08:07:30.564 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T08:07:30.604 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T08:07:30.782 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T08:07:30.818 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T08:07:30.847 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:07:30.869 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:07:30.869 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:30.869 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T08:07:30.869 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:07:30.870 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:07:30.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:30.896 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T08:07:30.969 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T08:07:31.018 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T08:07:31.073 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T08:07:31.260 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T08:07:31.315 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T08:07:31.384 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T08:07:31.472 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:07:31.551 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T08:07:31.612 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:07:31.648 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T08:07:31.667 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T08:07:31.688 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T08:07:31.716 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T08:07:32.151 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:07:32.181 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:07:32.213 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T08:07:32.301 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T08:07:32.330 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:07:32.920 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T08:07:32.996 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T08:07:33.024 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T08:07:33.057 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T08:07:33.362 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:07:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T08:07:33.399 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:07:33.420 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:07:33.502 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:07:33.533 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T08:07:33.577 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T08:07:34.055 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T08:07:34.081 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:07:34.152 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:07:34.174 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T08:07:34.357 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:07:34.390 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:07:34.492 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:07:34.653 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T08:07:34.684 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:07:34.706 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:07:34.707 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:34.707 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T08:07:34.707 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:07:34.707 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:07:34.707 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:34.751 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:07:34.762 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:07:34.860 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T08:07:34.908 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T08:07:35.831 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T08:07:35.870 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:07:35.892 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:07:35.892 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:35.893 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T08:07:35.893 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:07:35.893 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:07:35.893 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:35.938 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:07:35.959 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:07:35.959 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:35.960 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T08:07:35.960 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:36.005 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:07:36.027 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:07:36.027 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:36.027 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T08:07:36.027 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:07:36.027 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:07:36.027 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:36.339 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:07:36.361 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:07:36.362 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:36.362 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T08:07:36.362 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:07:36.362 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:07:36.362 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-04-24T08:07:36.418 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:36.508 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:36.539 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:36.682 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:07:36.730 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:07:36.730 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:36.730 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T08:07:36.730 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:37.098 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:07:37.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:07:37.228 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:07:37.229 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T08:07:37.229 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T08:07:37.229 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:37.277 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:07:38.377 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:07:38.452 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:07:38.452 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:38.452 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T08:07:38.452 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:07:38.453 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:07:38.453 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:38.728 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:07:38.823 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:07:38.823 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:38.823 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T08:07:38.823 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:07:38.823 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:07:38.823 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:39.353 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:07:39.449 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:07:39.449 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:39.449 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T08:07:39.449 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:07:39.449 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:07:39.450 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:39.587 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T08:07:39.647 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T08:07:39.723 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T08:07:39.978 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T08:07:40.040 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T08:07:40.127 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T08:07:40.262 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:07:40.429 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:07:40.500 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T08:07:40.541 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T08:07:40.588 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T08:07:40.951 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T08:07:40.992 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T08:07:41.033 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T08:07:41.065 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T08:07:41.093 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T08:07:41.241 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:07:41.282 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T08:07:41.283 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:07:41.322 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T08:07:41.345 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T08:07:41.352 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T08:07:41.456 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T08:07:41.510 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:07:41.943 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:07:41.983 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:07:42.002 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:07:42.841 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T08:07:42.848 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:07:42.895 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:07:42.930 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T08:07:42.971 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T08:07:43.053 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T08:07:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T08:07:43.084 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:07:43.176 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:07:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:07:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:07:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:07:43.366 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:44.003 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:07:44.028 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:07:44.175 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:07:44.353 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T08:07:44.389 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:07:44.476 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:07:44.476 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:44.476 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T08:07:44.476 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:07:44.476 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:07:44.476 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:44.540 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:07:44.578 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:07:44.713 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T08:07:44.787 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T08:07:45.796 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T08:07:45.834 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:07:45.932 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:07:45.933 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:45.933 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T08:07:45.933 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:07:45.933 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:07:45.933 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:45.985 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:07:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:07:46.081 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:46.081 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T08:07:46.081 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:46.160 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:07:46.229 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:07:46.229 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:46.229 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T08:07:46.230 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:07:46.230 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:07:46.230 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:46.578 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:07:46.652 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-04-24T08:07:48.654 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:48.836 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:07:48.851 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:07:48.851 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:48.851 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T08:07:48.851 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T08:07:49.106 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T08:07:49.107 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T08:07:49.108 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T08:07:49.122 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T08:07:49.124 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T08:07:49.125 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T08:07:49.126 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T08:07:49.127 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T08:07:49.128 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:07:49.129 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T08:07:50.338 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:07:50.364 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:07:50.364 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:50.364 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T08:07:50.364 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:07:50.364 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:07:50.364 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:50.606 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:07:50.628 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:07:50.628 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:50.628 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T08:07:50.629 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:07:50.629 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:07:50.629 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:51.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:07:51.096 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:07:51.096 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:51.096 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T08:07:51.096 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:07:51.096 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:07:51.096 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:51.178 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:51.178 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.179 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.180 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T08:07:51.181 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T08:07:51.182 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T08:07:51.183 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T08:07:51.184 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T08:07:51.185 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T08:07:51.186 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T08:07:51.187 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.188 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T08:07:51.189 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T08:07:51.190 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T08:07:51.191 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T08:07:51.191 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:07:51.191 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-24T08:07:51.204 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T08:07:51.261 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T08:07:51.317 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T08:07:51.530 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T08:07:51.575 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T08:07:51.579 DEBUG:teuthology.parallel:result is None 2024-04-24T08:07:51.653 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T08:07:51.686 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T08:07:51.741 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T08:07:51.742 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:07:51.826 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T08:07:51.882 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:07:51.897 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T08:07:51.932 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T08:07:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T08:07:51.964 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T08:07:51.992 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T08:07:52.276 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T08:07:52.332 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T08:07:52.360 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T08:07:52.436 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:07:52.465 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:07:52.498 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T08:07:52.568 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T08:07:52.597 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:07:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:07:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:07:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:07:53.637 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:07:53.673 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:07:54.640 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:07:54.666 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:07:54.769 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:07:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T08:07:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T08:07:54.936 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T08:07:54.959 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:07:54.982 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:07:54.982 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:54.982 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T08:07:54.983 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:07:54.983 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:07:54.983 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T08:07:55.027 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:07:55.038 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:07:55.143 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T08:07:55.165 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T08:07:55.208 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T08:07:55.211 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T08:07:55.211 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:07:55.363 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:07:55.402 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:07:55.403 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:07:55.542 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:07:55.674 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:55.970 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:55.970 INFO:teuthology.orchestra.run.smithi146.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-24T08:07:55.971 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:07:56.134 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T08:07:56.171 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:07:56.194 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:07:56.194 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:56.194 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T08:07:56.194 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:07:56.194 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:07:56.194 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:56.238 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:07:56.260 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:07:56.260 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:56.260 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T08:07:56.260 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:56.306 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:07:56.328 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:07:56.328 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:56.328 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T08:07:56.329 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:07:56.329 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:07:56.329 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:07:56.631 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:07:56.653 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:07:56.653 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:07:56.653 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T08:07:56.654 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:07:56.654 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:07:56.654 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:08:01.235 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T08:08:01.284 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T08:08:01.326 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T08:08:01.357 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T08:08:01.384 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T08:08:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T08:08:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T08:08:01.535 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T08:08:01.536 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T08:08:01.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T08:08:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T08:08:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T08:08:01.550 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T08:08:01.551 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T08:08:01.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T08:08:01.553 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T08:08:01.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T08:08:01.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T08:08:01.556 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T08:08:01.557 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T08:08:01.558 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T08:08:01.559 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T08:08:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T08:08:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T08:08:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:08:01.560 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T08:08:01.576 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T08:08:01.631 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T08:08:01.660 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T08:08:02.220 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:08:02.258 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:08:02.277 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:08:03.137 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T08:08:03.237 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T08:08:03.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T08:08:03.362 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T08:08:03.391 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T08:08:03.391 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:08:03.467 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:08:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:08:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:08:03.596 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:08:03.668 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:08:04.047 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:08:04.047 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:08:04.047 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-04-24T08:08:04.047 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.048 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.049 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T08:08:04.050 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.051 INFO:teuthology.orchestra.run.smithi146.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T08:08:04.052 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T08:08:04.053 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T08:08:04.054 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T08:08:04.055 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T08:08:04.056 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.057 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:08:04.058 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-04-24T08:08:04.640 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:08:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T08:08:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T08:08:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T08:08:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T08:08:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T08:08:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T08:08:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T08:08:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T08:08:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T08:08:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T08:08:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T08:08:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T08:08:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T08:08:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T08:08:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T08:08:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:08:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T08:08:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.787 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.788 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T08:08:11.789 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T08:08:11.790 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T08:08:11.791 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T08:08:11.792 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T08:08:11.793 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.794 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T08:08:11.795 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T08:08:11.796 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T08:08:11.797 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T08:08:11.797 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-24T08:08:11.797 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-24T08:08:12.227 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:12.228 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:08:12.448 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T08:08:12.514 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T08:08:12.515 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T08:08:12.515 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T08:08:12.517 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:08:12.656 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T08:08:12.678 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T08:08:12.678 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T08:08:12.679 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T08:08:12.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:08:12.902 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T08:08:12.931 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T08:08:12.931 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T08:08:12.931 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T08:08:12.933 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T08:08:12.933 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:08:12.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T08:08:13.023 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:08:13.023 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T08:08:13.052 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:08:13.052 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T08:08:13.083 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T08:08:13.083 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:08:13.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T08:08:13.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T08:08:13.226 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:08:13.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T08:08:13.250 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T08:08:13.315 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:08:13.315 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T08:08:13.343 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T08:08:13.409 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T08:08:13.409 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:08:13.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T08:08:13.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T08:08:13.606 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:08:13.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T08:08:13.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T08:08:13.692 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:08:13.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T08:08:13.717 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T08:08:13.790 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T08:08:13.790 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:08:13.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T08:08:13.866 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T08:08:13.978 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:08:13.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T08:08:14.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T08:08:14.066 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:08:14.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T08:08:14.089 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T08:08:14.152 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-24T08:08:14.258 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'subvols': {'create': 2}} 2024-04-24T08:08:14.259 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:08:14.259 INFO:tasks.cephadm:Cluster fsid is cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:08:14.259 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-24T08:08:14.260 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.146', 'mon.b': '172.21.15.153', 'mon.c': '172.21.15.177'} 2024-04-24T08:08:14.260 INFO:tasks.cephadm:First mon is mon.a on smithi146 2024-04-24T08:08:14.260 INFO:tasks.cephadm:First mgr is x 2024-04-24T08:08:14.260 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-24T08:08:14.260 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-04-24T08:08:14.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-04-24T08:08:14.367 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-04-24T08:08:14.391 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-24T08:08:14.391 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:08:14.533 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78823/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3116-g1f81fb39', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3116.g1f81fb39'}, 'url': 'https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 01:00:13.634707', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'rishabh-main-apr17-a654945', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-24T08:08:14.746 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref rishabh-main-apr17-a654945, sha1 1f81fb397ae98da7563d451a78c61574c8f4e6e0 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:08:14.747 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm 2024-04-24T08:08:14.748 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:15.514 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-r--r--. 1 ubuntu ubuntu 781486 Apr 24 08:08 /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:15.515 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:16.197 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 781486 Apr 24 08:08 /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:16.198 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:16.882 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 781486 Apr 24 08:08 /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:16.883 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:16.940 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:16.958 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T08:08:16.993 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 on all hosts... 2024-04-24T08:08:16.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 pull 2024-04-24T08:08:17.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 pull 2024-04-24T08:08:17.003 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 pull 2024-04-24T08:08:18.670 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T08:08:18.728 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T08:08:19.050 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T08:09:12.416 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:09:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:09:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02", 2024-04-24T08:09:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2024-04-24T08:09:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644" 2024-04-24T08:09:12.417 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:09:12.417 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:09:27.180 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-24T08:09:27.180 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:09:27.180 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02", 2024-04-24T08:09:27.180 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-04-24T08:09:27.180 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644" 2024-04-24T08:09:27.180 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-24T08:09:27.181 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-24T08:09:28.235 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-24T08:09:28.235 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:09:28.235 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02", 2024-04-24T08:09:28.235 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-04-24T08:09:28.235 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644" 2024-04-24T08:09:28.235 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-24T08:09:28.235 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-24T08:09:28.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-04-24T08:09:28.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-04-24T08:09:28.386 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-04-24T08:09:28.411 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-04-24T08:09:28.498 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-04-24T08:09:28.527 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-04-24T08:09:28.551 INFO:tasks.cephadm:Writing seed config... 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-24T08:09:28.552 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-24T08:09:28.553 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-24T08:09:28.554 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-24T08:09:28.555 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-24T08:09:28.555 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-24T08:09:28.555 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-24T08:09:28.555 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-24T08:09:28.555 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-24T08:09:28.555 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-24T08:09:28.556 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:09:28.556 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-24T08:09:28.612 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = cca14746-0211-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-24T08:09:28.612 DEBUG:teuthology.orchestra.run.smithi146:mon.a> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a.service 2024-04-24T08:09:28.661 DEBUG:teuthology.orchestra.run.smithi146:mgr.x> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.x.service 2024-04-24T08:09:28.707 INFO:tasks.cephadm:Bootstrapping... 2024-04-24T08:09:28.708 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 -v bootstrap --fsid cca14746-0211-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.146 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:09:28.975 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-04-24T08:09:28.975 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0', '-v', 'bootstrap', '--fsid', 'cca14746-0211-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.146', '--skip-admin-label'] 2024-04-24T08:09:28.976 INFO:teuthology.orchestra.run.smithi146.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-24T08:09:28.976 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying podman|docker is present... 2024-04-24T08:09:29.033 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-24T08:09:29.033 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying lvm2 is present... 2024-04-24T08:09:29.034 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying time synchronization is in place... 2024-04-24T08:09:29.049 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T08:09:29.049 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T08:09:29.063 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T08:09:29.063 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout inactive 2024-04-24T08:09:29.084 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout enabled 2024-04-24T08:09:29.100 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout active 2024-04-24T08:09:29.100 INFO:teuthology.orchestra.run.smithi146.stdout:Unit chronyd.service is enabled and running 2024-04-24T08:09:29.100 INFO:teuthology.orchestra.run.smithi146.stdout:Repeating the final host check... 2024-04-24T08:09:29.154 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-24T08:09:29.154 INFO:teuthology.orchestra.run.smithi146.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-24T08:09:29.155 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl is present 2024-04-24T08:09:29.155 INFO:teuthology.orchestra.run.smithi146.stdout:lvcreate is present 2024-04-24T08:09:29.174 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T08:09:29.174 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T08:09:29.196 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T08:09:29.197 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout inactive 2024-04-24T08:09:29.222 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout enabled 2024-04-24T08:09:29.244 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout active 2024-04-24T08:09:29.245 INFO:teuthology.orchestra.run.smithi146.stdout:Unit chronyd.service is enabled and running 2024-04-24T08:09:29.245 INFO:teuthology.orchestra.run.smithi146.stdout:Host looks OK 2024-04-24T08:09:29.245 INFO:teuthology.orchestra.run.smithi146.stdout:Cluster fsid: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:29.246 INFO:teuthology.orchestra.run.smithi146.stdout:Acquiring lock 140213350235296 on /run/cephadm/cca14746-0211-11ef-bc93-c7b262605968.lock 2024-04-24T08:09:29.246 INFO:teuthology.orchestra.run.smithi146.stdout:Lock 140213350235296 acquired on /run/cephadm/cca14746-0211-11ef-bc93-c7b262605968.lock 2024-04-24T08:09:29.248 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying IP 172.21.15.146 port 3300 ... 2024-04-24T08:09:29.248 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying IP 172.21.15.146 port 6789 ... 2024-04-24T08:09:29.249 INFO:teuthology.orchestra.run.smithi146.stdout:Base mon IP(s) is [172.21.15.146:3300, 172.21.15.146:6789], mon addrv is [v2:172.21.15.146:3300,v1:172.21.15.146:6789] 2024-04-24T08:09:29.263 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.146 metric 100 2024-04-24T08:09:29.264 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.146 metric 100 2024-04-24T08:09:29.276 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-24T08:09:29.276 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-24T08:09:29.276 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-04-24T08:09:29.288 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-24T08:09:29.288 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-24T08:09:29.288 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T08:09:29.288 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-24T08:09:29.288 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-04-24T08:09:29.288 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T08:09:29.291 INFO:teuthology.orchestra.run.smithi146.stdout:Mon IP `172.21.15.146` is in CIDR network `172.21.0.0/20` 2024-04-24T08:09:29.291 INFO:teuthology.orchestra.run.smithi146.stdout:Mon IP `172.21.15.146` is in CIDR network `172.21.0.0/20` 2024-04-24T08:09:29.291 INFO:teuthology.orchestra.run.smithi146.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-24T08:09:29.294 INFO:teuthology.orchestra.run.smithi146.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-24T08:09:29.296 INFO:teuthology.orchestra.run.smithi146.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T08:09:29.917 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stdout b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 2024-04-24T08:09:29.917 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T08:09:29.918 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stderr Getting image source signatures 2024-04-24T08:09:29.918 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-24T08:09:29.918 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stderr Copying blob sha256:b727accf024e55f70f86e348fc139074bb9b3de1cca3400f8b76b2ed7d35ac83 2024-04-24T08:09:29.918 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stderr Copying config sha256:b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 2024-04-24T08:09:29.918 INFO:teuthology.orchestra.run.smithi146.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-24T08:09:38.756 INFO:teuthology.orchestra.run.smithi146.stdout:ceph: stdout ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev) 2024-04-24T08:09:38.757 INFO:teuthology.orchestra.run.smithi146.stdout:Ceph version: ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev) 2024-04-24T08:09:38.757 INFO:teuthology.orchestra.run.smithi146.stdout:Extracting ceph user uid/gid from container image... 2024-04-24T08:09:39.959 INFO:teuthology.orchestra.run.smithi146.stdout:stat: stdout 167 167 2024-04-24T08:09:39.959 INFO:teuthology.orchestra.run.smithi146.stdout:Creating initial keys... 2024-04-24T08:09:41.176 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph-authtool: stdout AQBEvihmLijGGhAAW5uPtkPZKOjfFvFc5dJeOA== 2024-04-24T08:09:42.338 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph-authtool: stdout AQBFvihmGrYTKRAAjNfN+Ybx8xNDFTZgYnvOmw== 2024-04-24T08:09:43.926 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph-authtool: stdout AQBGvihmVyiXMBAAgs6xqlA3ZWD+ug7DKQlvyA== 2024-04-24T08:09:43.929 INFO:teuthology.orchestra.run.smithi146.stdout:Creating initial monmap... 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:monmaptool for a [v2:172.21.15.146:3300,v1:172.21.15.146:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:setting min_mon_release = quincy 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/monmaptool: set fsid to cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T08:09:45.191 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:09:45.192 INFO:teuthology.orchestra.run.smithi146.stdout:Creating mon... 2024-04-24T08:09:46.365 INFO:teuthology.orchestra.run.smithi146.stdout:create mon.a on 2024-04-24T08:09:46.752 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-24T08:09:46.997 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-24T08:09:47.284 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-cca14746-0211-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-cca14746-0211-11ef-bc93-c7b262605968.target. 2024-04-24T08:09:47.284 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-cca14746-0211-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-cca14746-0211-11ef-bc93-c7b262605968.target. 2024-04-24T08:09:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a 2024-04-24T08:09:47.654 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-24T08:09:47.955 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cca14746-0211-11ef-bc93-c7b262605968.target.wants/ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-cca14746-0211-11ef-bc93-c7b262605968@.service. 2024-04-24T08:09:48.607 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:48 smithi146 podman[27419]: 2024-04-24 08:09:48.257596648 +0000 UTC m=+0.035804046 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:09:48.607 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:48 smithi146 podman[27419]: 2024-04-24 08:09:48.605395557 +0000 UTC m=+0.383603036 container init 57788ad2bc4bb71d2c1aee5a236391895c408e6c72a9e6d7bb96255e2ef0c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-24T08:09:48.732 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T08:09:48.733 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout disabled 2024-04-24T08:09:48.752 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T08:09:48.752 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout inactive 2024-04-24T08:09:48.752 INFO:teuthology.orchestra.run.smithi146.stdout:firewalld.service is not enabled 2024-04-24T08:09:48.752 INFO:teuthology.orchestra.run.smithi146.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T08:09:48.753 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for mon to start... 2024-04-24T08:09:48.753 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for mon... 2024-04-24T08:09:48.900 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:48 smithi146 podman[27419]: 2024-04-24 08:09:48.618173513 +0000 UTC m=+0.396380926 container start 57788ad2bc4bb71d2c1aee5a236391895c408e6c72a9e6d7bb96255e2ef0c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True) 2024-04-24T08:09:48.900 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:48 smithi146 bash[27419]: 57788ad2bc4bb71d2c1aee5a236391895c408e6c72a9e6d7bb96255e2ef0c873 2024-04-24T08:09:48.900 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:48 smithi146 ceph-mon[27438]: mkfs cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:48.900 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:48 smithi146 ceph-mon[27438]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T08:09:48.900 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:48 smithi146 systemd[1]: Started Ceph mon.a for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: monmap epoch 1 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: last_changed 2024-04-24T08:09:44.411147+0000 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: min_mon_release 19 (squid) 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: election_strategy: 1 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: fsmap 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: mgrmap e1: no daemons active 2024-04-24T08:09:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:49 smithi146 ceph-mon[27438]: from='client.? 172.21.15.146:0/2052681317' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T08:09:50.137 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout cluster: 2024-04-24T08:09:50.143 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout id: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout services: 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.753733s) 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout data: 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-24T08:09:50.144 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout pgs: 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.355+0000 7f79a2800700 1 Processor -- start 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.355+0000 7f79a2800700 1 -- start start 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.355+0000 7f79a2800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.355+0000 7f79a2800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f799c1090b0 con 0x7f799c1086c0 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.357+0000 7f799b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.357+0000 7f799b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50030/0 (socket says 172.21.15.146:50030) 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.357+0000 7f799b400700 1 -- 172.21.15.146:0/2727791639 learned_addr learned my addr 172.21.15.146:0/2727791639 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.357+0000 7f799b400700 1 -- 172.21.15.146:0/2727791639 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c1098c0 con 0x7f799c1086c0 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.358+0000 7f799b400700 1 --2- 172.21.15.146:0/2727791639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c108ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098a0 tx=0x7f798c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeec1fa3e49c0e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.358+0000 7f799a000700 1 -- 172.21.15.146:0/2727791639 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f798c004160 con 0x7f799c1086c0 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.358+0000 7f799a000700 1 -- 172.21.15.146:0/2727791639 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798c004300 con 0x7f799c1086c0 2024-04-24T08:09:50.145 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.358+0000 7f799a000700 1 -- 172.21.15.146:0/2727791639 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f798c0036a0 con 0x7f799c1086c0 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.359+0000 7f79a2800700 1 -- 172.21.15.146:0/2727791639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 msgr2=0x7f799c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.359+0000 7f79a2800700 1 --2- 172.21.15.146:0/2727791639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c108ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f798c0098a0 tx=0x7f798c009bb0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.359+0000 7f79a2800700 1 -- 172.21.15.146:0/2727791639 shutdown_connections 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.359+0000 7f79a2800700 1 --2- 172.21.15.146:0/2727791639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c108ae0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.359+0000 7f79a2800700 1 -- 172.21.15.146:0/2727791639 >> 172.21.15.146:0/2727791639 conn(0x7f799c103cc0 msgr2=0x7f799c1060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.359+0000 7f79a2800700 1 -- 172.21.15.146:0/2727791639 shutdown_connections 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.359+0000 7f79a2800700 1 -- 172.21.15.146:0/2727791639 wait complete. 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.360+0000 7f79a2800700 1 Processor -- start 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.360+0000 7f79a2800700 1 -- start start 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.361+0000 7f79a2800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c19e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.361+0000 7f79a2800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f799c109df0 con 0x7f799c1086c0 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.361+0000 7f799b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c19e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.146 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.361+0000 7f799b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c19e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50032/0 (socket says 172.21.15.146:50032) 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.361+0000 7f799b400700 1 -- 172.21.15.146:0/2052681317 learned_addr learned my addr 172.21.15.146:0/2052681317 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.361+0000 7f799b400700 1 -- 172.21.15.146:0/2052681317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c19e6b0 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.362+0000 7f799b400700 1 --2- 172.21.15.146:0/2052681317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c19e170 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f798c000c00 tx=0x7f798c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.362+0000 7f7993e00700 1 -- 172.21.15.146:0/2052681317 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f798c003db0 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.362+0000 7f7993e00700 1 -- 172.21.15.146:0/2052681317 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f798c011630 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.362+0000 7f7993e00700 1 -- 172.21.15.146:0/2052681317 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f798c011950 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.362+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799c19e940 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.362+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799c19f460 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.363+0000 7f7993e00700 1 -- 172.21.15.146:0/2052681317 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f798c01a460 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.363+0000 7f7993e00700 1 -- 172.21.15.146:0/2052681317 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f798c01b780 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.363+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799c064150 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.366+0000 7f7993e00700 1 -- 172.21.15.146:0/2052681317 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f798c031020 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.446+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f799c19f860 con 0x7f799c1086c0 2024-04-24T08:09:50.147 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.447+0000 7f7993e00700 1 -- 172.21.15.146:0/2052681317 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f798c01b9d0 con 0x7f799c1086c0 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.449+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 msgr2=0x7f799c19e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.449+0000 7f79a2800700 1 --2- 172.21.15.146:0/2052681317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c19e170 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f798c000c00 tx=0x7f798c004750 comp rx=0 tx=0).stop 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.450+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 shutdown_connections 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.450+0000 7f79a2800700 1 --2- 172.21.15.146:0/2052681317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f799c1086c0 0x7f799c19e170 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.450+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 >> 172.21.15.146:0/2052681317 conn(0x7f799c103cc0 msgr2=0x7f799c07d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.450+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 shutdown_connections 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:49.450+0000 7f79a2800700 1 -- 172.21.15.146:0/2052681317 wait complete. 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:mon is available 2024-04-24T08:09:50.148 INFO:teuthology.orchestra.run.smithi146.stdout:Assimilating anything we can from ceph.conf... 2024-04-24T08:09:50.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:50 smithi146 ceph-mon[27438]: from='client.? 172.21.15.146:0/4272125749' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T08:09:50.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:50 smithi146 ceph-mon[27438]: from='client.? 172.21.15.146:0/4272125749' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-24T08:09:51.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:09:51.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T08:09:51.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout fsid = cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:51.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.146:3300,v1:172.21.15.146:6789] 2024-04-24T08:09:51.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T08:09:51.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.763+0000 7f815ee00700 1 Processor -- start 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.763+0000 7f815ee00700 1 -- start start 2024-04-24T08:09:51.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.763+0000 7f815ee00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815810acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.763+0000 7f815ee00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f815810b2a0 con 0x7f815810a8b0 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.764+0000 7f8157e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815810acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.764+0000 7f8157e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815810acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50046/0 (socket says 172.21.15.146:50046) 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.764+0000 7f8157e00700 1 -- 172.21.15.146:0/1177587406 learned_addr learned my addr 172.21.15.146:0/1177587406 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.765+0000 7f8157e00700 1 -- 172.21.15.146:0/1177587406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815810bab0 con 0x7f815810a8b0 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.765+0000 7f8157e00700 1 --2- 172.21.15.146:0/1177587406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815810acd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f81480098a0 tx=0x7f8148009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77c622cd0348379 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.765+0000 7f8156a00700 1 -- 172.21.15.146:0/1177587406 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8148004160 con 0x7f815810a8b0 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.765+0000 7f8156a00700 1 -- 172.21.15.146:0/1177587406 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8148004300 con 0x7f815810a8b0 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.765+0000 7f8156a00700 1 -- 172.21.15.146:0/1177587406 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81480036a0 con 0x7f815810a8b0 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.766+0000 7f815ee00700 1 -- 172.21.15.146:0/1177587406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 msgr2=0x7f815810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.766+0000 7f815ee00700 1 --2- 172.21.15.146:0/1177587406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815810acd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f81480098a0 tx=0x7f8148009bb0 comp rx=0 tx=0).stop 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.766+0000 7f815ee00700 1 -- 172.21.15.146:0/1177587406 shutdown_connections 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.766+0000 7f815ee00700 1 --2- 172.21.15.146:0/1177587406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815810acd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.766+0000 7f815ee00700 1 -- 172.21.15.146:0/1177587406 >> 172.21.15.146:0/1177587406 conn(0x7f815807dcc0 msgr2=0x7f815807e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:51.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.766+0000 7f815ee00700 1 -- 172.21.15.146:0/1177587406 shutdown_connections 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.766+0000 7f815ee00700 1 -- 172.21.15.146:0/1177587406 wait complete. 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.767+0000 7f815ee00700 1 Processor -- start 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.768+0000 7f815ee00700 1 -- start start 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.768+0000 7f815ee00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.768+0000 7f815ee00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f815810bfe0 con 0x7f815810a8b0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.768+0000 7f8157e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.768+0000 7f8157e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815819e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50056/0 (socket says 172.21.15.146:50056) 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.768+0000 7f8157e00700 1 -- 172.21.15.146:0/4272125749 learned_addr learned my addr 172.21.15.146:0/4272125749 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.769+0000 7f8157e00700 1 -- 172.21.15.146:0/4272125749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815819e670 con 0x7f815810a8b0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.769+0000 7f8157e00700 1 --2- 172.21.15.146:0/4272125749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815819e130 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8148000c00 tx=0x7f8148004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.769+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8148003db0 con 0x7f815810a8b0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.769+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8148011630 con 0x7f815810a8b0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.769+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8148011950 con 0x7f815810a8b0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.769+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f815819e900 con 0x7f815810a8b0 2024-04-24T08:09:51.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.769+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f815819f420 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.770+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f814801a460 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.770+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f814801b780 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.771+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8158064150 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.773+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f814801b980 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.851+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f815819fff0 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.857+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+458 (secure 0 0 0) 0x7f814801bb60 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.858+0000 7f8154c00700 1 -- 172.21.15.146:0/4272125749 <== mon.0 v2:172.21.15.146:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f814801ada0 con 0x7f815810a8b0 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.860+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 msgr2=0x7f815819e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.860+0000 7f815ee00700 1 --2- 172.21.15.146:0/4272125749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815819e130 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8148000c00 tx=0x7f8148004750 comp rx=0 tx=0).stop 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.860+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 shutdown_connections 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.860+0000 7f815ee00700 1 --2- 172.21.15.146:0/4272125749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f815810a8b0 0x7f815819e130 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.860+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 >> 172.21.15.146:0/4272125749 conn(0x7f815807dcc0 msgr2=0x7f8158194cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.860+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 shutdown_connections 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:50.860+0000 7f815ee00700 1 -- 172.21.15.146:0/4272125749 wait complete. 2024-04-24T08:09:51.583 INFO:teuthology.orchestra.run.smithi146.stdout:Generating new minimal ceph.conf... 2024-04-24T08:09:52.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:52 smithi146 ceph-mon[27438]: from='client.? 172.21.15.146:0/4159446297' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:09:52.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.197+0000 7f24bee00700 1 Processor -- start 2024-04-24T08:09:52.993 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.197+0000 7f24bee00700 1 -- start start 2024-04-24T08:09:52.993 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.198+0000 7f24bee00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b8108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:52.993 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.198+0000 7f24bee00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81090b0 con 0x7f24b81086c0 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.198+0000 7f24b7e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b8108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.198+0000 7f24b7e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b8108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43700/0 (socket says 172.21.15.146:43700) 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.198+0000 7f24b7e00700 1 -- 172.21.15.146:0/1137071917 learned_addr learned my addr 172.21.15.146:0/1137071917 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.199+0000 7f24b7e00700 1 -- 172.21.15.146:0/1137071917 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b81098c0 con 0x7f24b81086c0 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.199+0000 7f24b7e00700 1 --2- 172.21.15.146:0/1137071917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b8108ae0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009e80 tx=0x7f24a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efc2be35f5c7879 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.199+0000 7f24b7400700 1 -- 172.21.15.146:0/1137071917 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24a8017070 con 0x7f24b81086c0 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.199+0000 7f24b7400700 1 -- 172.21.15.146:0/1137071917 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f24a800b8c0 con 0x7f24b81086c0 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.199+0000 7f24b7400700 1 -- 172.21.15.146:0/1137071917 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24a80045c0 con 0x7f24b81086c0 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.200+0000 7f24bee00700 1 -- 172.21.15.146:0/1137071917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 msgr2=0x7f24b8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.200+0000 7f24bee00700 1 --2- 172.21.15.146:0/1137071917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b8108ae0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009e80 tx=0x7f24a800b330 comp rx=0 tx=0).stop 2024-04-24T08:09:52.995 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.200+0000 7f24bee00700 1 -- 172.21.15.146:0/1137071917 shutdown_connections 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.200+0000 7f24bee00700 1 --2- 172.21.15.146:0/1137071917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b8108ae0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.200+0000 7f24bee00700 1 -- 172.21.15.146:0/1137071917 >> 172.21.15.146:0/1137071917 conn(0x7f24b8103cc0 msgr2=0x7f24b81060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.200+0000 7f24bee00700 1 -- 172.21.15.146:0/1137071917 shutdown_connections 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.200+0000 7f24bee00700 1 -- 172.21.15.146:0/1137071917 wait complete. 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.201+0000 7f24bee00700 1 Processor -- start 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.202+0000 7f24bee00700 1 -- start start 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.202+0000 7f24bee00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.202+0000 7f24bee00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f24b810cc70 con 0x7f24b81086c0 2024-04-24T08:09:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.202+0000 7f24b7e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.202+0000 7f24b7e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b819c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43714/0 (socket says 172.21.15.146:43714) 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.202+0000 7f24b7e00700 1 -- 172.21.15.146:0/4159446297 learned_addr learned my addr 172.21.15.146:0/4159446297 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.202+0000 7f24b7e00700 1 -- 172.21.15.146:0/4159446297 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b819ccd0 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.203+0000 7f24b7e00700 1 --2- 172.21.15.146:0/4159446297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b819c790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f24a8004210 tx=0x7f24a800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.203+0000 7f24b5600700 1 -- 172.21.15.146:0/4159446297 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24a8020020 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.203+0000 7f24b5600700 1 -- 172.21.15.146:0/4159446297 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f24a8003b50 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.203+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b819cf60 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.203+0000 7f24b5600700 1 -- 172.21.15.146:0/4159446297 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24a8017040 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.203+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b819da80 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.203+0000 7f24b5600700 1 -- 172.21.15.146:0/4159446297 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f24a8023020 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.204+0000 7f24b5600700 1 -- 172.21.15.146:0/4159446297 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f24a800f050 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.205+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b8064150 con 0x7f24b81086c0 2024-04-24T08:09:52.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.207+0000 7f24b5600700 1 -- 172.21.15.146:0/4159446297 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f24a802a020 con 0x7f24b81086c0 2024-04-24T08:09:53.000 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.284+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f24b819df10 con 0x7f24b81086c0 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.285+0000 7f24b5600700 1 -- 172.21.15.146:0/4159446297 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f24a800b860 con 0x7f24b81086c0 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.286+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 msgr2=0x7f24b819c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.286+0000 7f24bee00700 1 --2- 172.21.15.146:0/4159446297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b819c790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f24a8004210 tx=0x7f24a800bd50 comp rx=0 tx=0).stop 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.286+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 shutdown_connections 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.286+0000 7f24bee00700 1 --2- 172.21.15.146:0/4159446297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b81086c0 0x7f24b819c790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.286+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 >> 172.21.15.146:0/4159446297 conn(0x7f24b8103cc0 msgr2=0x7f24b8104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.286+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 shutdown_connections 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:52.287+0000 7f24bee00700 1 -- 172.21.15.146:0/4159446297 wait complete. 2024-04-24T08:09:53.001 INFO:teuthology.orchestra.run.smithi146.stdout:Restarting the monitor... 2024-04-24T08:09:53.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:53 smithi146 systemd[1]: Stopping Ceph mon.a for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:09:53.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a[27434]: 2024-04-24T08:09:53.504+0000 7f1e6c200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:09:53.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a[27434]: 2024-04-24T08:09:53.504+0000 7f1e6c200700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-24T08:09:54.128 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:53 smithi146 podman[27681]: 2024-04-24 08:09:53.76195681 +0000 UTC m=+0.392314775 container died 57788ad2bc4bb71d2c1aee5a236391895c408e6c72a9e6d7bb96255e2ef0c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T08:09:54.416 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:54 smithi146 podman[27681]: 2024-04-24 08:09:54.128179805 +0000 UTC m=+0.758537810 container cleanup 57788ad2bc4bb71d2c1aee5a236391895c408e6c72a9e6d7bb96255e2ef0c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:09:54.416 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:54 smithi146 bash[27681]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a 2024-04-24T08:09:54.416 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:54 smithi146 podman[27693]: 2024-04-24 08:09:54.314344082 +0000 UTC m=+0.537277049 container remove 57788ad2bc4bb71d2c1aee5a236391895c408e6c72a9e6d7bb96255e2ef0c873 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-24T08:09:54.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:54 smithi146 systemd[1]: ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-24T08:09:54.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:54 smithi146 systemd[1]: Stopped Ceph mon.a for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:09:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:54 smithi146 systemd[1]: ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.910s CPU time. 2024-04-24T08:09:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:54 smithi146 systemd[1]: Starting Ceph mon.a for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:09:55.433 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 podman[27787]: 2024-04-24 08:09:55.156792903 +0000 UTC m=+0.123586704 container create efc8b69559a513a7efdf615e24e590399949638fbdf77ad57eef3c4043fcafce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-04-24T08:09:55.433 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 podman[27787]: 2024-04-24 08:09:55.066729058 +0000 UTC m=+0.033522821 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:09:55.433 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 podman[27787]: 2024-04-24 08:09:55.37682138 +0000 UTC m=+0.343615198 container init efc8b69559a513a7efdf615e24e590399949638fbdf77ad57eef3c4043fcafce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5) 2024-04-24T08:09:55.433 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 podman[27787]: 2024-04-24 08:09:55.38677253 +0000 UTC m=+0.353566334 container start efc8b69559a513a7efdf615e24e590399949638fbdf77ad57eef3c4043fcafce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T08:09:55.515 INFO:teuthology.orchestra.run.smithi146.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-24T08:09:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T08:09:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev), process ceph-mon, pid 2 2024-04-24T08:09:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: pidfile_write: ignore empty --pid-file 2024-04-24T08:09:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: load: jerasure load: lrc 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: RocksDB version: 7.9.2 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Git sha 0 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Compile date 2024-04-23 21:06:33 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: DB SUMMARY 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: DB Session ID: 68ZY1STN3VJ578AIBJYP 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: CURRENT file: CURRENT 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106058 ; 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.error_if_exists: 0 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.create_if_missing: 0 2024-04-24T08:09:55.747 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.paranoid_checks: 1 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.env: 0x561362b6ccc0 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.info_log: 0x561393f43c60 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.statistics: (nil) 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.use_fsync: 0 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_log_file_size: 0 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T08:09:55.748 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.allow_fallocate: 1 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.use_direct_reads: 0 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.db_log_dir: 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.wal_dir: 2024-04-24T08:09:55.749 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T08:09:55.751 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.write_buffer_manager: 0x561393f121e0 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.unordered_write: 0 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T08:09:55.752 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.row_cache: None 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.wal_filter: None 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.two_write_queues: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.wal_compression: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.atomic_flush: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.log_readahead_size: 0 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T08:09:55.753 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_background_jobs: 2 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_background_compactions: -1 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_subcompactions: 1 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T08:09:55.754 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_open_files: -1 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_background_flushes: -1 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Compression algorithms supported: 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kZSTD supported: 0 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kXpressCompression supported: 0 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T08:09:55.755 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kZlibCompression supported: 1 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kSnappyCompression supported: 1 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kLZ4Compression supported: 1 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: kBZip2Compression supported: 0 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.merge_operator: 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_filter: None 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561393f43e20) 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-04-24T08:09:55.756 INFO:journalctl@ceph.mon.a.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: index_type: 0 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: data_block_index_type: 0 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: index_shortening: 1 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: checksum: 4 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: no_block_cache: 0 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_cache: 0x561393f45090 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_cache_options: 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: capacity : 536870912 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: num_shard_bits : 4 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: strict_capacity_limit : 0 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-04-24T08:09:55.757 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_cache_compressed: (nil) 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: persistent_cache: (nil) 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_size: 4096 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_size_deviation: 10 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_restart_interval: 16 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: index_block_restart_interval: 1 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: metadata_block_size: 4096 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: partition_filters: 0 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: use_delta_encoding: 1 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: filter_policy: bloomfilter 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: whole_key_filtering: 1 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: verify_compression: 0 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: format_version: 5 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: enable_index_compression: 1 2024-04-24T08:09:55.758 INFO:journalctl@ceph.mon.a.smithi146.stdout: block_align: 0 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout: max_auto_readahead_size: 262144 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout: prepopulate_block_cache: 0 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout: initial_auto_readahead_size: 8192 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression: NoCompression 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.num_levels: 7 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T08:09:55.759 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T08:09:55.760 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T08:09:55.761 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T08:09:55.761 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T08:09:55.761 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T08:09:55.761 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T08:09:55.761 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T08:09:55.761 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T08:09:55.761 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T08:09:55.762 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T08:09:55.763 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.inplace_update_support: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.bloom_locality: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.max_successive_merges: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T08:09:55.764 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.ttl: 2592000 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.enable_blob_files: false 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.min_blob_size: 0 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T08:09:55.765 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 28b2875c-96be-4ee6-b3b6-ae5cc336c9c8 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713946195442399, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713946195444446, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101082, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 98971, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14409, "raw_average_key_size": 49, "raw_value_size": 91174, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713946195, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "28b2875c-96be-4ee6-b3b6-ae5cc336c9c8", "db_session_id": "68ZY1STN3VJ578AIBJYP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713946195444653, "job": 1, "event": "recovery_finished"} 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561394032000 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: DB pointer 0x56139401c000 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T08:09:55.766 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: ** DB Stats ** 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: ** Compaction Stats [default] ** 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: L0 2/0 100.57 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 69.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T08:09:55.767 INFO:journalctl@ceph.mon.a.smithi146.stdout: Sum 2/0 100.57 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 69.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 69.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: ** Compaction Stats [default] ** 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 69.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T08:09:55.768 INFO:journalctl@ceph.mon.a.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: Cumulative compaction: 0.00 GB write, 14.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: Interval compaction: 0.00 GB write, 14.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: Block cache BinnedLRUCache@0x561393f45090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: starting mon.a rank 0 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???) e1 preinit fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???).mds e1 new map 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???).mds e1 print_map 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: e1 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T08:09:55.769 INFO:journalctl@ceph.mon.a.smithi146.stdout: legacy client fscid: -1 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout: 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout: No filesystems configured 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 bash[27787]: efc8b69559a513a7efdf615e24e590399949638fbdf77ad57eef3c4043fcafce 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 systemd[1]: Started Ceph mon.a for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: monmap epoch 1 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: last_changed 2024-04-24T08:09:44.411147+0000 2024-04-24T08:09:55.770 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:09:55.771 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: min_mon_release 19 (squid) 2024-04-24T08:09:55.771 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: election_strategy: 1 2024-04-24T08:09:55.771 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:09:55.771 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: fsmap 2024-04-24T08:09:55.771 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T08:09:55.771 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:55 smithi146 ceph-mon[27805]: mgrmap e1: no daemons active 2024-04-24T08:09:56.928 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.174+0000 7fa7bd800700 1 Processor -- start 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.174+0000 7fa7bd800700 1 -- start start 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.175+0000 7fa7bd800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b8108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.175+0000 7fa7bd800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b81090b0 con 0x7fa7b81086c0 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.175+0000 7fa7b6a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b8108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.176+0000 7fa7b6a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b8108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43728/0 (socket says 172.21.15.146:43728) 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.176+0000 7fa7b6a00700 1 -- 172.21.15.146:0/2590945569 learned_addr learned my addr 172.21.15.146:0/2590945569 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.176+0000 7fa7b6a00700 1 -- 172.21.15.146:0/2590945569 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b81098c0 con 0x7fa7b81086c0 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.176+0000 7fa7b6a00700 1 --2- 172.21.15.146:0/2590945569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b8108ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80098a0 tx=0x7fa7a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f741c933a2fd94cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7b6000700 1 -- 172.21.15.146:0/2590945569 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8004160 con 0x7fa7b81086c0 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7b6000700 1 -- 172.21.15.146:0/2590945569 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7a8004300 con 0x7fa7b81086c0 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7b6000700 1 -- 172.21.15.146:0/2590945569 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8011b50 con 0x7fa7b81086c0 2024-04-24T08:09:56.929 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7bd800700 1 -- 172.21.15.146:0/2590945569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 msgr2=0x7fa7b8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7bd800700 1 --2- 172.21.15.146:0/2590945569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b8108ae0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80098a0 tx=0x7fa7a8009bb0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7bd800700 1 -- 172.21.15.146:0/2590945569 shutdown_connections 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7bd800700 1 --2- 172.21.15.146:0/2590945569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b8108ae0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.177+0000 7fa7bd800700 1 -- 172.21.15.146:0/2590945569 >> 172.21.15.146:0/2590945569 conn(0x7fa7b8103cc0 msgr2=0x7fa7b81060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.178+0000 7fa7bd800700 1 -- 172.21.15.146:0/2590945569 shutdown_connections 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.178+0000 7fa7bd800700 1 -- 172.21.15.146:0/2590945569 wait complete. 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.179+0000 7fa7bd800700 1 Processor -- start 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.179+0000 7fa7bd800700 1 -- start start 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.179+0000 7fa7bd800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.179+0000 7fa7bd800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b810cc70 con 0x7fa7b81086c0 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.179+0000 7fa7b6a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.179+0000 7fa7b6a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b819c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43742/0 (socket says 172.21.15.146:43742) 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.179+0000 7fa7b6a00700 1 -- 172.21.15.146:0/2549541406 learned_addr learned my addr 172.21.15.146:0/2549541406 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.180+0000 7fa7b6a00700 1 -- 172.21.15.146:0/2549541406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b819cd20 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.180+0000 7fa7b6a00700 1 --2- 172.21.15.146:0/2549541406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b819c7e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8011850 tx=0x7fa7a8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.180+0000 7fa7af400700 1 -- 172.21.15.146:0/2549541406 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8003af0 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.180+0000 7fa7af400700 1 -- 172.21.15.146:0/2549541406 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7a8020070 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.180+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b819cfb0 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.180+0000 7fa7af400700 1 -- 172.21.15.146:0/2549541406 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a80195a0 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.181+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b819dad0 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.181+0000 7fa7af400700 1 -- 172.21.15.146:0/2549541406 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa7a8019c30 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.181+0000 7fa7af400700 1 -- 172.21.15.146:0/2549541406 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa7a8011e00 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.182+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b8064150 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.185+0000 7fa7af400700 1 -- 172.21.15.146:0/2549541406 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa7a8003c90 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.261+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fa7b8074de0 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.265+0000 7fa7af400700 1 -- 172.21.15.146:0/2549541406 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fa7a801a430 con 0x7fa7b81086c0 2024-04-24T08:09:56.931 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.267+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 msgr2=0x7fa7b819c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.267+0000 7fa7bd800700 1 --2- 172.21.15.146:0/2549541406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b819c7e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8011850 tx=0x7fa7a8011c10 comp rx=0 tx=0).stop 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.267+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 shutdown_connections 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.267+0000 7fa7bd800700 1 --2- 172.21.15.146:0/2549541406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7b81086c0 0x7fa7b819c7e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.267+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 >> 172.21.15.146:0/2549541406 conn(0x7fa7b8103cc0 msgr2=0x7fa7b8104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.267+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 shutdown_connections 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:56.267+0000 7fa7bd800700 1 -- 172.21.15.146:0/2549541406 wait complete. 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi146.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-24T08:09:56.956 INFO:teuthology.orchestra.run.smithi146.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:09:56.956 INFO:teuthology.orchestra.run.smithi146.stdout:Creating mgr... 2024-04-24T08:09:56.958 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-24T08:09:56.959 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-24T08:09:57.385 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.x 2024-04-24T08:09:57.386 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-24T08:09:57.639 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cca14746-0211-11ef-bc93-c7b262605968.target.wants/ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-cca14746-0211-11ef-bc93-c7b262605968@.service. 2024-04-24T08:09:57.673 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:57 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2549541406' entity='client.admin' 2024-04-24T08:09:57.993 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:57 smithi146 systemd[1]: Starting Ceph mgr.x for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:09:58.273 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 podman[28016]: 2024-04-24 08:09:57.926204214 +0000 UTC m=+0.033322838 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:09:58.273 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 podman[28016]: 2024-04-24 08:09:58.041548859 +0000 UTC m=+0.148667514 container create 55a3410c0b3bf0977c82a65f076e47796d0a19b1f2c80da2afddc02cb277852c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:09:58.273 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 podman[28016]: 2024-04-24 08:09:58.261911177 +0000 UTC m=+0.369029839 container init 55a3410c0b3bf0977c82a65f076e47796d0a19b1f2c80da2afddc02cb277852c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, ceph=True) 2024-04-24T08:09:58.273 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 podman[28016]: 2024-04-24 08:09:58.272168888 +0000 UTC m=+0.379287533 container start 55a3410c0b3bf0977c82a65f076e47796d0a19b1f2c80da2afddc02cb277852c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS) 2024-04-24T08:09:58.387 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T08:09:58.387 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout disabled 2024-04-24T08:09:58.402 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T08:09:58.402 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout inactive 2024-04-24T08:09:58.403 INFO:teuthology.orchestra.run.smithi146.stdout:firewalld.service is not enabled 2024-04-24T08:09:58.403 INFO:teuthology.orchestra.run.smithi146.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T08:09:58.423 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T08:09:58.424 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout disabled 2024-04-24T08:09:58.444 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T08:09:58.444 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout inactive 2024-04-24T08:09:58.444 INFO:teuthology.orchestra.run.smithi146.stdout:firewalld.service is not enabled 2024-04-24T08:09:58.445 INFO:teuthology.orchestra.run.smithi146.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-24T08:09:58.445 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for mgr to start... 2024-04-24T08:09:58.445 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for mgr... 2024-04-24T08:09:58.598 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 bash[28016]: 55a3410c0b3bf0977c82a65f076e47796d0a19b1f2c80da2afddc02cb277852c 2024-04-24T08:09:58.598 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 systemd[1]: Started Ceph mgr.x for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:09:58.598 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:09:58.516+0000 7f065178f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:09:58.911 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:09:58.769+0000 7f065178f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:09:59.173 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:09:58.910+0000 7f065178f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:09:59.447 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:09:59 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/297004489' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T08:09:59.447 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:09:59.332+0000 7f065178f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:09:59.447 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:09:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:09:59.446+0000 7f065178f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:09:59.934 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:09:59.934 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 0 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:09:59.935 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T08:09:59.936 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T08:09:59.944 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:09:59.945 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T08:09:48.695676+0000", 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.082+0000 7fee13e00700 1 Processor -- start 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.082+0000 7fee13e00700 1 -- start start 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.082+0000 7fee13e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c07c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.082+0000 7fee13e00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c07c7b0 con 0x7fee0c07dd80 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.083+0000 7fee11400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c07c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.083+0000 7fee11400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c07c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43762/0 (socket says 172.21.15.146:43762) 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.083+0000 7fee11400700 1 -- 172.21.15.146:0/4202777580 learned_addr learned my addr 172.21.15.146:0/4202777580 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:59.946 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.083+0000 7fee11400700 1 -- 172.21.15.146:0/4202777580 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c07c930 con 0x7fee0c07dd80 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.084+0000 7fee11400700 1 --2- 172.21.15.146:0/4202777580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c07c1e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fee000098a0 tx=0x7fee00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e94b520efe2382 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.084+0000 7fee0be00700 1 -- 172.21.15.146:0/4202777580 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee00004160 con 0x7fee0c07dd80 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.084+0000 7fee0be00700 1 -- 172.21.15.146:0/4202777580 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fee00004300 con 0x7fee0c07dd80 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.084+0000 7fee0be00700 1 -- 172.21.15.146:0/4202777580 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee00011b50 con 0x7fee0c07dd80 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.085+0000 7fee13e00700 1 -- 172.21.15.146:0/4202777580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 msgr2=0x7fee0c07c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.085+0000 7fee13e00700 1 --2- 172.21.15.146:0/4202777580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c07c1e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fee000098a0 tx=0x7fee00009bb0 comp rx=0 tx=0).stop 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.085+0000 7fee13e00700 1 -- 172.21.15.146:0/4202777580 shutdown_connections 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.085+0000 7fee13e00700 1 --2- 172.21.15.146:0/4202777580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c07c1e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.085+0000 7fee13e00700 1 -- 172.21.15.146:0/4202777580 >> 172.21.15.146:0/4202777580 conn(0x7fee0c103cc0 msgr2=0x7fee0c1060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.085+0000 7fee13e00700 1 -- 172.21.15.146:0/4202777580 shutdown_connections 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.085+0000 7fee13e00700 1 -- 172.21.15.146:0/4202777580 wait complete. 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.086+0000 7fee13e00700 1 Processor -- start 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.086+0000 7fee13e00700 1 -- start start 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.087+0000 7fee13e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:59.947 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.087+0000 7fee13e00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c10ecd0 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.087+0000 7fee11400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.087+0000 7fee11400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c19e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43766/0 (socket says 172.21.15.146:43766) 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.087+0000 7fee11400700 1 -- 172.21.15.146:0/297004489 learned_addr learned my addr 172.21.15.146:0/297004489 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.087+0000 7fee11400700 1 -- 172.21.15.146:0/297004489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c19ef10 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee11400700 1 --2- 172.21.15.146:0/297004489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c19e9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fee00009fb0 tx=0x7fee00011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee0a000700 1 -- 172.21.15.146:0/297004489 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee000038b0 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee0a000700 1 -- 172.21.15.146:0/297004489 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fee00011e50 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c19f1a0 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee0a000700 1 -- 172.21.15.146:0/297004489 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee00019550 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c19fcc0 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee0a000700 1 -- 172.21.15.146:0/297004489 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fee00019b90 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.088+0000 7fee0a000700 1 -- 172.21.15.146:0/297004489 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fee00024730 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.089+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee0c064150 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.092+0000 7fee0a000700 1 -- 172.21.15.146:0/297004489 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fee00003a50 con 0x7fee0c07dd80 2024-04-24T08:09:59.948 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.172+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fee0c1a01b0 con 0x7fee0c07dd80 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.173+0000 7fee0a000700 1 -- 172.21.15.146:0/297004489 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fee00024980 con 0x7fee0c07dd80 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.174+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 msgr2=0x7fee0c19e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.174+0000 7fee13e00700 1 --2- 172.21.15.146:0/297004489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c19e9d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fee00009fb0 tx=0x7fee00011bf0 comp rx=0 tx=0).stop 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.174+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 shutdown_connections 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.174+0000 7fee13e00700 1 --2- 172.21.15.146:0/297004489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee0c07dd80 0x7fee0c19e9d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.174+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 >> 172.21.15.146:0/297004489 conn(0x7fee0c103cc0 msgr2=0x7fee0c104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.175+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 shutdown_connections 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:09:59.175+0000 7fee13e00700 1 -- 172.21.15.146:0/297004489 wait complete. 2024-04-24T08:09:59.949 INFO:teuthology.orchestra.run.smithi146.stdout:mgr not available, waiting (1/15)... 2024-04-24T08:10:01.490 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:01.048+0000 7f065178f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:10:01.490 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:01.274+0000 7f065178f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:01.490 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:01.489+0000 7f065178f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:10:02.148 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:01.871+0000 7f065178f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:10:02.148 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:01.997+0000 7f065178f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:10:02.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:02.264+0000 7f065178f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:10:02.792 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:02.524+0000 7f065178f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:03.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:02 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3916097444' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T08:10:03.628 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:03 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:03.337+0000 7f065178f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T08:10:03.640 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 0 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T08:10:03.641 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T08:10:03.642 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T08:10:03.642 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T08:10:03.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:10:03.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T08:09:48.695676+0000", 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.791+0000 7f8cee000700 1 Processor -- start 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.791+0000 7f8cee000700 1 -- start start 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.791+0000 7f8cee000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce8109370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.791+0000 7f8cee000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce80763d0 con 0x7f8ce8106f40 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.792+0000 7f8ce7400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce8109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.792+0000 7f8ce7400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce8109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38432/0 (socket says 172.21.15.146:38432) 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.792+0000 7f8ce7400700 1 -- 172.21.15.146:0/3354079112 learned_addr learned my addr 172.21.15.146:0/3354079112 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:03.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.792+0000 7f8ce7400700 1 -- 172.21.15.146:0/3354079112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce81098b0 con 0x7f8ce8106f40 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.793+0000 7f8ce7400700 1 --2- 172.21.15.146:0/3354079112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce8109370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80098a0 tx=0x7f8cd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77f5f6fb8abb662 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.793+0000 7f8ce6000700 1 -- 172.21.15.146:0/3354079112 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cd8004160 con 0x7f8ce8106f40 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.793+0000 7f8ce6000700 1 -- 172.21.15.146:0/3354079112 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8cd8004300 con 0x7f8ce8106f40 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.793+0000 7f8ce6000700 1 -- 172.21.15.146:0/3354079112 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cd8011b50 con 0x7f8ce8106f40 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.794+0000 7f8cee000700 1 -- 172.21.15.146:0/3354079112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 msgr2=0x7f8ce8109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.794+0000 7f8cee000700 1 --2- 172.21.15.146:0/3354079112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce8109370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80098a0 tx=0x7f8cd8009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.794+0000 7f8cee000700 1 -- 172.21.15.146:0/3354079112 shutdown_connections 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.794+0000 7f8cee000700 1 --2- 172.21.15.146:0/3354079112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce8109370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.794+0000 7f8cee000700 1 -- 172.21.15.146:0/3354079112 >> 172.21.15.146:0/3354079112 conn(0x7f8ce8102be0 msgr2=0x7f8ce8105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.794+0000 7f8cee000700 1 -- 172.21.15.146:0/3354079112 shutdown_connections 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.794+0000 7f8cee000700 1 -- 172.21.15.146:0/3354079112 wait complete. 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.795+0000 7f8cee000700 1 Processor -- start 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.795+0000 7f8cee000700 1 -- start start 2024-04-24T08:10:03.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.796+0000 7f8cee000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce81a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.796+0000 7f8cee000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce810a940 con 0x7f8ce8106f40 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.796+0000 7f8ce7400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce81a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.796+0000 7f8ce7400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce81a2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38438/0 (socket says 172.21.15.146:38438) 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.796+0000 7f8ce7400700 1 -- 172.21.15.146:0/3916097444 learned_addr learned my addr 172.21.15.146:0/3916097444 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.796+0000 7f8ce7400700 1 -- 172.21.15.146:0/3916097444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce81a3350 con 0x7f8ce8106f40 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8ce7400700 1 --2- 172.21.15.146:0/3916097444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce81a2e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8011b60 tx=0x7f8cd8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8cdfe00700 1 -- 172.21.15.146:0/3916097444 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cd8003970 con 0x7f8ce8106f40 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8cdfe00700 1 -- 172.21.15.146:0/3916097444 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8cd8003b10 con 0x7f8ce8106f40 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce81a35e0 con 0x7f8ce8106f40 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8cdfe00700 1 -- 172.21.15.146:0/3916097444 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8cd8024610 con 0x7f8ce8106f40 2024-04-24T08:10:03.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ce81a3fc0 con 0x7f8ce8106f40 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8cdfe00700 1 -- 172.21.15.146:0/3916097444 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8cd8024c50 con 0x7f8ce8106f40 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.797+0000 7f8cdfe00700 1 -- 172.21.15.146:0/3916097444 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8cd8011e00 con 0x7f8ce8106f40 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.798+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ce8064150 con 0x7f8ce8106f40 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.801+0000 7f8cdfe00700 1 -- 172.21.15.146:0/3916097444 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8cd801a9f0 con 0x7f8ce8106f40 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.880+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8ce8074de0 con 0x7f8ce8106f40 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.881+0000 7f8cdfe00700 1 -- 172.21.15.146:0/3916097444 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8cd801a430 con 0x7f8ce8106f40 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.882+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 msgr2=0x7f8ce81a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.882+0000 7f8cee000700 1 --2- 172.21.15.146:0/3916097444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce81a2e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8011b60 tx=0x7f8cd8011c40 comp rx=0 tx=0).stop 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.882+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 shutdown_connections 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.882+0000 7f8cee000700 1 --2- 172.21.15.146:0/3916097444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ce8106f40 0x7f8ce81a2e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.882+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 >> 172.21.15.146:0/3916097444 conn(0x7f8ce8102be0 msgr2=0x7f8ce8104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.883+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 shutdown_connections 2024-04-24T08:10:03.655 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:02.883+0000 7f8cee000700 1 -- 172.21.15.146:0/3916097444 wait complete. 2024-04-24T08:10:03.656 INFO:teuthology.orchestra.run.smithi146.stdout:mgr not available, waiting (2/15)... 2024-04-24T08:10:04.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:03 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:03.900+0000 7f065178f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:10:04.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:04.029+0000 7f065178f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:10:04.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:04.296+0000 7f065178f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:10:05.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:04.971+0000 7f065178f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:10:05.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:05.096+0000 7f065178f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:10:05.737 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:05.417+0000 7f065178f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:10:05.737 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:05.598+0000 7f065178f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:10:05.737 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:05.736+0000 7f065178f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:10:05.992 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:05.891+0000 7f065178f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:06.737 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:06 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3987288347' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T08:10:06.737 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:06.280+0000 7f065178f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:10:07.047 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 0 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T08:10:07.048 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T08:10:07.049 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T08:10:07.050 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:10:07.051 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T08:09:48.695676+0000", 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.244+0000 7eff8d800700 1 Processor -- start 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.244+0000 7eff8d800700 1 -- start start 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.244+0000 7eff8d800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff88108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.244+0000 7eff8d800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7eff881090b0 con 0x7eff881086c0 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.245+0000 7eff86a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff88108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.245+0000 7eff86a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff88108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38450/0 (socket says 172.21.15.146:38450) 2024-04-24T08:10:07.052 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.245+0000 7eff86a00700 1 -- 172.21.15.146:0/1443590678 learned_addr learned my addr 172.21.15.146:0/1443590678 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.246+0000 7eff86a00700 1 -- 172.21.15.146:0/1443590678 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff881098c0 con 0x7eff881086c0 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.246+0000 7eff86a00700 1 --2- 172.21.15.146:0/1443590678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff88108ae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff780098a0 tx=0x7eff78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6baafcefa2a8c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.246+0000 7eff85600700 1 -- 172.21.15.146:0/1443590678 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff78004160 con 0x7eff881086c0 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.246+0000 7eff85600700 1 -- 172.21.15.146:0/1443590678 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff78004300 con 0x7eff881086c0 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.246+0000 7eff85600700 1 -- 172.21.15.146:0/1443590678 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff78011b50 con 0x7eff881086c0 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.247+0000 7eff8d800700 1 -- 172.21.15.146:0/1443590678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 msgr2=0x7eff88108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.247+0000 7eff8d800700 1 --2- 172.21.15.146:0/1443590678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff88108ae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff780098a0 tx=0x7eff78009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.248+0000 7eff8d800700 1 -- 172.21.15.146:0/1443590678 shutdown_connections 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.248+0000 7eff8d800700 1 --2- 172.21.15.146:0/1443590678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff88108ae0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.248+0000 7eff8d800700 1 -- 172.21.15.146:0/1443590678 >> 172.21.15.146:0/1443590678 conn(0x7eff88103cc0 msgr2=0x7eff881060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.248+0000 7eff8d800700 1 -- 172.21.15.146:0/1443590678 shutdown_connections 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.248+0000 7eff8d800700 1 -- 172.21.15.146:0/1443590678 wait complete. 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.249+0000 7eff8d800700 1 Processor -- start 2024-04-24T08:10:07.053 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.249+0000 7eff8d800700 1 -- start start 2024-04-24T08:10:07.054 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.249+0000 7eff8d800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff8819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.054 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.249+0000 7eff8d800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810cc70 con 0x7eff881086c0 2024-04-24T08:10:07.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.249+0000 7eff86a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff8819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.249+0000 7eff86a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff8819c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38460/0 (socket says 172.21.15.146:38460) 2024-04-24T08:10:07.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.249+0000 7eff86a00700 1 -- 172.21.15.146:0/3987288347 learned_addr learned my addr 172.21.15.146:0/3987288347 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:07.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.250+0000 7eff86a00700 1 -- 172.21.15.146:0/3987288347 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8819cd20 con 0x7eff881086c0 2024-04-24T08:10:07.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.250+0000 7eff86a00700 1 --2- 172.21.15.146:0/3987288347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff8819c7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7eff78009fb0 tx=0x7eff78011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:07.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.250+0000 7eff7f400700 1 -- 172.21.15.146:0/3987288347 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff780038b0 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.250+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8819cfb0 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.250+0000 7eff7f400700 1 -- 172.21.15.146:0/3987288347 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff78011e50 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.250+0000 7eff7f400700 1 -- 172.21.15.146:0/3987288347 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff78019550 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.251+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8819dad0 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.251+0000 7eff7f400700 1 -- 172.21.15.146:0/3987288347 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7eff78019be0 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.251+0000 7eff7f400700 1 -- 172.21.15.146:0/3987288347 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7eff78024730 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.252+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff88064150 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.254+0000 7eff7f400700 1 -- 172.21.15.146:0/3987288347 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7eff78003a50 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.335+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7eff88074de0 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.336+0000 7eff7f400700 1 -- 172.21.15.146:0/3987288347 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7eff78024930 con 0x7eff881086c0 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.337+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 msgr2=0x7eff8819c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.337+0000 7eff8d800700 1 --2- 172.21.15.146:0/3987288347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff8819c7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7eff78009fb0 tx=0x7eff78011bf0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.337+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 shutdown_connections 2024-04-24T08:10:07.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.337+0000 7eff8d800700 1 --2- 172.21.15.146:0/3987288347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff881086c0 0x7eff8819c7e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.058 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.337+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 >> 172.21.15.146:0/3987288347 conn(0x7eff88103cc0 msgr2=0x7eff88104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:07.058 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.338+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 shutdown_connections 2024-04-24T08:10:07.058 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:06.338+0000 7eff8d800700 1 -- 172.21.15.146:0/3987288347 wait complete. 2024-04-24T08:10:07.058 INFO:teuthology.orchestra.run.smithi146.stdout:mgr not available, waiting (3/15)... 2024-04-24T08:10:07.354 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:07.071+0000 7f065178f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:10:07.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:07.353+0000 7f065178f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: Activating manager daemon x 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: mgrmap e2: x(active, starting, since 0.010894s) 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:10:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:07 smithi146 ceph-mon[27805]: Manager daemon x is now available 2024-04-24T08:10:08.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:08 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:08.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:08 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' 2024-04-24T08:10:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:08 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:08 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' 2024-04-24T08:10:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:08 smithi146 ceph-mon[27805]: from='mgr.14100 172.21.15.146:0/3741010281' entity='mgr.x' 2024-04-24T08:10:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:08 smithi146 ceph-mon[27805]: mgrmap e3: x(active, since 1.02218s) 2024-04-24T08:10:10.684 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:10 smithi146 ceph-mon[27805]: mgrmap e4: x(active, since 2s) 2024-04-24T08:10:10.684 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:10 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4059571503' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T08:10:10.701 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 0 2024-04-24T08:10:10.703 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:10.704 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T08:10:10.704 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T08:10:10.704 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:10.705 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-24T08:10:10.705 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T08:10:10.705 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:10.705 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T08:10:10.705 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T08:10:10.705 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:10.705 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T08:10:10.706 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:10.706 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T08:10:10.706 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T08:10:10.706 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T08:10:10.706 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T08:10:10.706 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T08:10:10.706 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T08:10:10.714 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:10.714 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T08:10:10.714 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T08:10:10.714 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T08:10:10.714 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T08:10:10.714 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T08:10:10.715 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ], 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T08:09:48.695676+0000", 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout }, 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.631+0000 7f9de8a00700 1 Processor -- start 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.632+0000 7f9de8a00700 1 -- start start 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.632+0000 7f9de8a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de410b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:10.716 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.632+0000 7f9de8a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9de4076370 con 0x7f9de41091b0 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.632+0000 7f9de1e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de410b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.632+0000 7f9de1e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de410b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38524/0 (socket says 172.21.15.146:38524) 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.632+0000 7f9de1e00700 1 -- 172.21.15.146:0/4276004222 learned_addr learned my addr 172.21.15.146:0/4276004222 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.633+0000 7f9de1e00700 1 -- 172.21.15.146:0/4276004222 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de410bae0 con 0x7f9de41091b0 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.633+0000 7f9de1e00700 1 --2- 172.21.15.146:0/4276004222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de410b5a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40098a0 tx=0x7f9dd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127804a64112e03a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de0a00700 1 -- 172.21.15.146:0/4276004222 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd4004160 con 0x7f9de41091b0 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de0a00700 1 -- 172.21.15.146:0/4276004222 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9dd4004300 con 0x7f9de41091b0 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de0a00700 1 -- 172.21.15.146:0/4276004222 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd4011b00 con 0x7f9de41091b0 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de8a00700 1 -- 172.21.15.146:0/4276004222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 msgr2=0x7f9de410b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de8a00700 1 --2- 172.21.15.146:0/4276004222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de410b5a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40098a0 tx=0x7f9dd4009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de8a00700 1 -- 172.21.15.146:0/4276004222 shutdown_connections 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de8a00700 1 --2- 172.21.15.146:0/4276004222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de410b5a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.634+0000 7f9de8a00700 1 -- 172.21.15.146:0/4276004222 >> 172.21.15.146:0/4276004222 conn(0x7f9de4102be0 msgr2=0x7f9de4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.635+0000 7f9de8a00700 1 -- 172.21.15.146:0/4276004222 shutdown_connections 2024-04-24T08:10:10.717 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.635+0000 7f9de8a00700 1 -- 172.21.15.146:0/4276004222 wait complete. 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.636+0000 7f9de8a00700 1 Processor -- start 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.636+0000 7f9de8a00700 1 -- start start 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.636+0000 7f9de8a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de4108c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.636+0000 7f9de8a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9de410caa0 con 0x7f9de41091b0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.637+0000 7f9de1e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de4108c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.637+0000 7f9de1e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de4108c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38536/0 (socket says 172.21.15.146:38536) 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.637+0000 7f9de1e00700 1 -- 172.21.15.146:0/4059571503 learned_addr learned my addr 172.21.15.146:0/4059571503 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.637+0000 7f9de1e00700 1 -- 172.21.15.146:0/4059571503 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de4107160 con 0x7f9de41091b0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.637+0000 7f9de1e00700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de4108c10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4011b60 tx=0x7f9dd4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.638+0000 7f9ddaa00700 1 -- 172.21.15.146:0/4059571503 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd4003b20 con 0x7f9de41091b0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.638+0000 7f9ddaa00700 1 -- 172.21.15.146:0/4059571503 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9dd4003cc0 con 0x7f9de41091b0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.638+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de41073f0 con 0x7f9de41091b0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.638+0000 7f9ddaa00700 1 -- 172.21.15.146:0/4059571503 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd40248a0 con 0x7f9de41091b0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.638+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de41078d0 con 0x7f9de41091b0 2024-04-24T08:10:10.718 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.639+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de4064150 con 0x7f9de41091b0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.639+0000 7f9ddaa00700 1 -- 172.21.15.146:0/4059571503 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f9dd4022020 con 0x7f9de41091b0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.640+0000 7f9ddaa00700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f9dc80395b0 0x7f9dc803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.640+0000 7f9ddaa00700 1 -- 172.21.15.146:0/4059571503 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9dd404d580 con 0x7f9de41091b0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.640+0000 7f9de1400700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f9dc80395b0 0x7f9dc803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.641+0000 7f9de1400700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f9dc80395b0 0x7f9dc803ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0006fd0 tx=0x7f9dd0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.646+0000 7f9ddaa00700 1 -- 172.21.15.146:0/4059571503 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dd400f2e0 con 0x7f9de41091b0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.952+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9de4107e70 con 0x7f9de41091b0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.953+0000 7f9ddaa00700 1 -- 172.21.15.146:0/4059571503 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9dd401a950 con 0x7f9de41091b0 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.956+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f9dc80395b0 msgr2=0x7f9dc803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.956+0000 7f9de8a00700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f9dc80395b0 0x7f9dc803ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0006fd0 tx=0x7f9dd0006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.956+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 msgr2=0x7f9de4108c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.956+0000 7f9de8a00700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de4108c10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4011b60 tx=0x7f9dd4011c40 comp rx=0 tx=0).stop 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.956+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 shutdown_connections 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.956+0000 7f9de8a00700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f9dc80395b0 0x7f9dc803ba70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.957+0000 7f9de8a00700 1 --2- 172.21.15.146:0/4059571503 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9de41091b0 0x7f9de4108c10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.719 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.957+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 >> 172.21.15.146:0/4059571503 conn(0x7f9de4102be0 msgr2=0x7f9de41036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:10.720 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.957+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 shutdown_connections 2024-04-24T08:10:10.720 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:09.957+0000 7f9de8a00700 1 -- 172.21.15.146:0/4059571503 wait complete. 2024-04-24T08:10:10.720 INFO:teuthology.orchestra.run.smithi146.stdout:mgr is available 2024-04-24T08:10:11.965 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:11 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1557780088' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T08:10:12.351 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout fsid = cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T08:10:12.352 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.324+0000 7f8e17200700 1 Processor -- start 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.324+0000 7f8e17200700 1 -- start start 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.325+0000 7f8e17200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1010acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.325+0000 7f8e17200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1010b2a0 con 0x7f8e1010a8b0 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.325+0000 7f8e0fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1010acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.325+0000 7f8e0fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1010acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43200/0 (socket says 172.21.15.146:43200) 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.325+0000 7f8e0fe00700 1 -- 172.21.15.146:0/1052905095 learned_addr learned my addr 172.21.15.146:0/1052905095 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.326+0000 7f8e0fe00700 1 -- 172.21.15.146:0/1052905095 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1010bab0 con 0x7f8e1010a8b0 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.326+0000 7f8e0fe00700 1 --2- 172.21.15.146:0/1052905095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1010acd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e000098a0 tx=0x7f8e00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62cc3d6312a79a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:12.353 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.326+0000 7f8e0ea00700 1 -- 172.21.15.146:0/1052905095 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e00004160 con 0x7f8e1010a8b0 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.327+0000 7f8e0ea00700 1 -- 172.21.15.146:0/1052905095 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e00004300 con 0x7f8e1010a8b0 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.327+0000 7f8e0ea00700 1 -- 172.21.15.146:0/1052905095 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e00011b50 con 0x7f8e1010a8b0 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.327+0000 7f8e17200700 1 -- 172.21.15.146:0/1052905095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 msgr2=0x7f8e1010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.327+0000 7f8e17200700 1 --2- 172.21.15.146:0/1052905095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1010acd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8e000098a0 tx=0x7f8e00009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.327+0000 7f8e17200700 1 -- 172.21.15.146:0/1052905095 shutdown_connections 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.327+0000 7f8e17200700 1 --2- 172.21.15.146:0/1052905095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1010acd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.327+0000 7f8e17200700 1 -- 172.21.15.146:0/1052905095 >> 172.21.15.146:0/1052905095 conn(0x7f8e1007dcc0 msgr2=0x7f8e1007e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.328+0000 7f8e17200700 1 -- 172.21.15.146:0/1052905095 shutdown_connections 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.328+0000 7f8e17200700 1 -- 172.21.15.146:0/1052905095 wait complete. 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.329+0000 7f8e17200700 1 Processor -- start 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.329+0000 7f8e17200700 1 -- start start 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.329+0000 7f8e17200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:12.354 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.329+0000 7f8e17200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1010ee60 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.329+0000 7f8e0fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.329+0000 7f8e0fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1019ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43212/0 (socket says 172.21.15.146:43212) 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.329+0000 7f8e0fe00700 1 -- 172.21.15.146:0/1557780088 learned_addr learned my addr 172.21.15.146:0/1557780088 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.330+0000 7f8e0fe00700 1 -- 172.21.15.146:0/1557780088 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1019ef50 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.330+0000 7f8e0fe00700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1019ea10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e00011850 tx=0x7f8e00011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.330+0000 7f8e0cc00700 1 -- 172.21.15.146:0/1557780088 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e00003af0 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.330+0000 7f8e0cc00700 1 -- 172.21.15.146:0/1557780088 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e00020070 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.330+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1019f1e0 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.330+0000 7f8e0cc00700 1 -- 172.21.15.146:0/1557780088 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e000195a0 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.331+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1019fd00 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.332+0000 7f8e0cc00700 1 -- 172.21.15.146:0/1557780088 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f8e0001a430 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.332+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e10064150 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.332+0000 7f8e0cc00700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f8df4039560 0x7f8df403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.332+0000 7f8e0cc00700 1 -- 172.21.15.146:0/1557780088 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e00011e00 con 0x7f8e1010a8b0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.333+0000 7f8e0f400700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f8df4039560 0x7f8df403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:12.355 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.333+0000 7f8e0f400700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f8df4039560 0x7f8df403ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc006fd0 tx=0x7f8dfc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.338+0000 7f8e0cc00700 1 -- 172.21.15.146:0/1557780088 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e0000e260 con 0x7f8e1010a8b0 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.567+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f8e10074de0 con 0x7f8e1010a8b0 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.569+0000 7f8e0cc00700 1 -- 172.21.15.146:0/1557780088 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+400 (secure 0 0 0) 0x7f8e0004d4d0 con 0x7f8e1010a8b0 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.571+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f8df4039560 msgr2=0x7f8df403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.571+0000 7f8e17200700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f8df4039560 0x7f8df403ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc006fd0 tx=0x7f8dfc006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.571+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 msgr2=0x7f8e1019ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.571+0000 7f8e17200700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1019ea10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e00011850 tx=0x7f8e00011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.572+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 shutdown_connections 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.572+0000 7f8e17200700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f8df4039560 0x7f8df403ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.572+0000 7f8e17200700 1 --2- 172.21.15.146:0/1557780088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1010a8b0 0x7f8e1019ea10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.572+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 >> 172.21.15.146:0/1557780088 conn(0x7f8e1007dcc0 msgr2=0x7f8e101085e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.572+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 shutdown_connections 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:11.572+0000 7f8e17200700 1 -- 172.21.15.146:0/1557780088 wait complete. 2024-04-24T08:10:12.356 INFO:teuthology.orchestra.run.smithi146.stdout:Enabling cephadm module... 2024-04-24T08:10:13.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:13 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3285813063' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-24T08:10:13.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:13 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: ignoring --setuser ceph since I am not root 2024-04-24T08:10:13.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:13 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: ignoring --setgroup ceph since I am not root 2024-04-24T08:10:13.924 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:13 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:13.551+0000 7f4c06a21200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:10:13.925 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:13 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:13.791+0000 7f4c06a21200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:13.925 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:13 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:13.922+0000 7f4c06a21200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:10:13.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.919+0000 7fe227e00700 1 Processor -- start 2024-04-24T08:10:13.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.919+0000 7fe227e00700 1 -- start start 2024-04-24T08:10:13.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.919+0000 7fe227e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22810b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:13.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.919+0000 7fe227e00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe2280763b0 con 0x7fe228108fa0 2024-04-24T08:10:13.949 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.920+0000 7fe226a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22810b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.920+0000 7fe226a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22810b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43214/0 (socket says 172.21.15.146:43214) 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.920+0000 7fe226a00700 1 -- 172.21.15.146:0/2635928146 learned_addr learned my addr 172.21.15.146:0/2635928146 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.920+0000 7fe226a00700 1 -- 172.21.15.146:0/2635928146 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22810b8d0 con 0x7fe228108fa0 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.921+0000 7fe226a00700 1 --2- 172.21.15.146:0/2635928146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22810b390 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe2180098a0 tx=0x7fe218009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64482711e8daeeaa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.921+0000 7fe226000700 1 -- 172.21.15.146:0/2635928146 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe218004160 con 0x7fe228108fa0 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.921+0000 7fe226000700 1 -- 172.21.15.146:0/2635928146 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe218004300 con 0x7fe228108fa0 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.921+0000 7fe226000700 1 -- 172.21.15.146:0/2635928146 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe218011b50 con 0x7fe228108fa0 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.922+0000 7fe227e00700 1 -- 172.21.15.146:0/2635928146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 msgr2=0x7fe22810b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.922+0000 7fe227e00700 1 --2- 172.21.15.146:0/2635928146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22810b390 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe2180098a0 tx=0x7fe218009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.922+0000 7fe227e00700 1 -- 172.21.15.146:0/2635928146 shutdown_connections 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.922+0000 7fe227e00700 1 --2- 172.21.15.146:0/2635928146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22810b390 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.922+0000 7fe227e00700 1 -- 172.21.15.146:0/2635928146 >> 172.21.15.146:0/2635928146 conn(0x7fe2281029d0 msgr2=0x7fe228104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:13.950 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.922+0000 7fe227e00700 1 -- 172.21.15.146:0/2635928146 shutdown_connections 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.922+0000 7fe227e00700 1 -- 172.21.15.146:0/2635928146 wait complete. 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.923+0000 7fe227e00700 1 Processor -- start 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.923+0000 7fe227e00700 1 -- start start 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.923+0000 7fe227e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.923+0000 7fe227e00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe22810c8f0 con 0x7fe228108fa0 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.924+0000 7fe226a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.924+0000 7fe226a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43228/0 (socket says 172.21.15.146:43228) 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.924+0000 7fe226a00700 1 -- 172.21.15.146:0/3285813063 learned_addr learned my addr 172.21.15.146:0/3285813063 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.924+0000 7fe226a00700 1 -- 172.21.15.146:0/3285813063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22819ca60 con 0x7fe228108fa0 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.924+0000 7fe226a00700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22819c520 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe218009fb0 tx=0x7fe218011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.925+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2180038b0 con 0x7fe228108fa0 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.925+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe22819ccf0 con 0x7fe228108fa0 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.925+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe218011e50 con 0x7fe228108fa0 2024-04-24T08:10:13.951 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.925+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe218019420 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.925+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe22819f9e0 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.926+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe228069c60 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.926+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fe218003a50 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.927+0000 7fe21f400700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fe20c039510 0x7fe20c03b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.927+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe21804d930 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.927+0000 7fe21fe00700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fe20c039510 0x7fe20c03b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.928+0000 7fe21fe00700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fe20c039510 0x7fe20c03b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe210006fd0 tx=0x7fe210006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:12.936+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe21804d350 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.223+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fe2281a0020 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.277+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fe21800fda0 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.279+0000 7fe21f400700 1 -- 172.21.15.146:0/3285813063 <== mon.0 v2:172.21.15.146:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fe218023390 con 0x7fe228108fa0 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.280+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fe20c039510 msgr2=0x7fe20c03b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.280+0000 7fe227e00700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fe20c039510 0x7fe20c03b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe210006fd0 tx=0x7fe210006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.280+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 msgr2=0x7fe22819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:13.952 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.280+0000 7fe227e00700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22819c520 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe218009fb0 tx=0x7fe218011bf0 comp rx=0 tx=0).stop 2024-04-24T08:10:13.953 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.281+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 shutdown_connections 2024-04-24T08:10:13.953 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.281+0000 7fe227e00700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fe20c039510 0x7fe20c03b9d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:13.953 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.281+0000 7fe227e00700 1 --2- 172.21.15.146:0/3285813063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe228108fa0 0x7fe22819c520 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:13.953 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.281+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 >> 172.21.15.146:0/3285813063 conn(0x7fe2281029d0 msgr2=0x7fe2281035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:13.953 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.282+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 shutdown_connections 2024-04-24T08:10:13.953 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:13.282+0000 7fe227e00700 1 -- 172.21.15.146:0/3285813063 wait complete. 2024-04-24T08:10:14.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:14 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:14.332+0000 7f4c06a21200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:10:14.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:14 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:14.450+0000 7f4c06a21200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:10:14.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:14 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3285813063' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-24T08:10:14.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:14 smithi146 ceph-mon[27805]: mgrmap e5: x(active, since 5s) 2024-04-24T08:10:15.477 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:15 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4084742829' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T08:10:15.489 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.514+0000 7fd693200700 1 Processor -- start 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.514+0000 7fd693200700 1 -- start start 2024-04-24T08:10:15.496 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.515+0000 7fd693200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c107810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.515+0000 7fd693200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c107de0 con 0x7fd68c1073f0 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.515+0000 7fd691e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c107810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.515+0000 7fd691e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c107810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43256/0 (socket says 172.21.15.146:43256) 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.515+0000 7fd691e00700 1 -- 172.21.15.146:0/2511563157 learned_addr learned my addr 172.21.15.146:0/2511563157 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.516+0000 7fd691e00700 1 -- 172.21.15.146:0/2511563157 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68c1085f0 con 0x7fd68c1073f0 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.516+0000 7fd691e00700 1 --2- 172.21.15.146:0/2511563157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c107810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd68000bb40 tx=0x7fd68000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a3f01edd444e3e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.516+0000 7fd690a00700 1 -- 172.21.15.146:0/2511563157 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd68000dbe0 con 0x7fd68c1073f0 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.516+0000 7fd690a00700 1 -- 172.21.15.146:0/2511563157 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd6800044f0 con 0x7fd68c1073f0 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.516+0000 7fd690a00700 1 -- 172.21.15.146:0/2511563157 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd680010470 con 0x7fd68c1073f0 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.517+0000 7fd693200700 1 -- 172.21.15.146:0/2511563157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 msgr2=0x7fd68c107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.517+0000 7fd693200700 1 --2- 172.21.15.146:0/2511563157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c107810 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd68000bb40 tx=0x7fd68000be50 comp rx=0 tx=0).stop 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.517+0000 7fd693200700 1 -- 172.21.15.146:0/2511563157 shutdown_connections 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.517+0000 7fd693200700 1 --2- 172.21.15.146:0/2511563157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c107810 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:15.497 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.517+0000 7fd693200700 1 -- 172.21.15.146:0/2511563157 >> 172.21.15.146:0/2511563157 conn(0x7fd68c1029d0 msgr2=0x7fd68c104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.517+0000 7fd693200700 1 -- 172.21.15.146:0/2511563157 shutdown_connections 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.518+0000 7fd693200700 1 -- 172.21.15.146:0/2511563157 wait complete. 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.518+0000 7fd693200700 1 Processor -- start 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.519+0000 7fd693200700 1 -- start start 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.519+0000 7fd693200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.519+0000 7fd693200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c075510 con 0x7fd68c1073f0 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.519+0000 7fd691e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.519+0000 7fd691e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c19e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43268/0 (socket says 172.21.15.146:43268) 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.519+0000 7fd691e00700 1 -- 172.21.15.146:0/4084742829 learned_addr learned my addr 172.21.15.146:0/4084742829 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.520+0000 7fd691e00700 1 -- 172.21.15.146:0/4084742829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68c19ed20 con 0x7fd68c1073f0 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.520+0000 7fd691e00700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c19e7e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b340 tx=0x7fd6800040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.520+0000 7fd686a00700 1 -- 172.21.15.146:0/4084742829 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd680010720 con 0x7fd68c1073f0 2024-04-24T08:10:15.498 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.520+0000 7fd686a00700 1 -- 172.21.15.146:0/4084742829 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd68001d070 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.520+0000 7fd686a00700 1 -- 172.21.15.146:0/4084742829 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd680021b80 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.520+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68c19efb0 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.520+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd68c1a1ca0 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.522+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68c069c60 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.522+0000 7fd686a00700 1 -- 172.21.15.146:0/4084742829 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fd68001f030 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.523+0000 7fd686a00700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 0x7fd67403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.523+0000 7fd686a00700 1 -- 172.21.15.146:0/4084742829 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd68004d550 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.523+0000 7fd691400700 1 -- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 msgr2=0x7fd67403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/520435616 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.523+0000 7fd691400700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 0x7fd67403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.528+0000 7fd686a00700 1 -- 172.21.15.146:0/4084742829 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd680015070 con 0x7fd68c1073f0 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.724+0000 7fd691400700 1 -- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 msgr2=0x7fd67403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/520435616 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.724+0000 7fd691400700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 0x7fd67403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:15.499 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.813+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fd68c1a1ec0 con 0x7fd68c1073f0 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.814+0000 7fd686a00700 1 -- 172.21.15.146:0/4084742829 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fd68c1a1ec0 con 0x7fd68c1073f0 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 msgr2=0x7fd67403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 0x7fd67403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 msgr2=0x7fd68c19e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c19e7e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd68000b340 tx=0x7fd6800040b0 comp rx=0 tx=0).stop 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 shutdown_connections 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7fd6740395b0 0x7fd67403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 --2- 172.21.15.146:0/4084742829 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd68c1073f0 0x7fd68c19e7e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.818+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 >> 172.21.15.146:0/4084742829 conn(0x7fd68c1029d0 msgr2=0x7fd68c1909f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.819+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 shutdown_connections 2024-04-24T08:10:15.500 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:14.819+0000 7fd693200700 1 -- 172.21.15.146:0/4084742829 wait complete. 2024-04-24T08:10:15.501 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for the mgr to restart... 2024-04-24T08:10:15.501 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for mgr epoch 5... 2024-04-24T08:10:16.115 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:16 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:16.056+0000 7f4c06a21200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:10:16.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:16 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:16.290+0000 7f4c06a21200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:16.890 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:16 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:16.505+0000 7f4c06a21200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:10:16.890 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:16 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:16.887+0000 7f4c06a21200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:10:17.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:17 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:17.010+0000 7f4c06a21200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:10:17.519 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:17 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:17.262+0000 7f4c06a21200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:10:17.519 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:17 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:17.516+0000 7f4c06a21200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:18.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:18 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:18.299+0000 7f4c06a21200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:10:19.128 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:18 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:18.845+0000 7f4c06a21200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:10:19.128 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:18 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:18.971+0000 7f4c06a21200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:10:19.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:19 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:19.238+0000 7f4c06a21200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:10:20.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:19 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:19.885+0000 7f4c06a21200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:10:20.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:20 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:20.010+0000 7f4c06a21200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:10:20.636 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:20 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:20.330+0000 7f4c06a21200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:10:20.636 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:20 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:20.509+0000 7f4c06a21200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:10:20.636 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:20 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:20.634+0000 7f4c06a21200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:10:20.992 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:20 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:20.784+0000 7f4c06a21200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:21.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:21 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:21.157+0000 7f4c06a21200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:10:22.199 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:21 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:21.910+0000 7f4c06a21200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:10:22.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:22.197+0000 7f4c06a21200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:10:22.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: Active manager daemon x restarted 2024-04-24T08:10:22.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: Activating manager daemon x 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: osdmap e2: 0 total, 0 up, 0 in 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: mgrmap e6: x(active, starting, since 0.00952202s) 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: Manager daemon x is now available 2024-04-24T08:10:22.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:22 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:23.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: Found migration_current of "None". Setting to last migration. 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:23.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:23 smithi146 ceph-mon[27805]: mgrmap e7: x(active, since 1.00958s) 2024-04-24T08:10:23.834 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:10:23.834 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-24T08:10:23.834 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.113+0000 7f25a4a00700 1 Processor -- start 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.113+0000 7f25a4a00700 1 -- start start 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.113+0000 7f25a4a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a0108900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.114+0000 7f25a4a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f25a0108ed0 con 0x7f25a01084e0 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.114+0000 7f259f400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a0108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.114+0000 7f259f400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a0108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43276/0 (socket says 172.21.15.146:43276) 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.114+0000 7f259f400700 1 -- 172.21.15.146:0/3842807229 learned_addr learned my addr 172.21.15.146:0/3842807229 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.115+0000 7f259f400700 1 -- 172.21.15.146:0/3842807229 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a0109690 con 0x7f25a01084e0 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.115+0000 7f259f400700 1 --2- 172.21.15.146:0/3842807229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a0108900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f25900098a0 tx=0x7f2590009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24307022bf62b9e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.115+0000 7f259e000700 1 -- 172.21.15.146:0/3842807229 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2590004160 con 0x7f25a01084e0 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.115+0000 7f259e000700 1 -- 172.21.15.146:0/3842807229 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2590004300 con 0x7f25a01084e0 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.115+0000 7f259e000700 1 -- 172.21.15.146:0/3842807229 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2590011b50 con 0x7f25a01084e0 2024-04-24T08:10:23.835 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.116+0000 7f25a4a00700 1 -- 172.21.15.146:0/3842807229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 msgr2=0x7f25a0108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.116+0000 7f25a4a00700 1 --2- 172.21.15.146:0/3842807229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a0108900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f25900098a0 tx=0x7f2590009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.116+0000 7f25a4a00700 1 -- 172.21.15.146:0/3842807229 shutdown_connections 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.116+0000 7f25a4a00700 1 --2- 172.21.15.146:0/3842807229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a0108900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.116+0000 7f25a4a00700 1 -- 172.21.15.146:0/3842807229 >> 172.21.15.146:0/3842807229 conn(0x7f25a0103b60 msgr2=0x7f25a0105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.116+0000 7f25a4a00700 1 -- 172.21.15.146:0/3842807229 shutdown_connections 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.117+0000 7f25a4a00700 1 -- 172.21.15.146:0/3842807229 wait complete. 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.117+0000 7f25a4a00700 1 Processor -- start 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.118+0000 7f25a4a00700 1 -- start start 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.118+0000 7f25a4a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a007eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.118+0000 7f25a4a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f25a010ca40 con 0x7f25a01084e0 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.118+0000 7f259f400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a007eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.118+0000 7f259f400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a007eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43278/0 (socket says 172.21.15.146:43278) 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.118+0000 7f259f400700 1 -- 172.21.15.146:0/2293229078 learned_addr learned my addr 172.21.15.146:0/2293229078 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.119+0000 7f259f400700 1 -- 172.21.15.146:0/2293229078 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a007f0a0 con 0x7f25a01084e0 2024-04-24T08:10:23.836 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.119+0000 7f259f400700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a007eb60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2590000c00 tx=0x7f2590011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.119+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2590003b20 con 0x7f25a01084e0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.119+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2590003cc0 con 0x7f25a01084e0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.119+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a007d290 con 0x7f25a01084e0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.119+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2590024910 con 0x7f25a01084e0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.119+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a007d7d0 con 0x7f25a01084e0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.121+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f2590022020 con 0x7f25a01084e0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.121+0000 7f2597e00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.121+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 --> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f258403c1d0 con 0x7f2584039600 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.121+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2590011e00 con 0x7f25a01084e0 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.121+0000 7f259ea00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 msgr2=0x7f258403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/520435616 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.121+0000 7f259ea00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.322+0000 7f259ea00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 msgr2=0x7f258403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/520435616 2024-04-24T08:10:23.837 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.322+0000 7f259ea00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.723+0000 7f259ea00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 msgr2=0x7f258403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/520435616 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:16.723+0000 7f259ea00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:17.524+0000 7f259ea00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 msgr2=0x7f258403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/520435616 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:17.524+0000 7f259ea00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:19.126+0000 7f259ea00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 msgr2=0x7f258403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/520435616 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:19.126+0000 7f259ea00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:22.206+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f259001acd0 con 0x7f25a01084e0 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:22.207+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 msgr2=0x7f258403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:22.207+0000 7f2597e00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.208+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f259004eea0 con 0x7f25a01084e0 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.208+0000 7f2597e00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f258403cef0 0x7f258403f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.208+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2590006880 con 0x7f258403cef0 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.210+0000 7f259ea00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f258403cef0 0x7f258403f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.838 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.213+0000 7f259ea00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f258403cef0 0x7f258403f2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f258c003a10 tx=0x7f258c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.214+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f2590006880 con 0x7f258403cef0 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.222+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f25a0069c60 con 0x7f258403cef0 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.222+0000 7f2597e00700 1 -- 172.21.15.146:0/2293229078 <== mgr.14120 v2:172.21.15.146:6800/888738583 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f25a0069c60 con 0x7f258403cef0 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f258403cef0 msgr2=0x7f258403f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f258403cef0 0x7f258403f2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f258c003a10 tx=0x7f258c0092b0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 msgr2=0x7f25a007eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a007eb60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2590000c00 tx=0x7f2590011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 shutdown_connections 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f258403cef0 0x7f258403f2e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25a01084e0 0x7f25a007eb60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 --2- 172.21.15.146:0/2293229078 >> [v2:172.21.15.146:6800/520435616,v1:172.21.15.146:6801/520435616] conn(0x7f2584039600 0x7f258403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 >> 172.21.15.146:0/2293229078 conn(0x7f25a0103b60 msgr2=0x7f25a01046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 shutdown_connections 2024-04-24T08:10:23.839 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:23.223+0000 7f25a4a00700 1 -- 172.21.15.146:0/2293229078 wait complete. 2024-04-24T08:10:23.840 INFO:teuthology.orchestra.run.smithi146.stdout:mgr epoch 5 is available 2024-04-24T08:10:23.840 INFO:teuthology.orchestra.run.smithi146.stdout:Setting orchestrator backend to cephadm... 2024-04-24T08:10:24.411 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:24 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:23] ENGINE Bus STARTING 2024-04-24T08:10:24.411 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:24 smithi146 ceph-mon[27805]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T08:10:24.411 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:24 smithi146 ceph-mon[27805]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T08:10:24.411 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:24 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:23] ENGINE Serving on https://172.21.15.146:7150 2024-04-24T08:10:24.411 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:24 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:23] ENGINE Serving on http://172.21.15.146:8765 2024-04-24T08:10:24.411 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:24 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:23] ENGINE Bus STARTED 2024-04-24T08:10:24.411 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:24 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:25.381 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:25 smithi146 ceph-mon[27805]: mgrmap e8: x(active, since 2s) 2024-04-24T08:10:25.381 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:25 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:25.381 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:25 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:25.403 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.450+0000 7efe9a400700 1 Processor -- start 2024-04-24T08:10:25.407 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.450+0000 7efe9a400700 1 -- start start 2024-04-24T08:10:25.408 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.451+0000 7efe9a400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94108ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.408 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.451+0000 7efe9a400700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efe941090a0 con 0x7efe941086b0 2024-04-24T08:10:25.409 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.451+0000 7efe93400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94108ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.409 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.451+0000 7efe93400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94108ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60100/0 (socket says 172.21.15.146:60100) 2024-04-24T08:10:25.409 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.451+0000 7efe93400700 1 -- 172.21.15.146:0/1014143063 learned_addr learned my addr 172.21.15.146:0/1014143063 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:25.409 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.452+0000 7efe93400700 1 -- 172.21.15.146:0/1014143063 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe941098b0 con 0x7efe941086b0 2024-04-24T08:10:25.410 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.452+0000 7efe93400700 1 --2- 172.21.15.146:0/1014143063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94108ad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe840098a0 tx=0x7efe84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efa870a2785c9f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.410 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.452+0000 7efe92000700 1 -- 172.21.15.146:0/1014143063 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe84004160 con 0x7efe941086b0 2024-04-24T08:10:25.410 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.452+0000 7efe92000700 1 -- 172.21.15.146:0/1014143063 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efe84004300 con 0x7efe941086b0 2024-04-24T08:10:25.410 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.452+0000 7efe92000700 1 -- 172.21.15.146:0/1014143063 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe84011b50 con 0x7efe941086b0 2024-04-24T08:10:25.410 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.453+0000 7efe9a400700 1 -- 172.21.15.146:0/1014143063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 msgr2=0x7efe94108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.410 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.453+0000 7efe9a400700 1 --2- 172.21.15.146:0/1014143063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94108ad0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe840098a0 tx=0x7efe84009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.410 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.453+0000 7efe9a400700 1 -- 172.21.15.146:0/1014143063 shutdown_connections 2024-04-24T08:10:25.411 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.453+0000 7efe9a400700 1 --2- 172.21.15.146:0/1014143063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94108ad0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.411 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.453+0000 7efe9a400700 1 -- 172.21.15.146:0/1014143063 >> 172.21.15.146:0/1014143063 conn(0x7efe94103cd0 msgr2=0x7efe941060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.411 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.453+0000 7efe9a400700 1 -- 172.21.15.146:0/1014143063 shutdown_connections 2024-04-24T08:10:25.412 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.453+0000 7efe9a400700 1 -- 172.21.15.146:0/1014143063 wait complete. 2024-04-24T08:10:25.412 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.454+0000 7efe9a400700 1 Processor -- start 2024-04-24T08:10:25.412 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.455+0000 7efe9a400700 1 -- start start 2024-04-24T08:10:25.412 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.455+0000 7efe9a400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94192230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.412 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.455+0000 7efe9a400700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efe9410cc60 con 0x7efe941086b0 2024-04-24T08:10:25.413 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.455+0000 7efe93400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94192230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.455+0000 7efe93400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94192230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60110/0 (socket says 172.21.15.146:60110) 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.455+0000 7efe93400700 1 -- 172.21.15.146:0/2344936771 learned_addr learned my addr 172.21.15.146:0/2344936771 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.456+0000 7efe93400700 1 -- 172.21.15.146:0/2344936771 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe94192770 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.456+0000 7efe93400700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94192230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe84011850 tx=0x7efe84011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.456+0000 7efe8be00700 1 -- 172.21.15.146:0/2344936771 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe84003af0 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.456+0000 7efe8be00700 1 -- 172.21.15.146:0/2344936771 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efe84020070 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.456+0000 7efe8be00700 1 -- 172.21.15.146:0/2344936771 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe840195a0 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.456+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe94192a00 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.456+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe94193570 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.457+0000 7efe8be00700 1 -- 172.21.15.146:0/2344936771 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7efe8401a430 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.457+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe94064150 con 0x7efe941086b0 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.458+0000 7efe8be00700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efe78039660 0x7efe7803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.414 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.458+0000 7efe8be00700 1 -- 172.21.15.146:0/2344936771 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efe84011e00 con 0x7efe941086b0 2024-04-24T08:10:25.416 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.458+0000 7efe92a00700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efe78039660 0x7efe7803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.416 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.459+0000 7efe92a00700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efe78039660 0x7efe7803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efe80006fd0 tx=0x7efe80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.416 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.464+0000 7efe8be00700 1 -- 172.21.15.146:0/2344936771 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe8400e260 con 0x7efe941086b0 2024-04-24T08:10:25.416 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.713+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7efe94104f50 con 0x7efe78039660 2024-04-24T08:10:25.416 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.724+0000 7efe8be00700 1 -- 172.21.15.146:0/2344936771 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efe94104f50 con 0x7efe78039660 2024-04-24T08:10:25.416 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efe78039660 msgr2=0x7efe7803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.416 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efe78039660 0x7efe7803bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efe80006fd0 tx=0x7efe80006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 msgr2=0x7efe94192230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94192230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efe84011850 tx=0x7efe84011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 shutdown_connections 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efe78039660 0x7efe7803bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 --2- 172.21.15.146:0/2344936771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe941086b0 0x7efe94192230 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.727+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 >> 172.21.15.146:0/2344936771 conn(0x7efe94103cd0 msgr2=0x7efe94104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.728+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 shutdown_connections 2024-04-24T08:10:25.417 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:24.728+0000 7efe9a400700 1 -- 172.21.15.146:0/2344936771 wait complete. 2024-04-24T08:10:26.691 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:26 smithi146 ceph-mon[27805]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.073+0000 7f83a2200700 1 Processor -- start 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.073+0000 7f83a2200700 1 -- start start 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.073+0000 7f83a2200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c10b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.073+0000 7f83a2200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f839c07ce40 con 0x7f839c109170 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.074+0000 7f839b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c10b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.074+0000 7f839b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c10b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60126/0 (socket says 172.21.15.146:60126) 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.074+0000 7f839b400700 1 -- 172.21.15.146:0/674320639 learned_addr learned my addr 172.21.15.146:0/674320639 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:27.039 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.074+0000 7f839b400700 1 -- 172.21.15.146:0/674320639 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c10baa0 con 0x7f839c109170 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.075+0000 7f839b400700 1 --2- 172.21.15.146:0/674320639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c10b560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f838c0098a0 tx=0x7f838c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb9e28180a41f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.075+0000 7f839a000700 1 -- 172.21.15.146:0/674320639 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f838c004160 con 0x7f839c109170 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.075+0000 7f839a000700 1 -- 172.21.15.146:0/674320639 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f838c004300 con 0x7f839c109170 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.075+0000 7f839a000700 1 -- 172.21.15.146:0/674320639 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f838c011b50 con 0x7f839c109170 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.075+0000 7f83a2200700 1 -- 172.21.15.146:0/674320639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 msgr2=0x7f839c10b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.075+0000 7f83a2200700 1 --2- 172.21.15.146:0/674320639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c10b560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f838c0098a0 tx=0x7f838c009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.076+0000 7f83a2200700 1 -- 172.21.15.146:0/674320639 shutdown_connections 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.076+0000 7f83a2200700 1 --2- 172.21.15.146:0/674320639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c10b560 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.076+0000 7f83a2200700 1 -- 172.21.15.146:0/674320639 >> 172.21.15.146:0/674320639 conn(0x7f839c102c00 msgr2=0x7f839c105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.076+0000 7f83a2200700 1 -- 172.21.15.146:0/674320639 shutdown_connections 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.076+0000 7f83a2200700 1 -- 172.21.15.146:0/674320639 wait complete. 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.077+0000 7f83a2200700 1 Processor -- start 2024-04-24T08:10:27.040 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.077+0000 7f83a2200700 1 -- start start 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.077+0000 7f83a2200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.077+0000 7f83a2200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10ca70 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.077+0000 7f839b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f839b400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c19c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60134/0 (socket says 172.21.15.146:60134) 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f839b400700 1 -- 172.21.15.146:0/345151639 learned_addr learned my addr 172.21.15.146:0/345151639 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f839b400700 1 -- 172.21.15.146:0/345151639 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c19cbb0 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f839b400700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c19c670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f838c011b60 tx=0x7f838c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f8393e00700 1 -- 172.21.15.146:0/345151639 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f838c003970 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f8393e00700 1 -- 172.21.15.146:0/345151639 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f838c003b10 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c19ce40 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.078+0000 7f8393e00700 1 -- 172.21.15.146:0/345151639 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f838c024610 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.079+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c19fb30 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.080+0000 7f8393e00700 1 -- 172.21.15.146:0/345151639 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f838c01a430 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.080+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839c069c60 con 0x7f839c109170 2024-04-24T08:10:27.041 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.080+0000 7f8393e00700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f8380039600 0x7f838003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.080+0000 7f8393e00700 1 -- 172.21.15.146:0/345151639 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f838c011e00 con 0x7f839c109170 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.081+0000 7f839aa00700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f8380039600 0x7f838003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.081+0000 7f839aa00700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f8380039600 0x7f838003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8388006fd0 tx=0x7f8388006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.087+0000 7f8393e00700 1 -- 172.21.15.146:0/345151639 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f838c016260 con 0x7f839c109170 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.319+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f839c103e80 con 0x7f8380039600 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.321+0000 7f8393e00700 1 -- 172.21.15.146:0/345151639 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f839c103e80 con 0x7f8380039600 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.323+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f8380039600 msgr2=0x7f838003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.323+0000 7f83a2200700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f8380039600 0x7f838003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8388006fd0 tx=0x7f8388006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.323+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 msgr2=0x7f839c19c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.323+0000 7f83a2200700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c19c670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f838c011b60 tx=0x7f838c011c40 comp rx=0 tx=0).stop 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.324+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 shutdown_connections 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.324+0000 7f83a2200700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f8380039600 0x7f838003bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.042 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.324+0000 7f83a2200700 1 --2- 172.21.15.146:0/345151639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c109170 0x7f839c19c670 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.043 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.324+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 >> 172.21.15.146:0/345151639 conn(0x7f839c102c00 msgr2=0x7f839c103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.043 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.324+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 shutdown_connections 2024-04-24T08:10:27.043 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:26.324+0000 7f83a2200700 1 -- 172.21.15.146:0/345151639 wait complete. 2024-04-24T08:10:27.043 INFO:teuthology.orchestra.run.smithi146.stdout:Generating ssh key... 2024-04-24T08:10:27.536 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:27 smithi146 ceph-mon[27805]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:28.243 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: Generating public/private rsa key pair. 2024-04-24T08:10:28.243 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: Your identification has been saved in /tmp/tmpoacmtise/key. 2024-04-24T08:10:28.243 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: Your public key has been saved in /tmp/tmpoacmtise/key.pub. 2024-04-24T08:10:28.243 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: The key fingerprint is: 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: SHA256:TEEbPEIkG121BW3SeynMzprQaZP5MgvgezMJK5Smn5k ceph-cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: The key's randomart image is: 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: +---[RSA 3072]----+ 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | o+o+=.o+. | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | +o o+.o+ | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | . .o..= . . | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | o = o | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | .. S. * o | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | +.... B o | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | + .o.+ = | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | . .+..=.= . | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: | .E... o.+ | 2024-04-24T08:10:28.244 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: +----[SHA256]-----+ 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.686+0000 7fa687200700 1 Processor -- start 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.686+0000 7fa687200700 1 -- start start 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.687+0000 7fa687200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68010acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.687+0000 7fa687200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa68010b2a0 con 0x7fa68010a8b0 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.687+0000 7fa67fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68010acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.687+0000 7fa67fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68010acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60140/0 (socket says 172.21.15.146:60140) 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.687+0000 7fa67fe00700 1 -- 172.21.15.146:0/1583098448 learned_addr learned my addr 172.21.15.146:0/1583098448 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.688+0000 7fa67fe00700 1 -- 172.21.15.146:0/1583098448 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68010bab0 con 0x7fa68010a8b0 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.688+0000 7fa67fe00700 1 --2- 172.21.15.146:0/1583098448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68010acd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6700098a0 tx=0x7fa670009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce9963423661e3d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.840 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.688+0000 7fa67ea00700 1 -- 172.21.15.146:0/1583098448 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa670004160 con 0x7fa68010a8b0 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.688+0000 7fa67ea00700 1 -- 172.21.15.146:0/1583098448 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa670004300 con 0x7fa68010a8b0 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.688+0000 7fa67ea00700 1 -- 172.21.15.146:0/1583098448 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa670011b50 con 0x7fa68010a8b0 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.689+0000 7fa687200700 1 -- 172.21.15.146:0/1583098448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 msgr2=0x7fa68010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.689+0000 7fa687200700 1 --2- 172.21.15.146:0/1583098448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68010acd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6700098a0 tx=0x7fa670009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.689+0000 7fa687200700 1 -- 172.21.15.146:0/1583098448 shutdown_connections 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.689+0000 7fa687200700 1 --2- 172.21.15.146:0/1583098448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68010acd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.689+0000 7fa687200700 1 -- 172.21.15.146:0/1583098448 >> 172.21.15.146:0/1583098448 conn(0x7fa68007dcc0 msgr2=0x7fa68007e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.689+0000 7fa687200700 1 -- 172.21.15.146:0/1583098448 shutdown_connections 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.689+0000 7fa687200700 1 -- 172.21.15.146:0/1583098448 wait complete. 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.690+0000 7fa687200700 1 Processor -- start 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.690+0000 7fa687200700 1 -- start start 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.691+0000 7fa687200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.691+0000 7fa687200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa68010ee60 con 0x7fa68010a8b0 2024-04-24T08:10:28.841 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.691+0000 7fa67fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.691+0000 7fa67fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68019ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60156/0 (socket says 172.21.15.146:60156) 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.691+0000 7fa67fe00700 1 -- 172.21.15.146:0/3389287119 learned_addr learned my addr 172.21.15.146:0/3389287119 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.691+0000 7fa67fe00700 1 -- 172.21.15.146:0/3389287119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68019ef50 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.692+0000 7fa67fe00700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68019ea10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa670011850 tx=0x7fa670011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.692+0000 7fa67cc00700 1 -- 172.21.15.146:0/3389287119 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa670003af0 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.692+0000 7fa67cc00700 1 -- 172.21.15.146:0/3389287119 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa670020070 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.692+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa68019f1e0 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.692+0000 7fa67cc00700 1 -- 172.21.15.146:0/3389287119 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6700195a0 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.692+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa68019fd00 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.693+0000 7fa67cc00700 1 -- 172.21.15.146:0/3389287119 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa6700198a0 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.693+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa680064150 con 0x7fa68010a8b0 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.694+0000 7fa67cc00700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7fa6640395b0 0x7fa66403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.842 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.694+0000 7fa67cc00700 1 -- 172.21.15.146:0/3389287119 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa670011e00 con 0x7fa68010a8b0 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.694+0000 7fa67f400700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7fa6640395b0 0x7fa66403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.695+0000 7fa67f400700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7fa6640395b0 0x7fa66403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006fd0 tx=0x7fa66c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.699+0000 7fa67cc00700 1 -- 172.21.15.146:0/3389287119 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa67000e260 con 0x7fa68010a8b0 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:27.925+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fa680108cf0 con 0x7fa6640395b0 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.080+0000 7fa67cc00700 1 -- 172.21.15.146:0/3389287119 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa680108cf0 con 0x7fa6640395b0 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.083+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7fa6640395b0 msgr2=0x7fa66403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7fa6640395b0 0x7fa66403ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006fd0 tx=0x7fa66c006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 msgr2=0x7fa68019ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68019ea10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa670011850 tx=0x7fa670011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 shutdown_connections 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7fa6640395b0 0x7fa66403ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 --2- 172.21.15.146:0/3389287119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa68010a8b0 0x7fa68019ea10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 >> 172.21.15.146:0/3389287119 conn(0x7fa68007dcc0 msgr2=0x7fa6801085e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.843 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 shutdown_connections 2024-04-24T08:10:28.844 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:28.084+0000 7fa687200700 1 -- 172.21.15.146:0/3389287119 wait complete. 2024-04-24T08:10:29.119 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:29 smithi146 ceph-mon[27805]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:29.119 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:29 smithi146 ceph-mon[27805]: Generating ssh key... 2024-04-24T08:10:29.119 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:29 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:29.119 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:29 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:30.262 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyUjgvPYjaBtkv5xswb5eTWRslbu3sD5yRgc7TCEAyiitMZQEyMfbopGWgpuHqJ5u1AS5yOKtx0w7/F7okbmtwfpj7e/PuNpzQu0TnqcqBgY5hYi0JxdCdnxoZCA+VxoAWHqzpdQ8uzGzZMnUNJz/X1rYUkMq5DCF26iOubp+8y+tLV/6cYqXsqhJcpK5f3bQ4zfkDfVYhQ5m4AklU8CGkhpHnsiOk4+LAWeozVRMNAv5CpzGVw4mxFDi5/AVrbsUhR8WTUO+nI3bZWaCo7tXVF93Zk0dyC+4iQn+jGcybYRS27GD4r/tsHAdGQf4ENHc/3BLdVEc5k3AQfMv2iW3PyOHMFG77kPXW5Es9stt+hrFc4bNeOj32wWPDmQLfR0PBT6xskJeem8+ePtXeNsnQBxwGmHb5gTHqOxLYjBQLBiH7C99ezAMMuJS6p8ysY/mF60z/rd4t+9511lSitU/cqAPZsxo/TTpX7jLYjI6e8Kpv+tep+2FHH2vP6Sow5p8= ceph-cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:10:30.262 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.428+0000 7efecd000700 1 Processor -- start 2024-04-24T08:10:30.262 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.428+0000 7efecd000700 1 -- start start 2024-04-24T08:10:30.262 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.429+0000 7efecd000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec8082ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.429+0000 7efecd000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efec8083080 con 0x7efec8082690 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.429+0000 7efec5e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec8082ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.429+0000 7efec5e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec8082ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60168/0 (socket says 172.21.15.146:60168) 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.429+0000 7efec5e00700 1 -- 172.21.15.146:0/4155553373 learned_addr learned my addr 172.21.15.146:0/4155553373 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.430+0000 7efec5e00700 1 -- 172.21.15.146:0/4155553373 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec810fed0 con 0x7efec8082690 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.430+0000 7efec5e00700 1 --2- 172.21.15.146:0/4155553373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec8082ab0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efeb80098a0 tx=0x7efeb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b0bb0f0ccf1322 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.430+0000 7efec4a00700 1 -- 172.21.15.146:0/4155553373 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efeb8004160 con 0x7efec8082690 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.430+0000 7efec4a00700 1 -- 172.21.15.146:0/4155553373 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efeb8004300 con 0x7efec8082690 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.430+0000 7efec4a00700 1 -- 172.21.15.146:0/4155553373 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efeb8011b50 con 0x7efec8082690 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.431+0000 7efecd000700 1 -- 172.21.15.146:0/4155553373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 msgr2=0x7efec8082ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.431+0000 7efecd000700 1 --2- 172.21.15.146:0/4155553373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec8082ab0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efeb80098a0 tx=0x7efeb8009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.431+0000 7efecd000700 1 -- 172.21.15.146:0/4155553373 shutdown_connections 2024-04-24T08:10:30.263 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.431+0000 7efecd000700 1 --2- 172.21.15.146:0/4155553373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec8082ab0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.431+0000 7efecd000700 1 -- 172.21.15.146:0/4155553373 >> 172.21.15.146:0/4155553373 conn(0x7efec807dcc0 msgr2=0x7efec80800e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.432+0000 7efecd000700 1 -- 172.21.15.146:0/4155553373 shutdown_connections 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.432+0000 7efecd000700 1 -- 172.21.15.146:0/4155553373 wait complete. 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.433+0000 7efecd000700 1 Processor -- start 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.433+0000 7efecd000700 1 -- start start 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.433+0000 7efecd000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec81a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.433+0000 7efecd000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efec8113280 con 0x7efec8082690 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.433+0000 7efec5e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec81a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.433+0000 7efec5e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec81a2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60174/0 (socket says 172.21.15.146:60174) 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.433+0000 7efec5e00700 1 -- 172.21.15.146:0/32394451 learned_addr learned my addr 172.21.15.146:0/32394451 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.434+0000 7efec5e00700 1 -- 172.21.15.146:0/32394451 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec81a3350 con 0x7efec8082690 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.434+0000 7efec5e00700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec81a2e10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efeb8011b60 tx=0x7efeb8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.264 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.434+0000 7efebea00700 1 -- 172.21.15.146:0/32394451 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efeb8003970 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.434+0000 7efebea00700 1 -- 172.21.15.146:0/32394451 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efeb8003b10 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.434+0000 7efebea00700 1 -- 172.21.15.146:0/32394451 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efeb8024610 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.434+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec81a35e0 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.434+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec81a4050 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.435+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec8064150 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.436+0000 7efebea00700 1 -- 172.21.15.146:0/32394451 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7efeb801a9f0 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.436+0000 7efebea00700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efeac039600 0x7efeac03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.436+0000 7efebea00700 1 -- 172.21.15.146:0/32394451 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efeb8011e00 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.436+0000 7efec5400700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efeac039600 0x7efeac03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.437+0000 7efec5400700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efeac039600 0x7efeac03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efeb4006fd0 tx=0x7efeb4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.441+0000 7efebea00700 1 -- 172.21.15.146:0/32394451 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efeb8016260 con 0x7efec8082690 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.667+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7efec807ef60 con 0x7efeac039600 2024-04-24T08:10:30.265 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.669+0000 7efebea00700 1 -- 172.21.15.146:0/32394451 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7efec807ef60 con 0x7efeac039600 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efeac039600 msgr2=0x7efeac03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efeac039600 0x7efeac03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efeb4006fd0 tx=0x7efeb4006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 msgr2=0x7efec81a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec81a2e10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efeb8011b60 tx=0x7efeb8011c40 comp rx=0 tx=0).stop 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 shutdown_connections 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7efeac039600 0x7efeac03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 --2- 172.21.15.146:0/32394451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec8082690 0x7efec81a2e10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.672+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 >> 172.21.15.146:0/32394451 conn(0x7efec807dcc0 msgr2=0x7efec807e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.673+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 shutdown_connections 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:29.673+0000 7efecd000700 1 -- 172.21.15.146:0/32394451 wait complete. 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-24T08:10:30.266 INFO:teuthology.orchestra.run.smithi146.stdout:Adding key to root@localhost authorized_keys... 2024-04-24T08:10:30.270 INFO:teuthology.orchestra.run.smithi146.stdout:Adding host smithi146... 2024-04-24T08:10:30.598 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:30 smithi146 ceph-mon[27805]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:31 smithi146 ceph-mon[27805]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "addr": "172.21.15.146", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:32.677 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:32 smithi146 ceph-mon[27805]: Deploying cephadm binary to smithi146 2024-04-24T08:10:37.848 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout Added host 'smithi146' with addr '172.21.15.146' 2024-04-24T08:10:37.848 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.898+0000 7f4e3dc00700 1 Processor -- start 2024-04-24T08:10:37.848 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.898+0000 7f4e3dc00700 1 -- start start 2024-04-24T08:10:37.848 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.899+0000 7f4e3dc00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e380828e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:37.848 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.899+0000 7f4e3dc00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4e38082eb0 con 0x7f4e380824c0 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.899+0000 7f4e37e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e380828e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.899+0000 7f4e37e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e380828e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60182/0 (socket says 172.21.15.146:60182) 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.899+0000 7f4e37e00700 1 -- 172.21.15.146:0/2177169245 learned_addr learned my addr 172.21.15.146:0/2177169245 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.900+0000 7f4e37e00700 1 -- 172.21.15.146:0/2177169245 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3810fd60 con 0x7f4e380824c0 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.900+0000 7f4e37e00700 1 --2- 172.21.15.146:0/2177169245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e380828e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e280098a0 tx=0x7f4e28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0d4509b22c2a65 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.901+0000 7f4e36a00700 1 -- 172.21.15.146:0/2177169245 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e28004160 con 0x7f4e380824c0 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.901+0000 7f4e36a00700 1 -- 172.21.15.146:0/2177169245 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4e28004300 con 0x7f4e380824c0 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.901+0000 7f4e36a00700 1 -- 172.21.15.146:0/2177169245 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e28011b50 con 0x7f4e380824c0 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.901+0000 7f4e3dc00700 1 -- 172.21.15.146:0/2177169245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 msgr2=0x7f4e380828e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.901+0000 7f4e3dc00700 1 --2- 172.21.15.146:0/2177169245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e380828e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4e280098a0 tx=0x7f4e28009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.902+0000 7f4e3dc00700 1 -- 172.21.15.146:0/2177169245 shutdown_connections 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.902+0000 7f4e3dc00700 1 --2- 172.21.15.146:0/2177169245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e380828e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.849 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.902+0000 7f4e3dc00700 1 -- 172.21.15.146:0/2177169245 >> 172.21.15.146:0/2177169245 conn(0x7f4e3807db70 msgr2=0x7f4e3807ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:37.853 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.902+0000 7f4e3dc00700 1 -- 172.21.15.146:0/2177169245 shutdown_connections 2024-04-24T08:10:37.853 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.902+0000 7f4e3dc00700 1 -- 172.21.15.146:0/2177169245 wait complete. 2024-04-24T08:10:37.853 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.903+0000 7f4e3dc00700 1 Processor -- start 2024-04-24T08:10:37.853 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.903+0000 7f4e3dc00700 1 -- start start 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.903+0000 7f4e3dc00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e381a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.903+0000 7f4e3dc00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4e38113110 con 0x7f4e380824c0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.903+0000 7f4e37e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e381a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.904+0000 7f4e37e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e381a2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60184/0 (socket says 172.21.15.146:60184) 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.904+0000 7f4e37e00700 1 -- 172.21.15.146:0/3671586818 learned_addr learned my addr 172.21.15.146:0/3671586818 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.904+0000 7f4e37e00700 1 -- 172.21.15.146:0/3671586818 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e381a3050 con 0x7f4e380824c0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.904+0000 7f4e37e00700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e381a2b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e28011850 tx=0x7f4e28011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.904+0000 7f4e34c00700 1 -- 172.21.15.146:0/3671586818 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e28003af0 con 0x7f4e380824c0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.904+0000 7f4e34c00700 1 -- 172.21.15.146:0/3671586818 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4e28020070 con 0x7f4e380824c0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.904+0000 7f4e34c00700 1 -- 172.21.15.146:0/3671586818 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e280195a0 con 0x7f4e380824c0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.905+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e381a32e0 con 0x7f4e380824c0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.905+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e381a5fd0 con 0x7f4e380824c0 2024-04-24T08:10:37.854 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.906+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e38069c60 con 0x7f4e380824c0 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.906+0000 7f4e34c00700 1 -- 172.21.15.146:0/3671586818 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4e2801a430 con 0x7f4e380824c0 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.907+0000 7f4e34c00700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f4e1c039600 0x7f4e1c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.907+0000 7f4e34c00700 1 -- 172.21.15.146:0/3671586818 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4e28011e00 con 0x7f4e380824c0 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.907+0000 7f4e37400700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f4e1c039600 0x7f4e1c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.908+0000 7f4e37400700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f4e1c039600 0x7f4e1c03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4e24006fd0 tx=0x7f4e24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:30.913+0000 7f4e34c00700 1 -- 172.21.15.146:0/3671586818 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e2802a030 con 0x7f4e380824c0 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:31.142+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi146", "addr": "172.21.15.146", "target": ["mon-mgr", ""]}) -- 0x7f4e3807edf0 con 0x7f4e1c039600 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.140+0000 7f4e34c00700 1 -- 172.21.15.146:0/3671586818 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4e3807edf0 con 0x7f4e1c039600 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f4e1c039600 msgr2=0x7f4e1c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f4e1c039600 0x7f4e1c03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4e24006fd0 tx=0x7f4e24006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 msgr2=0x7f4e381a2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e381a2b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e28011850 tx=0x7f4e28011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 shutdown_connections 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f4e1c039600 0x7f4e1c03bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.855 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 --2- 172.21.15.146:0/3671586818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e380824c0 0x7f4e381a2b10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.856 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 >> 172.21.15.146:0/3671586818 conn(0x7f4e3807db70 msgr2=0x7f4e3807e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:37.856 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 shutdown_connections 2024-04-24T08:10:37.856 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:37.144+0000 7f4e3dc00700 1 -- 172.21.15.146:0/3671586818 wait complete. 2024-04-24T08:10:37.856 INFO:teuthology.orchestra.run.smithi146.stdout:Deploying unmanaged mon service... 2024-04-24T08:10:38.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:38 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:38.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:38 smithi146 ceph-mon[27805]: Added host smithi146 2024-04-24T08:10:38.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:38 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:39.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:39 smithi146 ceph-mon[27805]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:39.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:39 smithi146 ceph-mon[27805]: Saving service mon spec with placement count:5 2024-04-24T08:10:39.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:39 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.528+0000 7f51af200700 1 Processor -- start 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.528+0000 7f51af200700 1 -- start start 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.528+0000 7f51af200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b00824c0 0x7f51b00828e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.528+0000 7f51af200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f51b0082eb0 con 0x7f51b00824c0 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.529+0000 7f51ade00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b00824c0 0x7f51b00828e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.529+0000 7f51ade00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b00824c0 0x7f51b00828e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47184/0 (socket says 172.21.15.146:47184) 2024-04-24T08:10:40.574 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.529+0000 7f51ade00700 1 -- 172.21.15.146:0/1958309721 learned_addr learned my addr 172.21.15.146:0/1958309721 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:40.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.529+0000 7f51ade00700 1 -- 172.21.15.146:0/1958309721 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b010fd60 con 0x7f51b00824c0 2024-04-24T08:10:40.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.530+0000 7f51ade00700 1 --2- 172.21.15.146:0/1958309721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b00824c0 0x7f51b00828e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f51a00098a0 tx=0x7f51a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4c7bf39d8ac1b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.530+0000 7f51aca00700 1 -- 172.21.15.146:0/1958309721 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51a0004160 con 0x7f51b00824c0 2024-04-24T08:10:40.579 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.530+0000 7f51aca00700 1 -- 172.21.15.146:0/1958309721 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f51a0004300 con 0x7f51b00824c0 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.530+0000 7f51aca00700 1 -- 172.21.15.146:0/1958309721 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51a0011b50 con 0x7f51b00824c0 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.531+0000 7f51af200700 1 -- 172.21.15.146:0/1958309721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b00824c0 msgr2=0x7f51b00828e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.531+0000 7f51af200700 1 --2- 172.21.15.146:0/1958309721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b00824c0 0x7f51b00828e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f51a00098a0 tx=0x7f51a0009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.531+0000 7f51af200700 1 -- 172.21.15.146:0/1958309721 shutdown_connections 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.531+0000 7f51af200700 1 --2- 172.21.15.146:0/1958309721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b00824c0 0x7f51b00828e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.531+0000 7f51af200700 1 -- 172.21.15.146:0/1958309721 >> 172.21.15.146:0/1958309721 conn(0x7f51b007db70 msgr2=0x7f51b007ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.531+0000 7f51af200700 1 -- 172.21.15.146:0/1958309721 shutdown_connections 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.531+0000 7f51af200700 1 -- 172.21.15.146:0/1958309721 wait complete. 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.532+0000 7f51af200700 1 Processor -- start 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.532+0000 7f51af200700 1 -- start start 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.533+0000 7f51af200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b01a2d40 0x7f51b01a3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.533+0000 7f51af200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f51b0113110 con 0x7f51b01a2d40 2024-04-24T08:10:40.580 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.533+0000 7f51ade00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b01a2d40 0x7f51b01a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.533+0000 7f51ade00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b01a2d40 0x7f51b01a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47188/0 (socket says 172.21.15.146:47188) 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.533+0000 7f51ade00700 1 -- 172.21.15.146:0/772644860 learned_addr learned my addr 172.21.15.146:0/772644860 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.533+0000 7f51ade00700 1 -- 172.21.15.146:0/772644860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b01a36c0 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.533+0000 7f51ade00700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b01a2d40 0x7f51b01a3180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f51a0000c00 tx=0x7f51a00119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.534+0000 7f51a6a00700 1 -- 172.21.15.146:0/772644860 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51a00038b0 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.534+0000 7f51a6a00700 1 -- 172.21.15.146:0/772644860 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f51a0003a50 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.534+0000 7f51a6a00700 1 -- 172.21.15.146:0/772644860 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51a001b440 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.534+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b01a3950 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.534+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b01a63f0 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.535+0000 7f51a6a00700 1 -- 172.21.15.146:0/772644860 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f51a001a440 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.535+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b0069c60 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.535+0000 7f51a6a00700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f5194039600 0x7f519403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.535+0000 7f51a6a00700 1 -- 172.21.15.146:0/772644860 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f51a004dac0 con 0x7f51b01a2d40 2024-04-24T08:10:40.581 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.536+0000 7f51ad400700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f5194039600 0x7f519403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.536+0000 7f51ad400700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f5194039600 0x7f519403bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f519c006fd0 tx=0x7f519c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.541+0000 7f51a6a00700 1 -- 172.21.15.146:0/772644860 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51a001a730 con 0x7f51b01a2d40 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.796+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f51b007ee40 con 0x7f5194039600 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.805+0000 7f51a6a00700 1 -- 172.21.15.146:0/772644860 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f51b007ee40 con 0x7f5194039600 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f5194039600 msgr2=0x7f519403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f5194039600 0x7f519403bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f519c006fd0 tx=0x7f519c006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b01a2d40 msgr2=0x7f51b01a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b01a2d40 0x7f51b01a3180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f51a0000c00 tx=0x7f51a00119e0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 shutdown_connections 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f5194039600 0x7f519403bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 --2- 172.21.15.146:0/772644860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51b01a2d40 0x7f51b01a3180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.582 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 >> 172.21.15.146:0/772644860 conn(0x7f51b007db70 msgr2=0x7f51b007e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:40.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.808+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 shutdown_connections 2024-04-24T08:10:40.583 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:38.809+0000 7f51af200700 1 -- 172.21.15.146:0/772644860 wait complete. 2024-04-24T08:10:40.583 INFO:teuthology.orchestra.run.smithi146.stdout:Deploying unmanaged mgr service... 2024-04-24T08:10:41.393 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:41 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:41.394 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:41 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:42.067 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-24T08:10:42.067 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.134+0000 7f65f0800700 1 Processor -- start 2024-04-24T08:10:42.067 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.134+0000 7f65f0800700 1 -- start start 2024-04-24T08:10:42.067 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.134+0000 7f65f0800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e8108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.067 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.134+0000 7f65f0800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f65e81090b0 con 0x7f65e81086c0 2024-04-24T08:10:42.067 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.135+0000 7f65ede00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e8108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.067 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.135+0000 7f65ede00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e8108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48076/0 (socket says 172.21.15.146:48076) 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.135+0000 7f65ede00700 1 -- 172.21.15.146:0/3074233401 learned_addr learned my addr 172.21.15.146:0/3074233401 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.135+0000 7f65ede00700 1 -- 172.21.15.146:0/3074233401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e81098c0 con 0x7f65e81086c0 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.136+0000 7f65ede00700 1 --2- 172.21.15.146:0/3074233401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e8108ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0098a0 tx=0x7f65dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70fa77dc53be375f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.136+0000 7f65eca00700 1 -- 172.21.15.146:0/3074233401 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65dc004160 con 0x7f65e81086c0 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.136+0000 7f65eca00700 1 -- 172.21.15.146:0/3074233401 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65dc004300 con 0x7f65e81086c0 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.136+0000 7f65eca00700 1 -- 172.21.15.146:0/3074233401 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65dc011b20 con 0x7f65e81086c0 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.137+0000 7f65f0800700 1 -- 172.21.15.146:0/3074233401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 msgr2=0x7f65e8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.137+0000 7f65f0800700 1 --2- 172.21.15.146:0/3074233401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e8108ae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0098a0 tx=0x7f65dc009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.137+0000 7f65f0800700 1 -- 172.21.15.146:0/3074233401 shutdown_connections 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.137+0000 7f65f0800700 1 --2- 172.21.15.146:0/3074233401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e8108ae0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.137+0000 7f65f0800700 1 -- 172.21.15.146:0/3074233401 >> 172.21.15.146:0/3074233401 conn(0x7f65e8103cc0 msgr2=0x7f65e81060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.138+0000 7f65f0800700 1 -- 172.21.15.146:0/3074233401 shutdown_connections 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.138+0000 7f65f0800700 1 -- 172.21.15.146:0/3074233401 wait complete. 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.139+0000 7f65f0800700 1 Processor -- start 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.139+0000 7f65f0800700 1 -- start start 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.139+0000 7f65f0800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.140+0000 7f65f0800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f65e810cc70 con 0x7f65e81086c0 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.140+0000 7f65ede00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.140+0000 7f65ede00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e819c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48090/0 (socket says 172.21.15.146:48090) 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.140+0000 7f65ede00700 1 -- 172.21.15.146:0/471327270 learned_addr learned my addr 172.21.15.146:0/471327270 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.140+0000 7f65ede00700 1 -- 172.21.15.146:0/471327270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e819ccd0 con 0x7f65e81086c0 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.141+0000 7f65ede00700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e819c790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f65dc000c00 tx=0x7f65dc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.141+0000 7f65e2a00700 1 -- 172.21.15.146:0/471327270 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65dc003b60 con 0x7f65e81086c0 2024-04-24T08:10:42.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.141+0000 7f65e2a00700 1 -- 172.21.15.146:0/471327270 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65dc01a910 con 0x7f65e81086c0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.141+0000 7f65e2a00700 1 -- 172.21.15.146:0/471327270 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65dc024930 con 0x7f65e81086c0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.141+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e819cf60 con 0x7f65e81086c0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.141+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e819da80 con 0x7f65e81086c0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.142+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e8064150 con 0x7f65e81086c0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.142+0000 7f65e2a00700 1 -- 172.21.15.146:0/471327270 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f65dc022020 con 0x7f65e81086c0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.143+0000 7f65e2a00700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f65d0039600 0x7f65d003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.143+0000 7f65e2a00700 1 -- 172.21.15.146:0/471327270 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f65dc04d9b0 con 0x7f65e81086c0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.144+0000 7f65ed400700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f65d0039600 0x7f65d003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.144+0000 7f65ed400700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f65d0039600 0x7f65d003bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f65d8006fd0 tx=0x7f65d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.070 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.149+0000 7f65e2a00700 1 -- 172.21.15.146:0/471327270 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65dc00b3e0 con 0x7f65e81086c0 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.393+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f65e8104f30 con 0x7f65d0039600 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.400+0000 7f65e2a00700 1 -- 172.21.15.146:0/471327270 <== mgr.14120 v2:172.21.15.146:6800/888738583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f65e8104f30 con 0x7f65d0039600 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f65d0039600 msgr2=0x7f65d003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f65d0039600 0x7f65d003bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f65d8006fd0 tx=0x7f65d8006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 msgr2=0x7f65e819c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e819c790 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f65dc000c00 tx=0x7f65dc011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 shutdown_connections 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f65d0039600 0x7f65d003bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 --2- 172.21.15.146:0/471327270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65e81086c0 0x7f65e819c790 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.403+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 >> 172.21.15.146:0/471327270 conn(0x7f65e8103cc0 msgr2=0x7f65e8104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.404+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 shutdown_connections 2024-04-24T08:10:42.071 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:41.404+0000 7f65f0800700 1 -- 172.21.15.146:0/471327270 wait complete. 2024-04-24T08:10:42.405 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:42 smithi146 ceph-mon[27805]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:42.405 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:42 smithi146 ceph-mon[27805]: Saving service mgr spec with placement count:2 2024-04-24T08:10:42.405 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:42 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:43.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:43 smithi146 ceph-mon[27805]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:10:43.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:43 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2275963357' entity='client.admin' 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.681+0000 7f1c4d600700 1 Processor -- start 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.681+0000 7f1c4d600700 1 -- start start 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.682+0000 7f1c4d600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4810b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.682+0000 7f1c4d600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c480763f0 con 0x7f1c48108f60 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.682+0000 7f1c47e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4810b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.682+0000 7f1c47e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4810b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48094/0 (socket says 172.21.15.146:48094) 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.683+0000 7f1c47e00700 1 -- 172.21.15.146:0/3654395712 learned_addr learned my addr 172.21.15.146:0/3654395712 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.683+0000 7f1c47e00700 1 -- 172.21.15.146:0/3654395712 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4810b890 con 0x7f1c48108f60 2024-04-24T08:10:44.099 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.683+0000 7f1c47e00700 1 --2- 172.21.15.146:0/3654395712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4810b350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c380098a0 tx=0x7f1c38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592ac49ca8fa8e72 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.683+0000 7f1c46a00700 1 -- 172.21.15.146:0/3654395712 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c38004160 con 0x7f1c48108f60 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c46a00700 1 -- 172.21.15.146:0/3654395712 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c38004300 con 0x7f1c48108f60 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c46a00700 1 -- 172.21.15.146:0/3654395712 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c38011b20 con 0x7f1c48108f60 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c4d600700 1 -- 172.21.15.146:0/3654395712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 msgr2=0x7f1c4810b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c4d600700 1 --2- 172.21.15.146:0/3654395712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4810b350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1c380098a0 tx=0x7f1c38009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c4d600700 1 -- 172.21.15.146:0/3654395712 shutdown_connections 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c4d600700 1 --2- 172.21.15.146:0/3654395712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4810b350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c4d600700 1 -- 172.21.15.146:0/3654395712 >> 172.21.15.146:0/3654395712 conn(0x7f1c481029d0 msgr2=0x7f1c48104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c4d600700 1 -- 172.21.15.146:0/3654395712 shutdown_connections 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.684+0000 7f1c4d600700 1 -- 172.21.15.146:0/3654395712 wait complete. 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.685+0000 7f1c4d600700 1 Processor -- start 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.686+0000 7f1c4d600700 1 -- start start 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.686+0000 7f1c4d600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:44.100 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.686+0000 7f1c4d600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c4810c900 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.686+0000 7f1c47e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.686+0000 7f1c47e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48098/0 (socket says 172.21.15.146:48098) 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.686+0000 7f1c47e00700 1 -- 172.21.15.146:0/2275963357 learned_addr learned my addr 172.21.15.146:0/2275963357 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.687+0000 7f1c47e00700 1 -- 172.21.15.146:0/2275963357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4819ec40 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.687+0000 7f1c47e00700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4819e700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c380116a0 tx=0x7f1c38011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.687+0000 7f1c44c00700 1 -- 172.21.15.146:0/2275963357 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c38003a40 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.687+0000 7f1c44c00700 1 -- 172.21.15.146:0/2275963357 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c38003be0 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.687+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4819eed0 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.687+0000 7f1c44c00700 1 -- 172.21.15.146:0/2275963357 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c3801a750 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.687+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4819f330 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.688+0000 7f1c44c00700 1 -- 172.21.15.146:0/2275963357 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1c38024430 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.689+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c48069c60 con 0x7f1c48108f60 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.689+0000 7f1c44c00700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f1c2c0395b0 0x7f1c2c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:44.101 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.689+0000 7f1c44c00700 1 -- 172.21.15.146:0/2275963357 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c3804dd30 con 0x7f1c48108f60 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.689+0000 7f1c47400700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f1c2c0395b0 0x7f1c2c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.690+0000 7f1c47400700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f1c2c0395b0 0x7f1c2c03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c34006fd0 tx=0x7f1c34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.695+0000 7f1c44c00700 1 -- 172.21.15.146:0/2275963357 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c38011e20 con 0x7f1c48108f60 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.914+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f1c481a1bc0 con 0x7f1c48108f60 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.918+0000 7f1c44c00700 1 -- 172.21.15.146:0/2275963357 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1c380247d0 con 0x7f1c48108f60 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.921+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f1c2c0395b0 msgr2=0x7f1c2c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.921+0000 7f1c4d600700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f1c2c0395b0 0x7f1c2c03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1c34006fd0 tx=0x7f1c34006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.921+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 msgr2=0x7f1c4819e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.921+0000 7f1c4d600700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4819e700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c380116a0 tx=0x7f1c38011ba0 comp rx=0 tx=0).stop 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.921+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 shutdown_connections 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.921+0000 7f1c4d600700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f1c2c0395b0 0x7f1c2c03ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.921+0000 7f1c4d600700 1 --2- 172.21.15.146:0/2275963357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c48108f60 0x7f1c4819e700 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.922+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 >> 172.21.15.146:0/2275963357 conn(0x7f1c481029d0 msgr2=0x7f1c48103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.922+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 shutdown_connections 2024-04-24T08:10:44.102 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:42.922+0000 7f1c4d600700 1 -- 172.21.15.146:0/2275963357 wait complete. 2024-04-24T08:10:44.909 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:44 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:44.910 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:44 smithi146 ceph-mon[27805]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.668+0000 7f9c00800700 1 Processor -- start 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.668+0000 7f9c00800700 1 -- start start 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.668+0000 7f9c00800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf810acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.668+0000 7f9c00800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf810b2a0 con 0x7f9bf810a8b0 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.669+0000 7f9bfde00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf810acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.669+0000 7f9bfde00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf810acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48102/0 (socket says 172.21.15.146:48102) 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.669+0000 7f9bfde00700 1 -- 172.21.15.146:0/3731462472 learned_addr learned my addr 172.21.15.146:0/3731462472 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.669+0000 7f9bfde00700 1 -- 172.21.15.146:0/3731462472 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf810bab0 con 0x7f9bf810a8b0 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.670+0000 7f9bfde00700 1 --2- 172.21.15.146:0/3731462472 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf810acd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0098a0 tx=0x7f9bec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23d88af42beb960 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:45.650 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.670+0000 7f9bfca00700 1 -- 172.21.15.146:0/3731462472 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bec004160 con 0x7f9bf810a8b0 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.670+0000 7f9bfca00700 1 -- 172.21.15.146:0/3731462472 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bec004300 con 0x7f9bf810a8b0 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.670+0000 7f9bfca00700 1 -- 172.21.15.146:0/3731462472 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bec011b20 con 0x7f9bf810a8b0 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.671+0000 7f9c00800700 1 -- 172.21.15.146:0/3731462472 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 msgr2=0x7f9bf810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.671+0000 7f9c00800700 1 --2- 172.21.15.146:0/3731462472 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf810acd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0098a0 tx=0x7f9bec009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.671+0000 7f9c00800700 1 -- 172.21.15.146:0/3731462472 shutdown_connections 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.671+0000 7f9c00800700 1 --2- 172.21.15.146:0/3731462472 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf810acd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.671+0000 7f9c00800700 1 -- 172.21.15.146:0/3731462472 >> 172.21.15.146:0/3731462472 conn(0x7f9bf807dcc0 msgr2=0x7f9bf807e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.671+0000 7f9c00800700 1 -- 172.21.15.146:0/3731462472 shutdown_connections 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.671+0000 7f9c00800700 1 -- 172.21.15.146:0/3731462472 wait complete. 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.672+0000 7f9c00800700 1 Processor -- start 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.672+0000 7f9c00800700 1 -- start start 2024-04-24T08:10:45.651 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.672+0000 7f9c00800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.673+0000 7f9c00800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf810ee60 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.673+0000 7f9bfde00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.673+0000 7f9bfde00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf819e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48112/0 (socket says 172.21.15.146:48112) 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.673+0000 7f9bfde00700 1 -- 172.21.15.146:0/172132248 learned_addr learned my addr 172.21.15.146:0/172132248 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.673+0000 7f9bfde00700 1 -- 172.21.15.146:0/172132248 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf819eec0 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.673+0000 7f9bfde00700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf819e980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9bec000c00 tx=0x7f9bec011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.674+0000 7f9bf2a00700 1 -- 172.21.15.146:0/172132248 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bec003b60 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.674+0000 7f9bf2a00700 1 -- 172.21.15.146:0/172132248 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bec01a910 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.674+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf819f150 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.674+0000 7f9bf2a00700 1 -- 172.21.15.146:0/172132248 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bec024930 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.674+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf819fc70 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.675+0000 7f9bf2a00700 1 -- 172.21.15.146:0/172132248 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9bec022020 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.675+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf8064150 con 0x7f9bf810a8b0 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.675+0000 7f9bf2a00700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f9be0039600 0x7f9be003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:45.652 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.675+0000 7f9bf2a00700 1 -- 172.21.15.146:0/172132248 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9bec04d9a0 con 0x7f9bf810a8b0 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.676+0000 7f9bfd400700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f9be0039600 0x7f9be003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.677+0000 7f9bfd400700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f9be0039600 0x7f9be003bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006fd0 tx=0x7f9be8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.681+0000 7f9bf2a00700 1 -- 172.21.15.146:0/172132248 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9bec01ab70 con 0x7f9bf810a8b0 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.908+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f9bf81a00f0 con 0x7f9bf810a8b0 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.914+0000 7f9bf2a00700 1 -- 172.21.15.146:0/172132248 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f9bec02d3f0 con 0x7f9bf810a8b0 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f9be0039600 msgr2=0x7f9be003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f9be0039600 0x7f9be003bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006fd0 tx=0x7f9be8006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 msgr2=0x7f9bf819e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf819e980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9bec000c00 tx=0x7f9bec011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 shutdown_connections 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f9be0039600 0x7f9be003bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 --2- 172.21.15.146:0/172132248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bf810a8b0 0x7f9bf819e980 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.917+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 >> 172.21.15.146:0/172132248 conn(0x7f9bf807dcc0 msgr2=0x7f9bf8108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:45.653 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.918+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 shutdown_connections 2024-04-24T08:10:45.654 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:44.918+0000 7f9c00800700 1 -- 172.21.15.146:0/172132248 wait complete. 2024-04-24T08:10:45.654 INFO:teuthology.orchestra.run.smithi146.stdout:Enabling the dashboard module... 2024-04-24T08:10:46.212 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:45 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/172132248' entity='client.admin' 2024-04-24T08:10:47.453 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:47 smithi146 ceph-mon[27805]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:10:47.453 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:47 smithi146 ceph-mon[27805]: from='mgr.14120 172.21.15.146:0/3680519138' entity='mgr.x' 2024-04-24T08:10:47.453 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:47 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/887418039' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-24T08:10:47.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: ignoring --setuser ceph since I am not root 2024-04-24T08:10:47.712 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: ignoring --setgroup ceph since I am not root 2024-04-24T08:10:47.712 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:47.639+0000 7f87337a5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:10:47.992 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:47.880+0000 7f87337a5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:48.351 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:48 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:48.010+0000 7f87337a5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:10:48.510 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.294+0000 7ff0e4000700 1 Processor -- start 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.295+0000 7ff0e4000700 1 -- start start 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.295+0000 7ff0e4000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc108b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.295+0000 7ff0e4000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc1090e0 con 0x7ff0dc1086f0 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.296+0000 7ff0e1600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc108b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.296+0000 7ff0e1600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc108b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48124/0 (socket says 172.21.15.146:48124) 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.296+0000 7ff0e1600700 1 -- 172.21.15.146:0/730834782 learned_addr learned my addr 172.21.15.146:0/730834782 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.296+0000 7ff0e1600700 1 -- 172.21.15.146:0/730834782 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc1098a0 con 0x7ff0dc1086f0 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.296+0000 7ff0e1600700 1 --2- 172.21.15.146:0/730834782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc108b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00098a0 tx=0x7ff0d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e810e7eb3b940b1b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.297+0000 7ff0dbe00700 1 -- 172.21.15.146:0/730834782 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0d0004160 con 0x7ff0dc1086f0 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.297+0000 7ff0dbe00700 1 -- 172.21.15.146:0/730834782 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d0004300 con 0x7ff0dc1086f0 2024-04-24T08:10:48.516 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.297+0000 7ff0dbe00700 1 -- 172.21.15.146:0/730834782 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0d0011b20 con 0x7ff0dc1086f0 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.298+0000 7ff0e4000700 1 -- 172.21.15.146:0/730834782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 msgr2=0x7ff0dc108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.298+0000 7ff0e4000700 1 --2- 172.21.15.146:0/730834782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc108b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00098a0 tx=0x7ff0d0009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.298+0000 7ff0e4000700 1 -- 172.21.15.146:0/730834782 shutdown_connections 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.298+0000 7ff0e4000700 1 --2- 172.21.15.146:0/730834782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc108b10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.298+0000 7ff0e4000700 1 -- 172.21.15.146:0/730834782 >> 172.21.15.146:0/730834782 conn(0x7ff0dc103cd0 msgr2=0x7ff0dc1060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.298+0000 7ff0e4000700 1 -- 172.21.15.146:0/730834782 shutdown_connections 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.298+0000 7ff0e4000700 1 -- 172.21.15.146:0/730834782 wait complete. 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.299+0000 7ff0e4000700 1 Processor -- start 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.299+0000 7ff0e4000700 1 -- start start 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.299+0000 7ff0e4000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc07ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.299+0000 7ff0e4000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc10cc50 con 0x7ff0dc1086f0 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.300+0000 7ff0e1600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc07ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.300+0000 7ff0e1600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc07ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48134/0 (socket says 172.21.15.146:48134) 2024-04-24T08:10:48.517 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.300+0000 7ff0e1600700 1 -- 172.21.15.146:0/887418039 learned_addr learned my addr 172.21.15.146:0/887418039 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.300+0000 7ff0e1600700 1 -- 172.21.15.146:0/887418039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc07d290 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.300+0000 7ff0e1600700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc07ed40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0000c00 tx=0x7ff0d0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.301+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0d0003b60 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.301+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0d001a910 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.301+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0dc07d520 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.301+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0d0024930 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.301+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0dc07d980 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.302+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7ff0d0022020 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.302+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0dc064150 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.303+0000 7ff0da000700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7ff0c40395b0 0x7ff0c403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.303+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff0d004d8d0 con 0x7ff0dc1086f0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.303+0000 7ff0e0c00700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7ff0c40395b0 0x7ff0c403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:48.518 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.304+0000 7ff0e0c00700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7ff0c40395b0 0x7ff0c403ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8006fd0 tx=0x7ff0c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.309+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0d000b5a0 con 0x7ff0dc1086f0 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:46.606+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7ff0dc07dea0 con 0x7ff0dc1086f0 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.342+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7ff0d0021d80 con 0x7ff0dc1086f0 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.347+0000 7ff0da000700 1 -- 172.21.15.146:0/887418039 <== mon.0 v2:172.21.15.146:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7ff0d002dc90 con 0x7ff0dc1086f0 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7ff0c40395b0 msgr2=0x7ff0c403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7ff0c40395b0 0x7ff0c403ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8006fd0 tx=0x7ff0c8006e40 comp rx=0 tx=0).stop 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 msgr2=0x7ff0dc07ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc07ed40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0000c00 tx=0x7ff0d0011c10 comp rx=0 tx=0).stop 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 shutdown_connections 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7ff0c40395b0 0x7ff0c403ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 --2- 172.21.15.146:0/887418039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc1086f0 0x7ff0dc07ed40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 >> 172.21.15.146:0/887418039 conn(0x7ff0dc103cd0 msgr2=0x7ff0dc104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 shutdown_connections 2024-04-24T08:10:48.519 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:47.348+0000 7ff0e4000700 1 -- 172.21.15.146:0/887418039 wait complete. 2024-04-24T08:10:48.669 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:48 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:48.409+0000 7f87337a5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:10:48.669 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:48 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:48.525+0000 7f87337a5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:10:48.670 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:48 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/887418039' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-24T08:10:48.670 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:48 smithi146 ceph-mon[27805]: mgrmap e9: x(active, since 25s) 2024-04-24T08:10:49.872 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:49 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3630093995' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T08:10:50.209 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.175+0000 7f603c200700 1 Processor -- start 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.175+0000 7f603c200700 1 -- start start 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.175+0000 7f603c200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f60341079e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.175+0000 7f603c200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6034107fb0 con 0x7f60341075c0 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.175+0000 7f6039800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f60341079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.175+0000 7f6039800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f60341079e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48174/0 (socket says 172.21.15.146:48174) 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.175+0000 7f6039800700 1 -- 172.21.15.146:0/3187617005 learned_addr learned my addr 172.21.15.146:0/3187617005 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:50.210 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.176+0000 7f6039800700 1 -- 172.21.15.146:0/3187617005 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60341087c0 con 0x7f60341075c0 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.176+0000 7f6039800700 1 --2- 172.21.15.146:0/3187617005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f60341079e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6028009e80 tx=0x7f602800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79c68ab9cc8cd9a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.176+0000 7f6033e00700 1 -- 172.21.15.146:0/3187617005 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6028017070 con 0x7f60341075c0 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f6033e00700 1 -- 172.21.15.146:0/3187617005 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f602800b8c0 con 0x7f60341075c0 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f6033e00700 1 -- 172.21.15.146:0/3187617005 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60280045c0 con 0x7f60341075c0 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f603c200700 1 -- 172.21.15.146:0/3187617005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 msgr2=0x7f60341079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f603c200700 1 --2- 172.21.15.146:0/3187617005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f60341079e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6028009e80 tx=0x7f602800b330 comp rx=0 tx=0).stop 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f603c200700 1 -- 172.21.15.146:0/3187617005 shutdown_connections 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f603c200700 1 --2- 172.21.15.146:0/3187617005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f60341079e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f603c200700 1 -- 172.21.15.146:0/3187617005 >> 172.21.15.146:0/3187617005 conn(0x7f6034102be0 msgr2=0x7f6034105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.177+0000 7f603c200700 1 -- 172.21.15.146:0/3187617005 shutdown_connections 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.178+0000 7f603c200700 1 -- 172.21.15.146:0/3187617005 wait complete. 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.178+0000 7f603c200700 1 Processor -- start 2024-04-24T08:10:50.211 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.179+0000 7f603c200700 1 -- start start 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.179+0000 7f603c200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f603410d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.179+0000 7f603c200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6034075510 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.179+0000 7f6039800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f603410d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.179+0000 7f6039800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f603410d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48188/0 (socket says 172.21.15.146:48188) 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.179+0000 7f6039800700 1 -- 172.21.15.146:0/3630093995 learned_addr learned my addr 172.21.15.146:0/3630093995 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.180+0000 7f6039800700 1 -- 172.21.15.146:0/3630093995 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603410bbb0 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.180+0000 7f6039800700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f603410d660 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6028009530 tx=0x7f6028003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.180+0000 7f6032000700 1 -- 172.21.15.146:0/3630093995 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6028020020 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.180+0000 7f6032000700 1 -- 172.21.15.146:0/3630093995 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f602800bba0 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.180+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603410be40 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.180+0000 7f6032000700 1 -- 172.21.15.146:0/3630093995 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6028017040 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.180+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f603410c380 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.181+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6034064150 con 0x7f60341075c0 2024-04-24T08:10:50.212 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.182+0000 7f6032000700 1 -- 172.21.15.146:0/3630093995 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f6028023020 con 0x7f60341075c0 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.182+0000 7f6032000700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 0x7f601c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.182+0000 7f6032000700 1 -- 172.21.15.146:0/3630093995 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f602800f050 con 0x7f60341075c0 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.182+0000 7f6038e00700 1 -- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 msgr2=0x7f601c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/888738583 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.182+0000 7f6038e00700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 0x7f601c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.188+0000 7f6032000700 1 -- 172.21.15.146:0/3630093995 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f602802a020 con 0x7f60341075c0 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.383+0000 7f6038e00700 1 -- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 msgr2=0x7f601c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/888738583 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.383+0000 7f6038e00700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 0x7f601c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.475+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f603402e120 con 0x7f60341075c0 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.476+0000 7f6032000700 1 -- 172.21.15.146:0/3630093995 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f602802d350 con 0x7f60341075c0 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 msgr2=0x7f601c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 0x7f601c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:50.213 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 msgr2=0x7f603410d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f603410d660 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6028009530 tx=0x7f6028003d00 comp rx=0 tx=0).stop 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 shutdown_connections 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f601c039600 0x7f601c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 --2- 172.21.15.146:0/3630093995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60341075c0 0x7f603410d660 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 >> 172.21.15.146:0/3630093995 conn(0x7f6034102be0 msgr2=0x7f6034103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 shutdown_connections 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:49.479+0000 7f603c200700 1 -- 172.21.15.146:0/3630093995 wait complete. 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for the mgr to restart... 2024-04-24T08:10:50.214 INFO:teuthology.orchestra.run.smithi146.stdout:Waiting for mgr epoch 9... 2024-04-24T08:10:50.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:50.128+0000 7f87337a5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:10:50.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:50.364+0000 7f87337a5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:50.857 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:50.587+0000 7f87337a5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:10:51.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:50.979+0000 7f87337a5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:10:51.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:51 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:51.099+0000 7f87337a5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:10:51.605 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:51 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:51.351+0000 7f87337a5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:10:51.992 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:51 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:51.604+0000 7f87337a5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:52.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:52.392+0000 7f87337a5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:10:53.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:52.951+0000 7f87337a5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:10:53.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:53.078+0000 7f87337a5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:10:53.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:53.342+0000 7f87337a5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:10:54.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:53.986+0000 7f87337a5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:10:54.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:54.108+0000 7f87337a5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:10:54.736 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:54.430+0000 7f87337a5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:10:54.736 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:54.608+0000 7f87337a5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:10:54.736 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:54.734+0000 7f87337a5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:10:54.992 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:54.885+0000 7f87337a5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:55.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:55 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:55.265+0000 7f87337a5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:10:56.304 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:56.017+0000 7f87337a5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:10:56.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:10:56.303+0000 7f87337a5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:10:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: Active manager daemon x restarted 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: Activating manager daemon x 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: osdmap e3: 0 total, 0 up, 0 in 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: mgrmap e10: x(active, starting, since 0.00968545s) 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:10:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:56 smithi146 ceph-mon[27805]: Manager daemon x is now available 2024-04-24T08:10:57.646 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:57.646 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:57.646 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:57.647 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:57 smithi146 ceph-mon[27805]: mgrmap e11: x(active, since 1.01043s) 2024-04-24T08:10:57.985 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout { 2024-04-24T08:10:57.985 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-24T08:10:57.985 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout } 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.856+0000 7f54c9600700 1 Processor -- start 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.856+0000 7f54c9600700 1 -- start start 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.857+0000 7f54c9600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c407c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.857+0000 7f54c9600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f54c407c790 con 0x7f54c407dd60 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.857+0000 7f54c3e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c407c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.857+0000 7f54c3e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c407c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48194/0 (socket says 172.21.15.146:48194) 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.857+0000 7f54c3e00700 1 -- 172.21.15.146:0/3096878961 learned_addr learned my addr 172.21.15.146:0/3096878961 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.858+0000 7f54c3e00700 1 -- 172.21.15.146:0/3096878961 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c407c910 con 0x7f54c407dd60 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.858+0000 7f54c3e00700 1 --2- 172.21.15.146:0/3096878961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c407c1c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f54b40098a0 tx=0x7f54b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687d4d22d2b2a6e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.858+0000 7f54c2a00700 1 -- 172.21.15.146:0/3096878961 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4004160 con 0x7f54c407dd60 2024-04-24T08:10:57.990 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.858+0000 7f54c2a00700 1 -- 172.21.15.146:0/3096878961 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54b4004300 con 0x7f54c407dd60 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.858+0000 7f54c2a00700 1 -- 172.21.15.146:0/3096878961 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4011b20 con 0x7f54c407dd60 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.859+0000 7f54c9600700 1 -- 172.21.15.146:0/3096878961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 msgr2=0x7f54c407c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.859+0000 7f54c9600700 1 --2- 172.21.15.146:0/3096878961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c407c1c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f54b40098a0 tx=0x7f54b4009bb0 comp rx=0 tx=0).stop 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.859+0000 7f54c9600700 1 -- 172.21.15.146:0/3096878961 shutdown_connections 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.859+0000 7f54c9600700 1 --2- 172.21.15.146:0/3096878961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c407c1c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.859+0000 7f54c9600700 1 -- 172.21.15.146:0/3096878961 >> 172.21.15.146:0/3096878961 conn(0x7f54c4103b30 msgr2=0x7f54c4105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.859+0000 7f54c9600700 1 -- 172.21.15.146:0/3096878961 shutdown_connections 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.859+0000 7f54c9600700 1 -- 172.21.15.146:0/3096878961 wait complete. 2024-04-24T08:10:57.991 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.860+0000 7f54c9600700 1 Processor -- start 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.860+0000 7f54c9600700 1 -- start start 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.861+0000 7f54c9600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.861+0000 7f54c9600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f54c410ebd0 con 0x7f54c407dd60 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.861+0000 7f54c3e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.861+0000 7f54c3e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48208/0 (socket says 172.21.15.146:48208) 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.861+0000 7f54c3e00700 1 -- 172.21.15.146:0/1699400261 learned_addr learned my addr 172.21.15.146:0/1699400261 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.861+0000 7f54c3e00700 1 -- 172.21.15.146:0/1699400261 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c419ebc0 con 0x7f54c407dd60 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.862+0000 7f54c3e00700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c419e680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54b4011b60 tx=0x7f54b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.862+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4003b60 con 0x7f54c407dd60 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.862+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54b401a910 con 0x7f54c407dd60 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.862+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c419ee50 con 0x7f54c407dd60 2024-04-24T08:10:57.992 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.862+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4024930 con 0x7f54c407dd60 2024-04-24T08:10:57.993 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.862+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c419f2b0 con 0x7f54c407dd60 2024-04-24T08:10:57.993 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.863+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f54b4022020 con 0x7f54c407dd60 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.864+0000 7f54c0c00700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.864+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 --> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f54a803c1d0 con 0x7f54a8039600 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.864+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f54b404da30 con 0x7f54c407dd60 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.864+0000 7f54c3400700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 msgr2=0x7f54a803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/888738583 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:50.864+0000 7f54c3400700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:51.065+0000 7f54c3400700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 msgr2=0x7f54a803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/888738583 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:51.065+0000 7f54c3400700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:51.466+0000 7f54c3400700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 msgr2=0x7f54a803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/888738583 2024-04-24T08:10:57.997 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:51.466+0000 7f54c3400700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:52.267+0000 7f54c3400700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 msgr2=0x7f54a803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/888738583 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:52.267+0000 7f54c3400700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:53.869+0000 7f54c3400700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 msgr2=0x7f54a803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/888738583 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:53.869+0000 7f54c3400700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:56.312+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f54b402d3f0 con 0x7f54c407dd60 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:56.312+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 msgr2=0x7f54a803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:56.312+0000 7f54c0c00700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.313+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f54b404fae0 con 0x7f54c407dd60 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.314+0000 7f54c0c00700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54a803cfe0 0x7f54a803f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.314+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f54b400af30 con 0x7f54a803cfe0 2024-04-24T08:10:57.998 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.314+0000 7f54c3400700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54a803cfe0 0x7f54a803f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.315+0000 7f54c3400700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54a803cfe0 0x7f54a803f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f54b0003a10 tx=0x7f54b00092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.316+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f54b400af30 con 0x7f54a803cfe0 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.324+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f54c4069c60 con 0x7f54a803cfe0 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c0c00700 1 -- 172.21.15.146:0/1699400261 <== mgr.14152 v2:172.21.15.146:6800/1475219332 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f54c4069c60 con 0x7f54a803cfe0 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54a803cfe0 msgr2=0x7f54a803f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54a803cfe0 0x7f54a803f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f54b0003a10 tx=0x7f54b00092b0 comp rx=0 tx=0).stop 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 msgr2=0x7f54c419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c419e680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54b4011b60 tx=0x7f54b4011c40 comp rx=0 tx=0).stop 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 shutdown_connections 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54a803cfe0 0x7f54a803f3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54c407dd60 0x7f54c419e680 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 --2- 172.21.15.146:0/1699400261 >> [v2:172.21.15.146:6800/888738583,v1:172.21.15.146:6801/888738583] conn(0x7f54a8039600 0x7f54a803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:57.999 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 >> 172.21.15.146:0/1699400261 conn(0x7f54c4103b30 msgr2=0x7f54c4104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:58.000 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 shutdown_connections 2024-04-24T08:10:58.000 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:57.325+0000 7f54c9600700 1 -- 172.21.15.146:0/1699400261 wait complete. 2024-04-24T08:10:58.000 INFO:teuthology.orchestra.run.smithi146.stdout:mgr epoch 9 is available 2024-04-24T08:10:58.000 INFO:teuthology.orchestra.run.smithi146.stdout:Generating a dashboard self-signed certificate... 2024-04-24T08:10:58.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:58 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:57] ENGINE Bus STARTING 2024-04-24T08:10:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:59 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:57] ENGINE Serving on http://172.21.15.146:8765 2024-04-24T08:10:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:59 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:57] ENGINE Serving on https://172.21.15.146:7150 2024-04-24T08:10:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:59 smithi146 ceph-mon[27805]: [24/Apr/2024:08:10:57] ENGINE Bus STARTED 2024-04-24T08:10:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:59 smithi146 ceph-mon[27805]: mgrmap e12: x(active, since 2s) 2024-04-24T08:10:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:10:59 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:00.182 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-24T08:11:00.183 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.104+0000 7f1107200700 1 Processor -- start 2024-04-24T08:11:00.183 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.104+0000 7f1107200700 1 -- start start 2024-04-24T08:11:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.105+0000 7f1107200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f1100108ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.105+0000 7f1107200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f11001090b0 con 0x7f11001086c0 2024-04-24T08:11:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.105+0000 7f10ffe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f1100108ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.105+0000 7f10ffe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f1100108ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40066/0 (socket says 172.21.15.146:40066) 2024-04-24T08:11:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.105+0000 7f10ffe00700 1 -- 172.21.15.146:0/525003599 learned_addr learned my addr 172.21.15.146:0/525003599 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.105+0000 7f10ffe00700 1 -- 172.21.15.146:0/525003599 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11001098c0 con 0x7f11001086c0 2024-04-24T08:11:00.184 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.106+0000 7f10ffe00700 1 --2- 172.21.15.146:0/525003599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f1100108ae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10f00098a0 tx=0x7f10f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc634f61c66db98 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.186 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.106+0000 7f10ff400700 1 -- 172.21.15.146:0/525003599 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10f0004160 con 0x7f11001086c0 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.106+0000 7f10ff400700 1 -- 172.21.15.146:0/525003599 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f0004300 con 0x7f11001086c0 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.106+0000 7f10ff400700 1 -- 172.21.15.146:0/525003599 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10f0011b20 con 0x7f11001086c0 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.107+0000 7f1107200700 1 -- 172.21.15.146:0/525003599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 msgr2=0x7f1100108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.107+0000 7f1107200700 1 --2- 172.21.15.146:0/525003599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f1100108ae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f10f00098a0 tx=0x7f10f0009bb0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.107+0000 7f1107200700 1 -- 172.21.15.146:0/525003599 shutdown_connections 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.107+0000 7f1107200700 1 --2- 172.21.15.146:0/525003599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f1100108ae0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.107+0000 7f1107200700 1 -- 172.21.15.146:0/525003599 >> 172.21.15.146:0/525003599 conn(0x7f1100103cc0 msgr2=0x7f11001060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.107+0000 7f1107200700 1 -- 172.21.15.146:0/525003599 shutdown_connections 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.107+0000 7f1107200700 1 -- 172.21.15.146:0/525003599 wait complete. 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.108+0000 7f1107200700 1 Processor -- start 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.108+0000 7f1107200700 1 -- start start 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.109+0000 7f1107200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f110019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.109+0000 7f1107200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f110010cc70 con 0x7f11001086c0 2024-04-24T08:11:00.187 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.109+0000 7f10ffe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f110019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.188 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.109+0000 7f10ffe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f110019e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40082/0 (socket says 172.21.15.146:40082) 2024-04-24T08:11:00.188 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.109+0000 7f10ffe00700 1 -- 172.21.15.146:0/2142055914 learned_addr learned my addr 172.21.15.146:0/2142055914 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:00.188 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.109+0000 7f10ffe00700 1 -- 172.21.15.146:0/2142055914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110019eeb0 con 0x7f11001086c0 2024-04-24T08:11:00.190 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.110+0000 7f10ffe00700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f110019e970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10f0000c00 tx=0x7f10f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.110+0000 7f10fd600700 1 -- 172.21.15.146:0/2142055914 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10f0003b60 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.110+0000 7f10fd600700 1 -- 172.21.15.146:0/2142055914 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f001a910 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.110+0000 7f10fd600700 1 -- 172.21.15.146:0/2142055914 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10f0024930 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.110+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110019f140 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.110+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110019fc60 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.111+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1100064150 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.111+0000 7f10fd600700 1 -- 172.21.15.146:0/2142055914 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f10f0022020 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.112+0000 7f10fd600700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f10e4039660 0x7f10e403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.112+0000 7f10fd600700 1 -- 172.21.15.146:0/2142055914 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f10f004d9d0 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.112+0000 7f10f7400700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f10e4039660 0x7f10e403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.113+0000 7f10f7400700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f10e4039660 0x7f10e403bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006fd0 tx=0x7f10e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.118+0000 7f10fd600700 1 -- 172.21.15.146:0/2142055914 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10f0021d80 con 0x7f11001086c0 2024-04-24T08:11:00.191 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.355+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f1100104e70 con 0x7f10e4039660 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.512+0000 7f10fd600700 1 -- 172.21.15.146:0/2142055914 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f1100104e70 con 0x7f10e4039660 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f10e4039660 msgr2=0x7f10e403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f10e4039660 0x7f10e403bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006fd0 tx=0x7f10e8006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 msgr2=0x7f110019e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f110019e970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10f0000c00 tx=0x7f10f0011c10 comp rx=0 tx=0).stop 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 shutdown_connections 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f10e4039660 0x7f10e403bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 --2- 172.21.15.146:0/2142055914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11001086c0 0x7f110019e970 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 >> 172.21.15.146:0/2142055914 conn(0x7f1100103cc0 msgr2=0x7f1100104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.515+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 shutdown_connections 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:10:59.516+0000 7f1107200700 1 -- 172.21.15.146:0/2142055914 wait complete. 2024-04-24T08:11:00.192 INFO:teuthology.orchestra.run.smithi146.stdout:Creating initial admin user... 2024-04-24T08:11:00.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:00 smithi146 ceph-mon[27805]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:00.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:00 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:00.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:00 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:02.054 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$r4Y18kwHGdWX54vnlcuaWuqU2Vdo7ktwM9yyDYgOaBQTg./cqH5LW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713946261, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-24T08:11:02.055 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.830+0000 7fa76c800700 1 Processor -- start 2024-04-24T08:11:02.055 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.831+0000 7fa76c800700 1 -- start start 2024-04-24T08:11:02.055 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.831+0000 7fa76c800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76410b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:02.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.831+0000 7fa76c800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7640763f0 con 0x7fa764109130 2024-04-24T08:11:02.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.831+0000 7fa769e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76410b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:02.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.832+0000 7fa769e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76410b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40088/0 (socket says 172.21.15.146:40088) 2024-04-24T08:11:02.056 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.832+0000 7fa769e00700 1 -- 172.21.15.146:0/1725771168 learned_addr learned my addr 172.21.15.146:0/1725771168 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:02.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.832+0000 7fa769e00700 1 -- 172.21.15.146:0/1725771168 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76410ba60 con 0x7fa764109130 2024-04-24T08:11:02.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.832+0000 7fa769e00700 1 --2- 172.21.15.146:0/1725771168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76410b520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7580098a0 tx=0x7fa758009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1f799652523b03 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:02.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.833+0000 7fa768a00700 1 -- 172.21.15.146:0/1725771168 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa758004160 con 0x7fa764109130 2024-04-24T08:11:02.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.833+0000 7fa768a00700 1 -- 172.21.15.146:0/1725771168 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa758004300 con 0x7fa764109130 2024-04-24T08:11:02.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.833+0000 7fa768a00700 1 -- 172.21.15.146:0/1725771168 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa758011b20 con 0x7fa764109130 2024-04-24T08:11:02.057 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.834+0000 7fa76c800700 1 -- 172.21.15.146:0/1725771168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 msgr2=0x7fa76410b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:02.058 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.834+0000 7fa76c800700 1 --2- 172.21.15.146:0/1725771168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76410b520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7580098a0 tx=0x7fa758009bb0 comp rx=0 tx=0).stop 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.834+0000 7fa76c800700 1 -- 172.21.15.146:0/1725771168 shutdown_connections 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.834+0000 7fa76c800700 1 --2- 172.21.15.146:0/1725771168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76410b520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.834+0000 7fa76c800700 1 -- 172.21.15.146:0/1725771168 >> 172.21.15.146:0/1725771168 conn(0x7fa764102be0 msgr2=0x7fa764105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.834+0000 7fa76c800700 1 -- 172.21.15.146:0/1725771168 shutdown_connections 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.834+0000 7fa76c800700 1 -- 172.21.15.146:0/1725771168 wait complete. 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.835+0000 7fa76c800700 1 Processor -- start 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.835+0000 7fa76c800700 1 -- start start 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.835+0000 7fa76c800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.835+0000 7fa76c800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa76410cad0 con 0x7fa764109130 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.836+0000 7fa769e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:02.059 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.836+0000 7fa769e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76419e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40096/0 (socket says 172.21.15.146:40096) 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.836+0000 7fa769e00700 1 -- 172.21.15.146:0/353739627 learned_addr learned my addr 172.21.15.146:0/353739627 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.836+0000 7fa769e00700 1 -- 172.21.15.146:0/353739627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76419ee50 con 0x7fa764109130 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.836+0000 7fa769e00700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76419e910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa758011b60 tx=0x7fa758011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.836+0000 7fa75ea00700 1 -- 172.21.15.146:0/353739627 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa758003b60 con 0x7fa764109130 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.837+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76419f0e0 con 0x7fa764109130 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.837+0000 7fa75ea00700 1 -- 172.21.15.146:0/353739627 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75801a910 con 0x7fa764109130 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.837+0000 7fa75ea00700 1 -- 172.21.15.146:0/353739627 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa758024930 con 0x7fa764109130 2024-04-24T08:11:02.061 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.837+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76419fc00 con 0x7fa764109130 2024-04-24T08:11:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.838+0000 7fa75ea00700 1 -- 172.21.15.146:0/353739627 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fa758022020 con 0x7fa764109130 2024-04-24T08:11:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.838+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa764064150 con 0x7fa764109130 2024-04-24T08:11:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.838+0000 7fa75ea00700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa74c039660 0x7fa74c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.839+0000 7fa75ea00700 1 -- 172.21.15.146:0/353739627 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa75804da90 con 0x7fa764109130 2024-04-24T08:11:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.839+0000 7fa769400700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa74c039660 0x7fa74c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.840+0000 7fa769400700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa74c039660 0x7fa74c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa754006fd0 tx=0x7fa754006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:02.062 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:00.844+0000 7fa75ea00700 1 -- 172.21.15.146:0/353739627 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa75800b3e0 con 0x7fa764109130 2024-04-24T08:11:02.063 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.093+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fa764103d50 con 0x7fa74c039660 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.366+0000 7fa75ea00700 1 -- 172.21.15.146:0/353739627 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fa764103d50 con 0x7fa74c039660 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa74c039660 msgr2=0x7fa74c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa74c039660 0x7fa74c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa754006fd0 tx=0x7fa754006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 msgr2=0x7fa76419e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76419e910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa758011b60 tx=0x7fa758011c40 comp rx=0 tx=0).stop 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 shutdown_connections 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa74c039660 0x7fa74c03bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:02.064 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 --2- 172.21.15.146:0/353739627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa764109130 0x7fa76419e910 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:02.065 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 >> 172.21.15.146:0/353739627 conn(0x7fa764102be0 msgr2=0x7fa764103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:02.065 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 shutdown_connections 2024-04-24T08:11:02.065 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:01.370+0000 7fa76c800700 1 -- 172.21.15.146:0/353739627 wait complete. 2024-04-24T08:11:02.065 INFO:teuthology.orchestra.run.smithi146.stdout:Fetching dashboard port number... 2024-04-24T08:11:02.361 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:02.361 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:02 smithi146 ceph-mon[27805]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:02.361 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:03.232 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:03 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3458617579' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stdout 8443 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.685+0000 7f7e2d000700 1 Processor -- start 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.685+0000 7f7e2d000700 1 -- start start 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.686+0000 7f7e2d000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e2807c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.686+0000 7f7e2d000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2807c7b0 con 0x7f7e2807dd80 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.686+0000 7f7e25e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e2807c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.686+0000 7f7e25e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e2807c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49076/0 (socket says 172.21.15.146:49076) 2024-04-24T08:11:04.197 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.686+0000 7f7e25e00700 1 -- 172.21.15.146:0/3250927644 learned_addr learned my addr 172.21.15.146:0/3250927644 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:04.198 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.687+0000 7f7e25e00700 1 -- 172.21.15.146:0/3250927644 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2807c930 con 0x7f7e2807dd80 2024-04-24T08:11:04.198 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.687+0000 7f7e25e00700 1 --2- 172.21.15.146:0/3250927644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e2807c1e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e180098a0 tx=0x7f7e18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f39e396464e747 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:04.198 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.687+0000 7f7e24a00700 1 -- 172.21.15.146:0/3250927644 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e18004160 con 0x7f7e2807dd80 2024-04-24T08:11:04.198 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.687+0000 7f7e24a00700 1 -- 172.21.15.146:0/3250927644 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e18004300 con 0x7f7e2807dd80 2024-04-24T08:11:04.198 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.687+0000 7f7e24a00700 1 -- 172.21.15.146:0/3250927644 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e18011b20 con 0x7f7e2807dd80 2024-04-24T08:11:04.198 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.688+0000 7f7e2d000700 1 -- 172.21.15.146:0/3250927644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 msgr2=0x7f7e2807c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:04.202 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.688+0000 7f7e2d000700 1 --2- 172.21.15.146:0/3250927644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e2807c1e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e180098a0 tx=0x7f7e18009bb0 comp rx=0 tx=0).stop 2024-04-24T08:11:04.202 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.688+0000 7f7e2d000700 1 -- 172.21.15.146:0/3250927644 shutdown_connections 2024-04-24T08:11:04.202 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.688+0000 7f7e2d000700 1 --2- 172.21.15.146:0/3250927644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e2807c1e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:04.202 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.688+0000 7f7e2d000700 1 -- 172.21.15.146:0/3250927644 >> 172.21.15.146:0/3250927644 conn(0x7f7e28103cc0 msgr2=0x7f7e281060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:04.202 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.688+0000 7f7e2d000700 1 -- 172.21.15.146:0/3250927644 shutdown_connections 2024-04-24T08:11:04.202 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.688+0000 7f7e2d000700 1 -- 172.21.15.146:0/3250927644 wait complete. 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.690+0000 7f7e2d000700 1 Processor -- start 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.690+0000 7f7e2d000700 1 -- start start 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.690+0000 7f7e2d000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e28108520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.690+0000 7f7e2d000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2810ed20 con 0x7f7e2807dd80 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.690+0000 7f7e25e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e28108520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.690+0000 7f7e25e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e28108520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49084/0 (socket says 172.21.15.146:49084) 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.690+0000 7f7e25e00700 1 -- 172.21.15.146:0/3458617579 learned_addr learned my addr 172.21.15.146:0/3458617579 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.691+0000 7f7e25e00700 1 -- 172.21.15.146:0/3458617579 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e28108a60 con 0x7f7e2807dd80 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.691+0000 7f7e25e00700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e28108520 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e18000c00 tx=0x7f7e18011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.692+0000 7f7e1ea00700 1 -- 172.21.15.146:0/3458617579 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e18003b60 con 0x7f7e2807dd80 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.692+0000 7f7e1ea00700 1 -- 172.21.15.146:0/3458617579 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e1801a910 con 0x7f7e2807dd80 2024-04-24T08:11:04.203 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.692+0000 7f7e1ea00700 1 -- 172.21.15.146:0/3458617579 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e18024930 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.692+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e28108cf0 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.692+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e28109860 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.693+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e28064150 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.694+0000 7f7e1ea00700 1 -- 172.21.15.146:0/3458617579 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f7e18022020 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.694+0000 7f7e1ea00700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e0c039660 0x7f7e0c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.694+0000 7f7e1ea00700 1 -- 172.21.15.146:0/3458617579 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7e1804da20 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.695+0000 7f7e25400700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e0c039660 0x7f7e0c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.696+0000 7f7e25400700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e0c039660 0x7f7e0c03bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006fd0 tx=0x7f7e14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.701+0000 7f7e1ea00700 1 -- 172.21.15.146:0/3458617579 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e1800b3e0 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.933+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f7e28109d30 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.933+0000 7f7e1ea00700 1 -- 172.21.15.146:0/3458617579 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f7e1802ddb0 con 0x7f7e2807dd80 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.936+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e0c039660 msgr2=0x7f7e0c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.936+0000 7f7e2d000700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e0c039660 0x7f7e0c03bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006fd0 tx=0x7f7e14006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:04.204 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.936+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 msgr2=0x7f7e28108520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:04.205 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.936+0000 7f7e2d000700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e28108520 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7e18000c00 tx=0x7f7e18011c10 comp rx=0 tx=0).stop 2024-04-24T08:11:04.205 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.937+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 shutdown_connections 2024-04-24T08:11:04.205 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.937+0000 7f7e2d000700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e0c039660 0x7f7e0c03bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:04.205 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.937+0000 7f7e2d000700 1 --2- 172.21.15.146:0/3458617579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2807dd80 0x7f7e28108520 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:04.205 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.937+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 >> 172.21.15.146:0/3458617579 conn(0x7f7e28103cc0 msgr2=0x7f7e28104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:04.205 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.937+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 shutdown_connections 2024-04-24T08:11:04.205 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:02.937+0000 7f7e2d000700 1 -- 172.21.15.146:0/3458617579 wait complete. 2024-04-24T08:11:04.229 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T08:11:04.230 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout disabled 2024-04-24T08:11:04.251 INFO:teuthology.orchestra.run.smithi146.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T08:11:04.251 INFO:teuthology.orchestra.run.smithi146.stdout:systemctl: stdout inactive 2024-04-24T08:11:04.251 INFO:teuthology.orchestra.run.smithi146.stdout:firewalld.service is not enabled 2024-04-24T08:11:04.251 INFO:teuthology.orchestra.run.smithi146.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-24T08:11:04.255 INFO:teuthology.orchestra.run.smithi146.stdout:Ceph Dashboard is now available at: 2024-04-24T08:11:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: URL: https://smithi146.front.sepia.ceph.com:8443/ 2024-04-24T08:11:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: User: admin 2024-04-24T08:11:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: Password: mbf70l43f8 2024-04-24T08:11:04.256 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:04.256 INFO:teuthology.orchestra.run.smithi146.stdout:Saving cluster configuration to /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config directory 2024-04-24T08:11:04.258 INFO:teuthology.orchestra.run.smithi146.stdout:Enabling autotune for osd_memory_target 2024-04-24T08:11:04.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:04 smithi146 ceph-mon[27805]: mgrmap e13: x(active, since 6s) 2024-04-24T08:11:06.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.173+0000 7fb6d3a00700 1 Processor -- start 2024-04-24T08:11:06.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.173+0000 7fb6d3a00700 1 -- start start 2024-04-24T08:11:06.068 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.173+0000 7fb6d3a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc1073c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.174+0000 7fb6d3a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc076370 con 0x7fb6cc106fa0 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.174+0000 7fb6d1000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc1073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.174+0000 7fb6d1000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc1073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49086/0 (socket says 172.21.15.146:49086) 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.174+0000 7fb6d1000700 1 -- 172.21.15.146:0/4178440638 learned_addr learned my addr 172.21.15.146:0/4178440638 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.175+0000 7fb6d1000700 1 -- 172.21.15.146:0/4178440638 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6cc107900 con 0x7fb6cc106fa0 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.175+0000 7fb6d1000700 1 --2- 172.21.15.146:0/4178440638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc1073c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6c00098a0 tx=0x7fb6c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cb065ed2f5dd2c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.176+0000 7fb6cb400700 1 -- 172.21.15.146:0/4178440638 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6c0004160 con 0x7fb6cc106fa0 2024-04-24T08:11:06.069 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.176+0000 7fb6cb400700 1 -- 172.21.15.146:0/4178440638 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6c0004300 con 0x7fb6cc106fa0 2024-04-24T08:11:06.072 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.176+0000 7fb6cb400700 1 -- 172.21.15.146:0/4178440638 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6c0011b20 con 0x7fb6cc106fa0 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.176+0000 7fb6d3a00700 1 -- 172.21.15.146:0/4178440638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 msgr2=0x7fb6cc1073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.176+0000 7fb6d3a00700 1 --2- 172.21.15.146:0/4178440638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc1073c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6c00098a0 tx=0x7fb6c0009bb0 comp rx=0 tx=0).stop 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.177+0000 7fb6d3a00700 1 -- 172.21.15.146:0/4178440638 shutdown_connections 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.177+0000 7fb6d3a00700 1 --2- 172.21.15.146:0/4178440638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc1073c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.177+0000 7fb6d3a00700 1 -- 172.21.15.146:0/4178440638 >> 172.21.15.146:0/4178440638 conn(0x7fb6cc102be0 msgr2=0x7fb6cc105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.177+0000 7fb6d3a00700 1 -- 172.21.15.146:0/4178440638 shutdown_connections 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.177+0000 7fb6d3a00700 1 -- 172.21.15.146:0/4178440638 wait complete. 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.178+0000 7fb6d3a00700 1 Processor -- start 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.178+0000 7fb6d3a00700 1 -- start start 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.178+0000 7fb6d3a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.178+0000 7fb6d3a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb6cc1089d0 con 0x7fb6cc106fa0 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.179+0000 7fb6d1000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.179+0000 7fb6d1000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc19e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49098/0 (socket says 172.21.15.146:49098) 2024-04-24T08:11:06.073 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.179+0000 7fb6d1000700 1 -- 172.21.15.146:0/557428660 learned_addr learned my addr 172.21.15.146:0/557428660 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.179+0000 7fb6d1000700 1 -- 172.21.15.146:0/557428660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6cc19ee60 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.179+0000 7fb6d1000700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc19e920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb6c00116a0 tx=0x7fb6c0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.180+0000 7fb6c9600700 1 -- 172.21.15.146:0/557428660 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6c0003a40 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.180+0000 7fb6c9600700 1 -- 172.21.15.146:0/557428660 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6c0003be0 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.180+0000 7fb6c9600700 1 -- 172.21.15.146:0/557428660 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb6c001a750 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.180+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6cc19f0f0 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.180+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6cc19fc10 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.181+0000 7fb6c9600700 1 -- 172.21.15.146:0/557428660 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fb6c0024aa0 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.181+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6cc064150 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.182+0000 7fb6c9600700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb6b40396b0 0x7fb6b403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.182+0000 7fb6c9600700 1 -- 172.21.15.146:0/557428660 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb6c004df90 con 0x7fb6cc106fa0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.182+0000 7fb6cbe00700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb6b40396b0 0x7fb6b403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:06.074 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.183+0000 7fb6cbe00700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb6b40396b0 0x7fb6b403bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8006fd0 tx=0x7fb6b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:06.076 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.187+0000 7fb6c9600700 1 -- 172.21.15.146:0/557428660 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6c0011e20 con 0x7fb6cc106fa0 2024-04-24T08:11:06.076 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.411+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb6cc1a0860 con 0x7fb6cc106fa0 2024-04-24T08:11:06.076 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.412+0000 7fb6c9600700 1 -- 172.21.15.146:0/557428660 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb6c000fdd0 con 0x7fb6cc106fa0 2024-04-24T08:11:06.076 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb6b40396b0 msgr2=0x7fb6b403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:06.076 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb6b40396b0 0x7fb6b403bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8006fd0 tx=0x7fb6b8006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 msgr2=0x7fb6cc19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc19e920 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb6c00116a0 tx=0x7fb6c0011ba0 comp rx=0 tx=0).stop 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 shutdown_connections 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb6b40396b0 0x7fb6b403bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 --2- 172.21.15.146:0/557428660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6cc106fa0 0x7fb6cc19e920 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 >> 172.21.15.146:0/557428660 conn(0x7fb6cc102be0 msgr2=0x7fb6cc1954b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 shutdown_connections 2024-04-24T08:11:06.077 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:05.415+0000 7fb6d3a00700 1 -- 172.21.15.146:0/557428660 wait complete. 2024-04-24T08:11:07.678 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.676+0000 7f8319a00700 1 Processor -- start 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.677+0000 7f8319a00700 1 -- start start 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.677+0000 7f8319a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831407d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.677+0000 7f8319a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f831407d900 con 0x7f831407ee80 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.677+0000 7f8313e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831407d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.678+0000 7f8313e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831407d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49110/0 (socket says 172.21.15.146:49110) 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.678+0000 7f8313e00700 1 -- 172.21.15.146:0/1598039272 learned_addr learned my addr 172.21.15.146:0/1598039272 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.678+0000 7f8313e00700 1 -- 172.21.15.146:0/1598039272 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831407da80 con 0x7f831407ee80 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.678+0000 7f8313e00700 1 --2- 172.21.15.146:0/1598039272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831407d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8304009e80 tx=0x7f830400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac321d2f6b70caf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.679+0000 7f8313400700 1 -- 172.21.15.146:0/1598039272 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304017070 con 0x7f831407ee80 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.679+0000 7f8313400700 1 -- 172.21.15.146:0/1598039272 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f830400b8c0 con 0x7f831407ee80 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.679+0000 7f8313400700 1 -- 172.21.15.146:0/1598039272 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83040045c0 con 0x7f831407ee80 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.679+0000 7f8319a00700 1 -- 172.21.15.146:0/1598039272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 msgr2=0x7f831407d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:07.679 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.679+0000 7f8319a00700 1 --2- 172.21.15.146:0/1598039272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831407d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8304009e80 tx=0x7f830400b330 comp rx=0 tx=0).stop 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.680+0000 7f8319a00700 1 -- 172.21.15.146:0/1598039272 shutdown_connections 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.680+0000 7f8319a00700 1 --2- 172.21.15.146:0/1598039272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831407d330 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.680+0000 7f8319a00700 1 -- 172.21.15.146:0/1598039272 >> 172.21.15.146:0/1598039272 conn(0x7f8314103b60 msgr2=0x7f8314105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.680+0000 7f8319a00700 1 -- 172.21.15.146:0/1598039272 shutdown_connections 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.680+0000 7f8319a00700 1 -- 172.21.15.146:0/1598039272 wait complete. 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.681+0000 7f8319a00700 1 Processor -- start 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.681+0000 7f8319a00700 1 -- start start 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.681+0000 7f8319a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831410b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.682+0000 7f8319a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f831410eb00 con 0x7f831407ee80 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.682+0000 7f8313e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831410b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.682+0000 7f8313e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831410b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49116/0 (socket says 172.21.15.146:49116) 2024-04-24T08:11:07.680 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.682+0000 7f8313e00700 1 -- 172.21.15.146:0/3052437188 learned_addr learned my addr 172.21.15.146:0/3052437188 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.682+0000 7f8313e00700 1 -- 172.21.15.146:0/3052437188 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831410bc70 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.682+0000 7f8313e00700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831410b730 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f830400b7b0 tx=0x7f830400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.683+0000 7f8311600700 1 -- 172.21.15.146:0/3052437188 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304020020 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.683+0000 7f8311600700 1 -- 172.21.15.146:0/3052437188 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8304003d90 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.683+0000 7f8311600700 1 -- 172.21.15.146:0/3052437188 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304017040 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.683+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831410bf00 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.683+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83141084c0 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.684+0000 7f8311600700 1 -- 172.21.15.146:0/3052437188 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f8304023020 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.684+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8314069c60 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.685+0000 7f8311600700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82f80396b0 0x7f82f803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.685+0000 7f8311600700 1 -- 172.21.15.146:0/3052437188 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f830400f050 con 0x7f831407ee80 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.685+0000 7f830b400700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82f80396b0 0x7f82f803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:07.681 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.686+0000 7f830b400700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82f80396b0 0x7f82f803bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f82fc006fd0 tx=0x7f82fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:06.690+0000 7f8311600700 1 -- 172.21.15.146:0/3052437188 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f830402a020 con 0x7f831407ee80 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.027+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f8314108b60 con 0x7f831407ee80 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.033+0000 7f8311600700 1 -- 172.21.15.146:0/3052437188 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f8304012a50 con 0x7f831407ee80 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82f80396b0 msgr2=0x7f82f803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82f80396b0 0x7f82f803bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f82fc006fd0 tx=0x7f82fc006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 msgr2=0x7f831410b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831410b730 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f830400b7b0 tx=0x7f830400bd50 comp rx=0 tx=0).stop 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 shutdown_connections 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82f80396b0 0x7f82f803bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 --2- 172.21.15.146:0/3052437188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831407ee80 0x7f831410b730 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.035+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 >> 172.21.15.146:0/3052437188 conn(0x7f8314103b60 msgr2=0x7f83141046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.036+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 shutdown_connections 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr 2024-04-24T08:11:07.036+0000 7f8319a00700 1 -- 172.21.15.146:0/3052437188 wait complete. 2024-04-24T08:11:07.682 INFO:teuthology.orchestra.run.smithi146.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout:Or, if you are only running a single cluster on this host: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: ceph telemetry on 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout:For more information see: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.683 INFO:teuthology.orchestra.run.smithi146.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-24T08:11:07.684 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:07.684 INFO:teuthology.orchestra.run.smithi146.stdout:Bootstrap complete. 2024-04-24T08:11:07.759 INFO:tasks.cephadm:Fetching config... 2024-04-24T08:11:07.760 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:11:07.760 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-24T08:11:07.809 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-24T08:11:07.809 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:11:07.809 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-24T08:11:07.894 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-24T08:11:07.894 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:11:07.894 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-24T08:11:08.009 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-24T08:11:08.009 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:11:08.009 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-24T08:11:08.102 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-24T08:11:08.103 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyUjgvPYjaBtkv5xswb5eTWRslbu3sD5yRgc7TCEAyiitMZQEyMfbopGWgpuHqJ5u1AS5yOKtx0w7/F7okbmtwfpj7e/PuNpzQu0TnqcqBgY5hYi0JxdCdnxoZCA+VxoAWHqzpdQ8uzGzZMnUNJz/X1rYUkMq5DCF26iOubp+8y+tLV/6cYqXsqhJcpK5f3bQ4zfkDfVYhQ5m4AklU8CGkhpHnsiOk4+LAWeozVRMNAv5CpzGVw4mxFDi5/AVrbsUhR8WTUO+nI3bZWaCo7tXVF93Zk0dyC+4iQn+jGcybYRS27GD4r/tsHAdGQf4ENHc/3BLdVEc5k3AQfMv2iW3PyOHMFG77kPXW5Es9stt+hrFc4bNeOj32wWPDmQLfR0PBT6xskJeem8+ePtXeNsnQBxwGmHb5gTHqOxLYjBQLBiH7C99ezAMMuJS6p8ysY/mF60z/rd4t+9511lSitU/cqAPZsxo/TTpX7jLYjI6e8Kpv+tep+2FHH2vP6Sow5p8= ceph-cca14746-0211-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T08:11:08.194 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyUjgvPYjaBtkv5xswb5eTWRslbu3sD5yRgc7TCEAyiitMZQEyMfbopGWgpuHqJ5u1AS5yOKtx0w7/F7okbmtwfpj7e/PuNpzQu0TnqcqBgY5hYi0JxdCdnxoZCA+VxoAWHqzpdQ8uzGzZMnUNJz/X1rYUkMq5DCF26iOubp+8y+tLV/6cYqXsqhJcpK5f3bQ4zfkDfVYhQ5m4AklU8CGkhpHnsiOk4+LAWeozVRMNAv5CpzGVw4mxFDi5/AVrbsUhR8WTUO+nI3bZWaCo7tXVF93Zk0dyC+4iQn+jGcybYRS27GD4r/tsHAdGQf4ENHc/3BLdVEc5k3AQfMv2iW3PyOHMFG77kPXW5Es9stt+hrFc4bNeOj32wWPDmQLfR0PBT6xskJeem8+ePtXeNsnQBxwGmHb5gTHqOxLYjBQLBiH7C99ezAMMuJS6p8ysY/mF60z/rd4t+9511lSitU/cqAPZsxo/TTpX7jLYjI6e8Kpv+tep+2FHH2vP6Sow5p8= ceph-cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:11:08.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyUjgvPYjaBtkv5xswb5eTWRslbu3sD5yRgc7TCEAyiitMZQEyMfbopGWgpuHqJ5u1AS5yOKtx0w7/F7okbmtwfpj7e/PuNpzQu0TnqcqBgY5hYi0JxdCdnxoZCA+VxoAWHqzpdQ8uzGzZMnUNJz/X1rYUkMq5DCF26iOubp+8y+tLV/6cYqXsqhJcpK5f3bQ4zfkDfVYhQ5m4AklU8CGkhpHnsiOk4+LAWeozVRMNAv5CpzGVw4mxFDi5/AVrbsUhR8WTUO+nI3bZWaCo7tXVF93Zk0dyC+4iQn+jGcybYRS27GD4r/tsHAdGQf4ENHc/3BLdVEc5k3AQfMv2iW3PyOHMFG77kPXW5Es9stt+hrFc4bNeOj32wWPDmQLfR0PBT6xskJeem8+ePtXeNsnQBxwGmHb5gTHqOxLYjBQLBiH7C99ezAMMuJS6p8ysY/mF60z/rd4t+9511lSitU/cqAPZsxo/TTpX7jLYjI6e8Kpv+tep+2FHH2vP6Sow5p8= ceph-cca14746-0211-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T08:11:08.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:08 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3052437188' entity='client.admin' 2024-04-24T08:11:08.266 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyUjgvPYjaBtkv5xswb5eTWRslbu3sD5yRgc7TCEAyiitMZQEyMfbopGWgpuHqJ5u1AS5yOKtx0w7/F7okbmtwfpj7e/PuNpzQu0TnqcqBgY5hYi0JxdCdnxoZCA+VxoAWHqzpdQ8uzGzZMnUNJz/X1rYUkMq5DCF26iOubp+8y+tLV/6cYqXsqhJcpK5f3bQ4zfkDfVYhQ5m4AklU8CGkhpHnsiOk4+LAWeozVRMNAv5CpzGVw4mxFDi5/AVrbsUhR8WTUO+nI3bZWaCo7tXVF93Zk0dyC+4iQn+jGcybYRS27GD4r/tsHAdGQf4ENHc/3BLdVEc5k3AQfMv2iW3PyOHMFG77kPXW5Es9stt+hrFc4bNeOj32wWPDmQLfR0PBT6xskJeem8+ePtXeNsnQBxwGmHb5gTHqOxLYjBQLBiH7C99ezAMMuJS6p8ysY/mF60z/rd4t+9511lSitU/cqAPZsxo/TTpX7jLYjI6e8Kpv+tep+2FHH2vP6Sow5p8= ceph-cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:11:08.278 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyUjgvPYjaBtkv5xswb5eTWRslbu3sD5yRgc7TCEAyiitMZQEyMfbopGWgpuHqJ5u1AS5yOKtx0w7/F7okbmtwfpj7e/PuNpzQu0TnqcqBgY5hYi0JxdCdnxoZCA+VxoAWHqzpdQ8uzGzZMnUNJz/X1rYUkMq5DCF26iOubp+8y+tLV/6cYqXsqhJcpK5f3bQ4zfkDfVYhQ5m4AklU8CGkhpHnsiOk4+LAWeozVRMNAv5CpzGVw4mxFDi5/AVrbsUhR8WTUO+nI3bZWaCo7tXVF93Zk0dyC+4iQn+jGcybYRS27GD4r/tsHAdGQf4ENHc/3BLdVEc5k3AQfMv2iW3PyOHMFG77kPXW5Es9stt+hrFc4bNeOj32wWPDmQLfR0PBT6xskJeem8+ePtXeNsnQBxwGmHb5gTHqOxLYjBQLBiH7C99ezAMMuJS6p8ysY/mF60z/rd4t+9511lSitU/cqAPZsxo/TTpX7jLYjI6e8Kpv+tep+2FHH2vP6Sow5p8= ceph-cca14746-0211-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T08:11:08.321 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyUjgvPYjaBtkv5xswb5eTWRslbu3sD5yRgc7TCEAyiitMZQEyMfbopGWgpuHqJ5u1AS5yOKtx0w7/F7okbmtwfpj7e/PuNpzQu0TnqcqBgY5hYi0JxdCdnxoZCA+VxoAWHqzpdQ8uzGzZMnUNJz/X1rYUkMq5DCF26iOubp+8y+tLV/6cYqXsqhJcpK5f3bQ4zfkDfVYhQ5m4AklU8CGkhpHnsiOk4+LAWeozVRMNAv5CpzGVw4mxFDi5/AVrbsUhR8WTUO+nI3bZWaCo7tXVF93Zk0dyC+4iQn+jGcybYRS27GD4r/tsHAdGQf4ENHc/3BLdVEc5k3AQfMv2iW3PyOHMFG77kPXW5Es9stt+hrFc4bNeOj32wWPDmQLfR0PBT6xskJeem8+ePtXeNsnQBxwGmHb5gTHqOxLYjBQLBiH7C99ezAMMuJS6p8ysY/mF60z/rd4t+9511lSitU/cqAPZsxo/TTpX7jLYjI6e8Kpv+tep+2FHH2vP6Sow5p8= ceph-cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:11:08.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-24T08:11:08.739 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:11:11.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.445+0000 7f0184a00700 1 -- 172.21.15.146:0/2284945883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 msgr2=0x7f0180103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:11.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.445+0000 7f0184a00700 1 --2- 172.21.15.146:0/2284945883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f0180103270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0170009910 tx=0x7f0170009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:11.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.445+0000 7f0184a00700 1 -- 172.21.15.146:0/2284945883 shutdown_connections 2024-04-24T08:11:11.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.445+0000 7f0184a00700 1 --2- 172.21.15.146:0/2284945883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f0180103270 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:11.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.445+0000 7f0184a00700 1 -- 172.21.15.146:0/2284945883 >> 172.21.15.146:0/2284945883 conn(0x7f01800fca60 msgr2=0x7f01800feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.445+0000 7f0184a00700 1 -- 172.21.15.146:0/2284945883 shutdown_connections 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.445+0000 7f0184a00700 1 -- 172.21.15.146:0/2284945883 wait complete. 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.446+0000 7f0184a00700 1 Processor -- start 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.446+0000 7f0184a00700 1 -- start start 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.447+0000 7f0184a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f018019e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.447+0000 7f0184a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0180106490 con 0x7f0180100e40 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.447+0000 7f017de00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f018019e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.447+0000 7f017de00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f018019e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35010/0 (socket says 172.21.15.146:35010) 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.447+0000 7f017de00700 1 -- 172.21.15.146:0/1837416501 learned_addr learned my addr 172.21.15.146:0/1837416501 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.448+0000 7f017de00700 1 -- 172.21.15.146:0/1837416501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018019e760 con 0x7f0180100e40 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.448+0000 7f017de00700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f018019e220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0170004fa0 tx=0x7f0170005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.448+0000 7f0176a00700 1 -- 172.21.15.146:0/1837416501 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0170027070 con 0x7f0180100e40 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.448+0000 7f0176a00700 1 -- 172.21.15.146:0/1837416501 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f017000be40 con 0x7f0180100e40 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.448+0000 7f0176a00700 1 -- 172.21.15.146:0/1837416501 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f017000f460 con 0x7f0180100e40 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.448+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018019e9f0 con 0x7f0180100e40 2024-04-24T08:11:11.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.448+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01801a16e0 con 0x7f0180100e40 2024-04-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.449+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0180106c10 con 0x7f0180100e40 2024-04-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.450+0000 7f0176a00700 1 -- 172.21.15.146:0/1837416501 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f017002a070 con 0x7f0180100e40 2024-04-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.450+0000 7f0176a00700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01640396b0 0x7f016403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.450+0000 7f0176a00700 1 -- 172.21.15.146:0/1837416501 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f017001e030 con 0x7f0180100e40 2024-04-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.451+0000 7f017d400700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01640396b0 0x7f016403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:11.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.451+0000 7f017d400700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01640396b0 0x7f016403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f016c006fd0 tx=0x7f016c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:11.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.456+0000 7f0176a00700 1 -- 172.21.15.146:0/1837416501 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0170013070 con 0x7f0180100e40 2024-04-24T08:11:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.688+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f018002e0f0 con 0x7f0180100e40 2024-04-24T08:11:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.692+0000 7f0176a00700 1 -- 172.21.15.146:0/1837416501 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f017000b960 con 0x7f0180100e40 2024-04-24T08:11:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01640396b0 msgr2=0x7f016403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01640396b0 0x7f016403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f016c006fd0 tx=0x7f016c006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 msgr2=0x7f018019e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:11.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f018019e220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0170004fa0 tx=0x7f0170005080 comp rx=0 tx=0).stop 2024-04-24T08:11:11.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 shutdown_connections 2024-04-24T08:11:11.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01640396b0 0x7f016403bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:11.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 --2- 172.21.15.146:0/1837416501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0180100e40 0x7f018019e220 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:11.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.696+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 >> 172.21.15.146:0/1837416501 conn(0x7f01800fca60 msgr2=0x7f01800fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:11.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.697+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 shutdown_connections 2024-04-24T08:11:11.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:11.697+0000 7f0184a00700 1 -- 172.21.15.146:0/1837416501 wait complete. 2024-04-24T08:11:12.478 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-24T08:11:12.478 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-24T08:11:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:12 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1837416501' entity='client.admin' 2024-04-24T08:11:13.135 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:11:14.897 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:14.897 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:14.897 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:11:14.897 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:14.898 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:14.898 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:15.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.714+0000 7f2d98600700 1 -- 172.21.15.146:0/3430223225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 msgr2=0x7f2d90109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:15.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.714+0000 7f2d98600700 1 --2- 172.21.15.146:0/3430223225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d90109120 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009910 tx=0x7f2d84009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:15.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.715+0000 7f2d98600700 1 -- 172.21.15.146:0/3430223225 shutdown_connections 2024-04-24T08:11:15.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.715+0000 7f2d98600700 1 --2- 172.21.15.146:0/3430223225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d90109120 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:15.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.715+0000 7f2d98600700 1 -- 172.21.15.146:0/3430223225 >> 172.21.15.146:0/3430223225 conn(0x7f2d90077de0 msgr2=0x7f2d9007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:15.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.715+0000 7f2d98600700 1 -- 172.21.15.146:0/3430223225 shutdown_connections 2024-04-24T08:11:15.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.715+0000 7f2d98600700 1 -- 172.21.15.146:0/3430223225 wait complete. 2024-04-24T08:11:15.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.716+0000 7f2d98600700 1 Processor -- start 2024-04-24T08:11:15.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.716+0000 7f2d98600700 1 -- start start 2024-04-24T08:11:15.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.717+0000 7f2d98600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d9019e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.717+0000 7f2d98600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9010eb70 con 0x7f2d90108d00 2024-04-24T08:11:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.717+0000 7f2d95c00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d9019e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.717+0000 7f2d95c00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d9019e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35038/0 (socket says 172.21.15.146:35038) 2024-04-24T08:11:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.717+0000 7f2d95c00700 1 -- 172.21.15.146:0/1847254675 learned_addr learned my addr 172.21.15.146:0/1847254675 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.717+0000 7f2d95c00700 1 -- 172.21.15.146:0/1847254675 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9019e780 con 0x7f2d90108d00 2024-04-24T08:11:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.718+0000 7f2d95c00700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d9019e240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d84005950 tx=0x7f2d84005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.718+0000 7f2d8e000700 1 -- 172.21.15.146:0/1847254675 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d84027070 con 0x7f2d90108d00 2024-04-24T08:11:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.718+0000 7f2d8e000700 1 -- 172.21.15.146:0/1847254675 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d84022470 con 0x7f2d90108d00 2024-04-24T08:11:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.718+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9019ea10 con 0x7f2d90108d00 2024-04-24T08:11:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.718+0000 7f2d8e000700 1 -- 172.21.15.146:0/1847254675 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d8400f5d0 con 0x7f2d90108d00 2024-04-24T08:11:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.718+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d901a1700 con 0x7f2d90108d00 2024-04-24T08:11:15.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.719+0000 7f2d8e000700 1 -- 172.21.15.146:0/1847254675 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f2d8402a070 con 0x7f2d90108d00 2024-04-24T08:11:15.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.719+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d9010f2f0 con 0x7f2d90108d00 2024-04-24T08:11:15.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.720+0000 7f2d8e000700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d780396b0 0x7f2d7803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:15.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.720+0000 7f2d8e000700 1 -- 172.21.15.146:0/1847254675 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2d8401e030 con 0x7f2d90108d00 2024-04-24T08:11:15.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.720+0000 7f2d95200700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d780396b0 0x7f2d7803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:15.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.721+0000 7f2d95200700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d780396b0 0x7f2d7803bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c006fd0 tx=0x7f2d7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:15.727 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.725+0000 7f2d8e000700 1 -- 172.21.15.146:0/1847254675 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d84013070 con 0x7f2d90108d00 2024-04-24T08:11:15.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.952+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f2d9002e0f0 con 0x7f2d780396b0 2024-04-24T08:11:15.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.959+0000 7f2d8e000700 1 -- 172.21.15.146:0/1847254675 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2d9002e0f0 con 0x7f2d780396b0 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.962+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d780396b0 msgr2=0x7f2d7803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.962+0000 7f2d98600700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d780396b0 0x7f2d7803bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c006fd0 tx=0x7f2d7c006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.962+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 msgr2=0x7f2d9019e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.962+0000 7f2d98600700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d9019e240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d84005950 tx=0x7f2d84005030 comp rx=0 tx=0).stop 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.963+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 shutdown_connections 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.963+0000 7f2d98600700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d780396b0 0x7f2d7803bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.963+0000 7f2d98600700 1 --2- 172.21.15.146:0/1847254675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d90108d00 0x7f2d9019e240 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.963+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 >> 172.21.15.146:0/1847254675 conn(0x7f2d90077de0 msgr2=0x7f2d90078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.963+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 shutdown_connections 2024-04-24T08:11:15.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:15.963+0000 7f2d98600700 1 -- 172.21.15.146:0/1847254675 wait complete. 2024-04-24T08:11:16.743 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-04-24T08:11:16.744 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:11:16.744 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-04-24T08:11:16.762 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:11:16.763 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:16.821 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-04-24T08:11:16.821 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch host add smithi153 2024-04-24T08:11:17.220 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:11:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:16 smithi146 ceph-mon[27805]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:16 smithi146 ceph-mon[27805]: Updating smithi146:/etc/ceph/ceph.conf 2024-04-24T08:11:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:16 smithi146 ceph-mon[27805]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:18.227 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:17 smithi146 ceph-mon[27805]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:11:18.228 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:17 smithi146 ceph-mon[27805]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:19.077 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:18 smithi146 ceph-mon[27805]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:11:19.077 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:18 smithi146 ceph-mon[27805]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:19.077 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:19.078 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:19.078 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:19.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.248+0000 7f4ab0200700 1 -- 172.21.15.146:0/2053304947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 msgr2=0x7f4aa8107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.248+0000 7f4ab0200700 1 --2- 172.21.15.146:0/2053304947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa8107880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009910 tx=0x7f4a9c009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:19.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.248+0000 7f4ab0200700 1 -- 172.21.15.146:0/2053304947 shutdown_connections 2024-04-24T08:11:19.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.248+0000 7f4ab0200700 1 --2- 172.21.15.146:0/2053304947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa8107880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.248+0000 7f4ab0200700 1 -- 172.21.15.146:0/2053304947 >> 172.21.15.146:0/2053304947 conn(0x7f4aa80fcf20 msgr2=0x7f4aa80ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:19.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.248+0000 7f4ab0200700 1 -- 172.21.15.146:0/2053304947 shutdown_connections 2024-04-24T08:11:19.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.248+0000 7f4ab0200700 1 -- 172.21.15.146:0/2053304947 wait complete. 2024-04-24T08:11:19.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.250+0000 7f4ab0200700 1 Processor -- start 2024-04-24T08:11:19.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.250+0000 7f4ab0200700 1 -- start start 2024-04-24T08:11:19.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.250+0000 7f4ab0200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa80fd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.250+0000 7f4ab0200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa810ba50 con 0x7f4aa81074a0 2024-04-24T08:11:19.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.251+0000 7f4aad800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa80fd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.251+0000 7f4aad800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa80fd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35068/0 (socket says 172.21.15.146:35068) 2024-04-24T08:11:19.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.251+0000 7f4aad800700 1 -- 172.21.15.146:0/3240464129 learned_addr learned my addr 172.21.15.146:0/3240464129 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.251+0000 7f4aad800700 1 -- 172.21.15.146:0/3240464129 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa80ff2c0 con 0x7f4aa81074a0 2024-04-24T08:11:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.251+0000 7f4aad800700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa80fd3d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c006010 tx=0x7f4a9c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.252+0000 7f4aa6000700 1 -- 172.21.15.146:0/3240464129 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a9c026070 con 0x7f4aa81074a0 2024-04-24T08:11:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.252+0000 7f4aa6000700 1 -- 172.21.15.146:0/3240464129 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c00bc30 con 0x7f4aa81074a0 2024-04-24T08:11:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.252+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa80fd970 con 0x7f4aa81074a0 2024-04-24T08:11:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.252+0000 7f4aa6000700 1 -- 172.21.15.146:0/3240464129 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a9c0039d0 con 0x7f4aa81074a0 2024-04-24T08:11:19.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.252+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa80fde30 con 0x7f4aa81074a0 2024-04-24T08:11:19.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.253+0000 7f4aa6000700 1 -- 172.21.15.146:0/3240464129 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f4a9c029070 con 0x7f4aa81074a0 2024-04-24T08:11:19.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.253+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa8102b50 con 0x7f4aa81074a0 2024-04-24T08:11:19.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.253+0000 7f4aa6000700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4a90039650 0x7f4a9003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.253+0000 7f4aa6000700 1 -- 172.21.15.146:0/3240464129 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a9c01e070 con 0x7f4aa81074a0 2024-04-24T08:11:19.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.254+0000 7f4aace00700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4a90039650 0x7f4a9003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.254+0000 7f4aace00700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4a90039650 0x7f4a9003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a94006fd0 tx=0x7f4a94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.259+0000 7f4aa6000700 1 -- 172.21.15.146:0/3240464129 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a9c013070 con 0x7f4aa81074a0 2024-04-24T08:11:19.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:19.491+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) -- 0x7f4aa80fe900 con 0x7f4a90039650 2024-04-24T08:11:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:21 smithi146 ceph-mon[27805]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:21 smithi146 ceph-mon[27805]: Deploying cephadm binary to smithi153 2024-04-24T08:11:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:21 smithi146 ceph-mon[27805]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:23.640 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:23 smithi146 ceph-mon[27805]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.130+0000 7f4aa6000700 1 -- 172.21.15.146:0/3240464129 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4aa80fe900 con 0x7f4a90039650 2024-04-24T08:11:24.132 INFO:teuthology.orchestra.run.smithi146.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-04-24T08:11:24.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.133+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4a90039650 msgr2=0x7f4a9003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.133+0000 7f4ab0200700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4a90039650 0x7f4a9003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4a94006fd0 tx=0x7f4a94006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:24.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.133+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 msgr2=0x7f4aa80fd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.133+0000 7f4ab0200700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa80fd3d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c006010 tx=0x7f4a9c005030 comp rx=0 tx=0).stop 2024-04-24T08:11:24.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.134+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 shutdown_connections 2024-04-24T08:11:24.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.134+0000 7f4ab0200700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4a90039650 0x7f4a9003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.134+0000 7f4ab0200700 1 --2- 172.21.15.146:0/3240464129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa81074a0 0x7f4aa80fd3d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.134+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 >> 172.21.15.146:0/3240464129 conn(0x7f4aa80fcf20 msgr2=0x7f4aa806a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:24.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.134+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 shutdown_connections 2024-04-24T08:11:24.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:24.134+0000 7f4ab0200700 1 -- 172.21.15.146:0/3240464129 wait complete. 2024-04-24T08:11:25.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T08:11:25.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:25 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:25.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:25 smithi146 ceph-mon[27805]: Added host smithi153 2024-04-24T08:11:25.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:25 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:25.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:25 smithi146 ceph-mon[27805]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:25.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:25 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:25.570 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:11:26.732 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:27.364 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:27 smithi146 ceph-mon[27805]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:27.364 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:27.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.529+0000 7f9bae000700 1 -- 172.21.15.146:0/1472070331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 msgr2=0x7f9ba810de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.529+0000 7f9bae000700 1 --2- 172.21.15.146:0/1472070331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba810de70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009910 tx=0x7f9b98009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:27.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.529+0000 7f9bae000700 1 -- 172.21.15.146:0/1472070331 shutdown_connections 2024-04-24T08:11:27.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.529+0000 7f9bae000700 1 --2- 172.21.15.146:0/1472070331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba810de70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.529+0000 7f9bae000700 1 -- 172.21.15.146:0/1472070331 >> 172.21.15.146:0/1472070331 conn(0x7f9ba8078350 msgr2=0x7f9ba807a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:27.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.530+0000 7f9bae000700 1 -- 172.21.15.146:0/1472070331 shutdown_connections 2024-04-24T08:11:27.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.530+0000 7f9bae000700 1 -- 172.21.15.146:0/1472070331 wait complete. 2024-04-24T08:11:27.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.531+0000 7f9bae000700 1 Processor -- start 2024-04-24T08:11:27.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.531+0000 7f9bae000700 1 -- start start 2024-04-24T08:11:27.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.531+0000 7f9bae000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.531+0000 7f9bae000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8113150 con 0x7f9ba810da90 2024-04-24T08:11:27.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.531+0000 7f9ba7400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.531+0000 7f9ba7400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58544/0 (socket says 172.21.15.146:58544) 2024-04-24T08:11:27.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.531+0000 7f9ba7400700 1 -- 172.21.15.146:0/510657801 learned_addr learned my addr 172.21.15.146:0/510657801 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:27.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.532+0000 7f9ba7400700 1 -- 172.21.15.146:0/510657801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba819eb50 con 0x7f9ba810da90 2024-04-24T08:11:27.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.532+0000 7f9ba7400700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba819e610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9b98006010 tx=0x7f9b98005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:27.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.532+0000 7f9b9fe00700 1 -- 172.21.15.146:0/510657801 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b98026070 con 0x7f9ba810da90 2024-04-24T08:11:27.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.532+0000 7f9b9fe00700 1 -- 172.21.15.146:0/510657801 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9800bbd0 con 0x7f9ba810da90 2024-04-24T08:11:27.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.532+0000 7f9b9fe00700 1 -- 172.21.15.146:0/510657801 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b980039d0 con 0x7f9ba810da90 2024-04-24T08:11:27.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.532+0000 7f9bae000700 1 -- 172.21.15.146:0/510657801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba819ee40 con 0x7f9ba810da90 2024-04-24T08:11:27.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.532+0000 7f9bae000700 1 -- 172.21.15.146:0/510657801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba81a2ee0 con 0x7f9ba810da90 2024-04-24T08:11:27.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.534+0000 7f9b9fe00700 1 -- 172.21.15.146:0/510657801 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f9b98029070 con 0x7f9ba810da90 2024-04-24T08:11:27.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.534+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba810a1e0 con 0x7f9ba810da90 2024-04-24T08:11:27.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.534+0000 7f9b9fe00700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9b8c0396b0 0x7f9b8c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.534+0000 7f9b9fe00700 1 -- 172.21.15.146:0/510657801 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9b9801e070 con 0x7f9ba810da90 2024-04-24T08:11:27.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.534+0000 7f9ba6a00700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9b8c0396b0 0x7f9b8c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.535+0000 7f9ba6a00700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9b8c0396b0 0x7f9b8c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9b94006fd0 tx=0x7f9b94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:27.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.540+0000 7f9b9fe00700 1 -- 172.21.15.146:0/510657801 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b98013070 con 0x7f9ba810da90 2024-04-24T08:11:27.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.773+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ba8064150 con 0x7f9b8c0396b0 2024-04-24T08:11:27.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.775+0000 7f9b9fe00700 1 -- 172.21.15.146:0/510657801 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f9ba8064150 con 0x7f9b8c0396b0 2024-04-24T08:11:27.776 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:27.776 INFO:teuthology.orchestra.run.smithi146.stdout:[{"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-04-24T08:11:27.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.777+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9b8c0396b0 msgr2=0x7f9b8c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.777+0000 7f9b9d600700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9b8c0396b0 0x7f9b8c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9b94006fd0 tx=0x7f9b94006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:27.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.777+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 msgr2=0x7f9ba819e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.777+0000 7f9b9d600700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba819e610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9b98006010 tx=0x7f9b98005030 comp rx=0 tx=0).stop 2024-04-24T08:11:27.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.778+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 shutdown_connections 2024-04-24T08:11:27.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.778+0000 7f9b9d600700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9b8c0396b0 0x7f9b8c03bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.778+0000 7f9b9d600700 1 --2- 172.21.15.146:0/510657801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba810da90 0x7f9ba819e610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.778+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 >> 172.21.15.146:0/510657801 conn(0x7f9ba8078350 msgr2=0x7f9ba81105d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:27.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.778+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 shutdown_connections 2024-04-24T08:11:27.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:27.778+0000 7f9b9d600700 1 -- 172.21.15.146:0/510657801 wait complete. 2024-04-24T08:11:28.417 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-04-24T08:11:28.417 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:11:28.417 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-04-24T08:11:28.435 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:11:28.435 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:28.493 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-04-24T08:11:28.493 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch host add smithi177 2024-04-24T08:11:28.920 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:11:29.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:29 smithi146 ceph-mon[27805]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:11:29.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:29 smithi146 ceph-mon[27805]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.915+0000 7f0ad5e00700 1 -- 172.21.15.146:0/1577911256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 msgr2=0x7f0ad0075270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.915+0000 7f0ad5e00700 1 --2- 172.21.15.146:0/1577911256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad0075270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009910 tx=0x7f0ac4009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.915+0000 7f0ad5e00700 1 -- 172.21.15.146:0/1577911256 shutdown_connections 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.915+0000 7f0ad5e00700 1 --2- 172.21.15.146:0/1577911256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad0075270 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.915+0000 7f0ad5e00700 1 -- 172.21.15.146:0/1577911256 >> 172.21.15.146:0/1577911256 conn(0x7f0ad00fde40 msgr2=0x7f0ad0100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.915+0000 7f0ad5e00700 1 -- 172.21.15.146:0/1577911256 shutdown_connections 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.915+0000 7f0ad5e00700 1 -- 172.21.15.146:0/1577911256 wait complete. 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.917+0000 7f0ad5e00700 1 Processor -- start 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.917+0000 7f0ad5e00700 1 -- start start 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.917+0000 7f0ad5e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.917+0000 7f0ad5e00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad0112fb0 con 0x7f0ad0074e90 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.917+0000 7f0ad4a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad019e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.917+0000 7f0ad4a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad019e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:58556/0 (socket says 172.21.15.146:58556) 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.917+0000 7f0ad4a00700 1 -- 172.21.15.146:0/3068746263 learned_addr learned my addr 172.21.15.146:0/3068746263 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.918+0000 7f0ad4a00700 1 -- 172.21.15.146:0/3068746263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad019e9f0 con 0x7f0ad0074e90 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.918+0000 7f0ad4a00700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad019e4b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4006010 tx=0x7f0ac4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.918+0000 7f0acd600700 1 -- 172.21.15.146:0/3068746263 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ac4026070 con 0x7f0ad0074e90 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.918+0000 7f0acd600700 1 -- 172.21.15.146:0/3068746263 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac400bc30 con 0x7f0ad0074e90 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.918+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad019ec80 con 0x7f0ad0074e90 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.918+0000 7f0acd600700 1 -- 172.21.15.146:0/3068746263 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ac4003ab0 con 0x7f0ad0074e90 2024-04-24T08:11:30.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.919+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad019f0e0 con 0x7f0ad0074e90 2024-04-24T08:11:30.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.920+0000 7f0acd600700 1 -- 172.21.15.146:0/3068746263 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f0ac4029070 con 0x7f0ad0074e90 2024-04-24T08:11:30.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.920+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad00765d0 con 0x7f0ad0074e90 2024-04-24T08:11:30.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.920+0000 7f0acd600700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ab80396b0 0x7f0ab803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.920+0000 7f0acd600700 1 -- 172.21.15.146:0/3068746263 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0ac401e070 con 0x7f0ad0074e90 2024-04-24T08:11:30.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.921+0000 7f0acfe00700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ab80396b0 0x7f0ab803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.921+0000 7f0acfe00700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ab80396b0 0x7f0ab803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0abc006fd0 tx=0x7f0abc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:30.929+0000 7f0acd600700 1 -- 172.21.15.146:0/3068746263 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ac4013070 con 0x7f0ad0074e90 2024-04-24T08:11:31.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:31.166+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}) -- 0x7f0ad019f810 con 0x7f0ab80396b0 2024-04-24T08:11:31.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:31.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:31.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:11:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:11:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:31.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:31 smithi146 ceph-mon[27805]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:11:32.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:32 smithi146 ceph-mon[27805]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:32.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:32 smithi146 ceph-mon[27805]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:11:32.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:32 smithi146 ceph-mon[27805]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:32.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:32.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:32.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:33 smithi146 ceph-mon[27805]: Deploying cephadm binary to smithi177 2024-04-24T08:11:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:33 smithi146 ceph-mon[27805]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:35 smithi146 ceph-mon[27805]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:35.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.966+0000 7f0acd600700 1 -- 172.21.15.146:0/3068746263 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f0ad019f810 con 0x7f0ab80396b0 2024-04-24T08:11:35.968 INFO:teuthology.orchestra.run.smithi146.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-04-24T08:11:35.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ab80396b0 msgr2=0x7f0ab803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ab80396b0 0x7f0ab803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0abc006fd0 tx=0x7f0abc006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:35.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 msgr2=0x7f0ad019e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad019e4b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4006010 tx=0x7f0ac4005050 comp rx=0 tx=0).stop 2024-04-24T08:11:35.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 shutdown_connections 2024-04-24T08:11:35.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ab80396b0 0x7f0ab803bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 --2- 172.21.15.146:0/3068746263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ad0074e90 0x7f0ad019e4b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.970+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 >> 172.21.15.146:0/3068746263 conn(0x7f0ad00fde40 msgr2=0x7f0ad00fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:35.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.972+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 shutdown_connections 2024-04-24T08:11:35.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:35.972+0000 7f0ad5e00700 1 -- 172.21.15.146:0/3068746263 wait complete. 2024-04-24T08:11:36.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T08:11:37.195 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:11:37.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:37.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:36 smithi146 ceph-mon[27805]: Added host smithi177 2024-04-24T08:11:37.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:37.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:36 smithi146 ceph-mon[27805]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:37.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:38.416 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:38 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:39.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.198+0000 7f94c1a00700 1 -- 172.21.15.146:0/3229672088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 msgr2=0x7f94bc077400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.198+0000 7f94c1a00700 1 --2- 172.21.15.146:0/3229672088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc077400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009910 tx=0x7f94ac009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:39.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.198+0000 7f94c1a00700 1 -- 172.21.15.146:0/3229672088 shutdown_connections 2024-04-24T08:11:39.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.198+0000 7f94c1a00700 1 --2- 172.21.15.146:0/3229672088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc077400 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.198+0000 7f94c1a00700 1 -- 172.21.15.146:0/3229672088 >> 172.21.15.146:0/3229672088 conn(0x7f94bc100000 msgr2=0x7f94bc102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:39.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.198+0000 7f94c1a00700 1 -- 172.21.15.146:0/3229672088 shutdown_connections 2024-04-24T08:11:39.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.199+0000 7f94c1a00700 1 -- 172.21.15.146:0/3229672088 wait complete. 2024-04-24T08:11:39.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.200+0000 7f94c1a00700 1 Processor -- start 2024-04-24T08:11:39.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.200+0000 7f94c1a00700 1 -- start start 2024-04-24T08:11:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.200+0000 7f94c1a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.200+0000 7f94c1a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc112fa0 con 0x7f94bc077020 2024-04-24T08:11:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.200+0000 7f94bbe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.200+0000 7f94bbe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc19e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48300/0 (socket says 172.21.15.146:48300) 2024-04-24T08:11:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.200+0000 7f94bbe00700 1 -- 172.21.15.146:0/796136056 learned_addr learned my addr 172.21.15.146:0/796136056 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.201+0000 7f94bbe00700 1 -- 172.21.15.146:0/796136056 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc19e9c0 con 0x7f94bc077020 2024-04-24T08:11:39.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.201+0000 7f94bbe00700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc19e480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94ac006010 tx=0x7f94ac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:39.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.201+0000 7f94b8c00700 1 -- 172.21.15.146:0/796136056 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac026070 con 0x7f94bc077020 2024-04-24T08:11:39.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.201+0000 7f94b8c00700 1 -- 172.21.15.146:0/796136056 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94ac00bc30 con 0x7f94bc077020 2024-04-24T08:11:39.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.201+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc19ec50 con 0x7f94bc077020 2024-04-24T08:11:39.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.201+0000 7f94b8c00700 1 -- 172.21.15.146:0/796136056 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac0039d0 con 0x7f94bc077020 2024-04-24T08:11:39.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.202+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc19f0b0 con 0x7f94bc077020 2024-04-24T08:11:39.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.203+0000 7f94b8c00700 1 -- 172.21.15.146:0/796136056 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f94ac029070 con 0x7f94bc077020 2024-04-24T08:11:39.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.203+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc078760 con 0x7f94bc077020 2024-04-24T08:11:39.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.203+0000 7f94b8c00700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f94a0039650 0x7f94a003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.203+0000 7f94b8c00700 1 -- 172.21.15.146:0/796136056 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f94ac01e070 con 0x7f94bc077020 2024-04-24T08:11:39.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.203+0000 7f94bb400700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f94a0039650 0x7f94a003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.204+0000 7f94bb400700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f94a0039650 0x7f94a003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006fd0 tx=0x7f94a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:39.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.209+0000 7f94b8c00700 1 -- 172.21.15.146:0/796136056 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94ac013070 con 0x7f94bc077020 2024-04-24T08:11:39.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.442+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f94bc19f7e0 con 0x7f94a0039650 2024-04-24T08:11:39.444 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:39 smithi146 ceph-mon[27805]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:39.445 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:39.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.443+0000 7f94b8c00700 1 -- 172.21.15.146:0/796136056 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f94bc19f7e0 con 0x7f94a0039650 2024-04-24T08:11:39.445 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:11:39.445 INFO:teuthology.orchestra.run.smithi146.stdout:[{"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-04-24T08:11:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f94a0039650 msgr2=0x7f94a003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f94a0039650 0x7f94a003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006fd0 tx=0x7f94a4006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 msgr2=0x7f94bc19e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc19e480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f94ac006010 tx=0x7f94ac005030 comp rx=0 tx=0).stop 2024-04-24T08:11:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 shutdown_connections 2024-04-24T08:11:39.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f94a0039650 0x7f94a003bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 --2- 172.21.15.146:0/796136056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94bc077020 0x7f94bc19e480 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.446+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 >> 172.21.15.146:0/796136056 conn(0x7f94bc100000 msgr2=0x7f94bc1003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:39.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.447+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 shutdown_connections 2024-04-24T08:11:39.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:39.447+0000 7f94c1a00700 1 -- 172.21.15.146:0/796136056 wait complete. 2024-04-24T08:11:40.105 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-24T08:11:40.105 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-24T08:11:40.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:40 smithi146 ceph-mon[27805]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:11:40.546 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:11:41.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:41 smithi146 ceph-mon[27805]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:42.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.493+0000 7f3197600700 1 -- 172.21.15.146:0/392914410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 msgr2=0x7f3190103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.493+0000 7f3197600700 1 --2- 172.21.15.146:0/392914410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f3190103b30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3184009910 tx=0x7f3184009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:42.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.493+0000 7f3197600700 1 -- 172.21.15.146:0/392914410 shutdown_connections 2024-04-24T08:11:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.493+0000 7f3197600700 1 --2- 172.21.15.146:0/392914410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f3190103b30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.493+0000 7f3197600700 1 -- 172.21.15.146:0/392914410 >> 172.21.15.146:0/392914410 conn(0x7f31900780c0 msgr2=0x7f31900ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.494+0000 7f3197600700 1 -- 172.21.15.146:0/392914410 shutdown_connections 2024-04-24T08:11:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.494+0000 7f3197600700 1 -- 172.21.15.146:0/392914410 wait complete. 2024-04-24T08:11:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.495+0000 7f3197600700 1 Processor -- start 2024-04-24T08:11:42.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.495+0000 7f3197600700 1 -- start start 2024-04-24T08:11:42.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.496+0000 7f3197600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f319019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.496+0000 7f3197600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f319010cb70 con 0x7f3190103750 2024-04-24T08:11:42.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.496+0000 7f3194c00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f319019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.496+0000 7f3194c00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f319019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56986/0 (socket says 172.21.15.146:56986) 2024-04-24T08:11:42.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.496+0000 7f3194c00700 1 -- 172.21.15.146:0/1685016343 learned_addr learned my addr 172.21.15.146:0/1685016343 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:11:42.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.496+0000 7f3194c00700 1 -- 172.21.15.146:0/1685016343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319019a7a0 con 0x7f3190103750 2024-04-24T08:11:42.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.497+0000 7f3194c00700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f319019a260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3184006010 tx=0x7f3184005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.497+0000 7f318d600700 1 -- 172.21.15.146:0/1685016343 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3184026070 con 0x7f3190103750 2024-04-24T08:11:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.497+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319019aa30 con 0x7f3190103750 2024-04-24T08:11:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.497+0000 7f318d600700 1 -- 172.21.15.146:0/1685016343 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f318400bc30 con 0x7f3190103750 2024-04-24T08:11:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.497+0000 7f318d600700 1 -- 172.21.15.146:0/1685016343 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31840039a0 con 0x7f3190103750 2024-04-24T08:11:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.497+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f319019ae90 con 0x7f3190103750 2024-04-24T08:11:42.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.498+0000 7f318d600700 1 -- 172.21.15.146:0/1685016343 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f3184029070 con 0x7f3190103750 2024-04-24T08:11:42.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.499+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f319006a740 con 0x7f3190103750 2024-04-24T08:11:42.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.499+0000 7f318d600700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3178039650 0x7f317803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.499+0000 7f318d600700 1 -- 172.21.15.146:0/1685016343 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f318401e070 con 0x7f3190103750 2024-04-24T08:11:42.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.499+0000 7f318fe00700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3178039650 0x7f317803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.500+0000 7f318fe00700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3178039650 0x7f317803bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f317c006fd0 tx=0x7f317c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:42.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.505+0000 7f318d600700 1 -- 172.21.15.146:0/1685016343 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3184013070 con 0x7f3190103750 2024-04-24T08:11:42.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.741+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f319019b5c0 con 0x7f3190103750 2024-04-24T08:11:42.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.960+0000 7f318d600700 1 -- 172.21.15.146:0/1685016343 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f318400bde0 con 0x7f3190103750 2024-04-24T08:11:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3178039650 msgr2=0x7f317803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3178039650 0x7f317803bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f317c006fd0 tx=0x7f317c006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 msgr2=0x7f319019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f319019a260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3184006010 tx=0x7f3184005030 comp rx=0 tx=0).stop 2024-04-24T08:11:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 shutdown_connections 2024-04-24T08:11:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3178039650 0x7f317803bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 --2- 172.21.15.146:0/1685016343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3190103750 0x7f319019a260 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 >> 172.21.15.146:0/1685016343 conn(0x7f31900780c0 msgr2=0x7f3190108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:42.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 shutdown_connections 2024-04-24T08:11:42.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:11:42.964+0000 7f3197600700 1 -- 172.21.15.146:0/1685016343 wait complete. 2024-04-24T08:11:42.967 INFO:teuthology.orchestra.run.smithi146.stderr:adjusted tunables profile to default 2024-04-24T08:11:43.238 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:43.238 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:43.238 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:43.238 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:43.239 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:11:43.239 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:43.239 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:43.239 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:11:43.239 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:11:43.239 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:43.239 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:42 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1685016343' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T08:11:43.696 INFO:tasks.cephadm:Adding mon.a on smithi146 2024-04-24T08:11:43.696 INFO:tasks.cephadm:Adding mon.b on smithi153 2024-04-24T08:11:43.696 INFO:tasks.cephadm:Adding mon.c on smithi177 2024-04-24T08:11:43.697 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi146:172.21.15.146=a;smithi153:172.21.15.153=b;smithi177:172.21.15.177=c' 2024-04-24T08:11:43.920 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:11:44.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:43 smithi146 ceph-mon[27805]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:44.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:43 smithi146 ceph-mon[27805]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:11:44.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:43 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1685016343' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T08:11:44.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:43 smithi146 ceph-mon[27805]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:11:44.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:44.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:44.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:45.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:44 smithi146 ceph-mon[27805]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:45.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 -- 172.21.15.177:0/3407910364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 msgr2=0x7ff6f8103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 --2- 172.21.15.177:0/3407910364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8103af0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec009910 tx=0x7ff6ec009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:45.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 -- 172.21.15.177:0/3407910364 shutdown_connections 2024-04-24T08:11:45.386 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 --2- 172.21.15.177:0/3407910364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8103af0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 -- 172.21.15.177:0/3407910364 >> 172.21.15.177:0/3407910364 conn(0x7ff6f80fcf40 msgr2=0x7ff6f80ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 -- 172.21.15.177:0/3407910364 shutdown_connections 2024-04-24T08:11:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 -- 172.21.15.177:0/3407910364 wait complete. 2024-04-24T08:11:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 Processor -- start 2024-04-24T08:11:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.385+0000 7ff6ff187700 1 -- start start 2024-04-24T08:11:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.386+0000 7ff6ff187700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.386+0000 7ff6ff187700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f810cb90 con 0x7ff6f8103710 2024-04-24T08:11:45.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.386+0000 7ff6fcf23700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.386+0000 7ff6fcf23700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.177:57734/0 (socket says 172.21.15.177:57734) 2024-04-24T08:11:45.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.386+0000 7ff6fcf23700 1 -- 172.21.15.177:0/2294330807 learned_addr learned my addr 172.21.15.177:0/2294330807 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:11:45.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.387+0000 7ff6fcf23700 1 -- 172.21.15.177:0/2294330807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f81985d0 con 0x7ff6f8103710 2024-04-24T08:11:45.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.387+0000 7ff6fcf23700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8198090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec006010 tx=0x7ff6ec005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.387+0000 7ff6f5ffb700 1 -- 172.21.15.177:0/2294330807 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6ec026070 con 0x7ff6f8103710 2024-04-24T08:11:45.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.387+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f8198860 con 0x7ff6f8103710 2024-04-24T08:11:45.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.388+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f8198cc0 con 0x7ff6f8103710 2024-04-24T08:11:45.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.388+0000 7ff6f5ffb700 1 -- 172.21.15.177:0/2294330807 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6ec004830 con 0x7ff6f8103710 2024-04-24T08:11:45.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.388+0000 7ff6f5ffb700 1 -- 172.21.15.177:0/2294330807 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6ec030060 con 0x7ff6f8103710 2024-04-24T08:11:45.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.388+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6dc005330 con 0x7ff6f8103710 2024-04-24T08:11:45.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.389+0000 7ff6f5ffb700 1 -- 172.21.15.177:0/2294330807 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7ff6ec029070 con 0x7ff6f8103710 2024-04-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.389+0000 7ff6f5ffb700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6e8039650 0x7ff6e803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.389+0000 7ff6f5ffb700 1 -- 172.21.15.177:0/2294330807 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff6ec01e070 con 0x7ff6f8103710 2024-04-24T08:11:45.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.390+0000 7ff6f7fff700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6e8039650 0x7ff6e803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.391+0000 7ff6f7fff700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6e8039650 0x7ff6e803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4006fd0 tx=0x7ff6e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.405 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.404+0000 7ff6f5ffb700 1 -- 172.21.15.177:0/2294330807 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6ec013070 con 0x7ff6f8103710 2024-04-24T08:11:45.618 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.617+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi146:172.21.15.146=a;smithi153:172.21.15.153=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}) -- 0x7ff6dc000c80 con 0x7ff6e8039650 2024-04-24T08:11:45.633 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.632+0000 7ff6f5ffb700 1 -- 172.21.15.177:0/2294330807 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff6dc000c80 con 0x7ff6e8039650 2024-04-24T08:11:45.633 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-04-24T08:11:45.635 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.634+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6e8039650 msgr2=0x7ff6e803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.635 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.634+0000 7ff6ff187700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6e8039650 0x7ff6e803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4006fd0 tx=0x7ff6e4006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.634+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 msgr2=0x7ff6f8198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.634+0000 7ff6ff187700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8198090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec006010 tx=0x7ff6ec005050 comp rx=0 tx=0).stop 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.635+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 shutdown_connections 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.635+0000 7ff6ff187700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6e8039650 0x7ff6e803bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.635+0000 7ff6ff187700 1 --2- 172.21.15.177:0/2294330807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6f8103710 0x7ff6f8198090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.635+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 >> 172.21.15.177:0/2294330807 conn(0x7ff6f80fcf40 msgr2=0x7ff6f81064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.635+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 shutdown_connections 2024-04-24T08:11:45.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:45.635+0000 7ff6ff187700 1 -- 172.21.15.177:0/2294330807 wait complete. 2024-04-24T08:11:46.146 DEBUG:teuthology.orchestra.run.smithi153:mon.b> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.b.service 2024-04-24T08:11:46.149 DEBUG:teuthology.orchestra.run.smithi177:mon.c> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.c.service 2024-04-24T08:11:46.151 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T08:11:46.151 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi146:172.21.15.146=a;smithi153:172.21.15.153=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: Saving service mon spec with placement smithi146:172.21.15.146=a;smithi153:172.21.15.153=b;smithi177:172.21.15.177=c;count:3 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: Deploying daemon mon.c on smithi177 2024-04-24T08:11:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:46 smithi146 ceph-mon[27805]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:48.059 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:11:49.233 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:49 smithi177 systemd[1]: Starting Ceph mon.c for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:11:49.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:49 smithi146 ceph-mon[27805]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:49.814 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 -- 172.21.15.177:0/3091212968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 msgr2=0x7fced4107920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:49.814 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 --2- 172.21.15.177:0/3091212968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4107920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009910 tx=0x7fcec4009c20 comp rx=0 tx=0).stop 2024-04-24T08:11:49.814 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 -- 172.21.15.177:0/3091212968 shutdown_connections 2024-04-24T08:11:49.814 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 --2- 172.21.15.177:0/3091212968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4107920 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:49.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 -- 172.21.15.177:0/3091212968 >> 172.21.15.177:0/3091212968 conn(0x7fced4077910 msgr2=0x7fced4077d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:49.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 -- 172.21.15.177:0/3091212968 shutdown_connections 2024-04-24T08:11:49.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 -- 172.21.15.177:0/3091212968 wait complete. 2024-04-24T08:11:49.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 Processor -- start 2024-04-24T08:11:49.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.813+0000 7fceda768700 1 -- start start 2024-04-24T08:11:49.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.814+0000 7fceda768700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.815 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.814+0000 7fceda768700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fced410bb60 con 0x7fced4107540 2024-04-24T08:11:49.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.814+0000 7fced3fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.814+0000 7fced3fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.177:47142/0 (socket says 172.21.15.177:47142) 2024-04-24T08:11:49.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.814+0000 7fced3fff700 1 -- 172.21.15.177:0/1040802852 learned_addr learned my addr 172.21.15.177:0/1040802852 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:11:49.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.815+0000 7fced3fff700 1 -- 172.21.15.177:0/1040802852 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced4198640 con 0x7fced4107540 2024-04-24T08:11:49.816 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.815+0000 7fced3fff700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4198100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcec4006010 tx=0x7fcec4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:49.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.816+0000 7fced17fa700 1 -- 172.21.15.177:0/1040802852 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcec4026070 con 0x7fced4107540 2024-04-24T08:11:49.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.816+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced4198930 con 0x7fced4107540 2024-04-24T08:11:49.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.816+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced419c9d0 con 0x7fced4107540 2024-04-24T08:11:49.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.816+0000 7fced17fa700 1 -- 172.21.15.177:0/1040802852 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec4004830 con 0x7fced4107540 2024-04-24T08:11:49.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.816+0000 7fced17fa700 1 -- 172.21.15.177:0/1040802852 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcec4030060 con 0x7fced4107540 2024-04-24T08:11:49.817 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.816+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec0005330 con 0x7fced4107540 2024-04-24T08:11:49.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.817+0000 7fced17fa700 1 -- 172.21.15.177:0/1040802852 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fcec4029070 con 0x7fced4107540 2024-04-24T08:11:49.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.817+0000 7fced17fa700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcebc039650 0x7fcebc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.817+0000 7fced17fa700 1 -- 172.21.15.177:0/1040802852 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcec401e070 con 0x7fced4107540 2024-04-24T08:11:49.818 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.818+0000 7fced37fe700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcebc039650 0x7fcebc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.819+0000 7fced37fe700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcebc039650 0x7fcebc03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcec8006fd0 tx=0x7fcec8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:49.833 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:49.833+0000 7fced17fa700 1 -- 172.21.15.177:0/1040802852 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcec4013070 con 0x7fced4107540 2024-04-24T08:11:50.064 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:49 smithi177 podman[33013]: 2024-04-24 08:11:49.862745173 +0000 UTC m=+0.018987072 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:11:50.064 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:49 smithi177 podman[33013]: 2024-04-24 08:11:49.961505405 +0000 UTC m=+0.117747287 container create 7230be4a4d0bde41f33bae0674f3b47ab4e2df76f667e8556f6289dafec7de54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS) 2024-04-24T08:11:50.126 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.126+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fcec00058f0 con 0x7fced4107540 2024-04-24T08:11:50.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.127+0000 7fced17fa700 1 -- 172.21.15.177:0/1040802852 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7fcec4003a00 con 0x7fced4107540 2024-04-24T08:11:50.128 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:11:50.128 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","modified":"2024-04-24T08:09:44.411147Z","created":"2024-04-24T08:09:44.411147Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcebc039650 msgr2=0x7fcebc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcebc039650 0x7fcebc03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcec8006fd0 tx=0x7fcec8006e40 comp rx=0 tx=0).stop 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 msgr2=0x7fced4198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4198100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcec4006010 tx=0x7fcec4005050 comp rx=0 tx=0).stop 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 shutdown_connections 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcebc039650 0x7fcebc03bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 --2- 172.21.15.177:0/1040802852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced4107540 0x7fced4198100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 >> 172.21.15.177:0/1040802852 conn(0x7fced4077910 msgr2=0x7fced4105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 shutdown_connections 2024-04-24T08:11:50.130 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:50.129+0000 7fceda768700 1 -- 172.21.15.177:0/1040802852 wait complete. 2024-04-24T08:11:50.131 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-04-24T08:11:50.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 podman[33013]: 2024-04-24 08:11:50.197847904 +0000 UTC m=+0.354089775 container init 7230be4a4d0bde41f33bae0674f3b47ab4e2df76f667e8556f6289dafec7de54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-24T08:11:50.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 podman[33013]: 2024-04-24 08:11:50.201453189 +0000 UTC m=+0.357695063 container start 7230be4a4d0bde41f33bae0674f3b47ab4e2df76f667e8556f6289dafec7de54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-24T08:11:50.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T08:11:50.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev), process ceph-mon, pid 2 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pidfile_write: ignore empty --pid-file 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: load: jerasure load: lrc 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: RocksDB version: 7.9.2 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Git sha 0 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Compile date 2024-04-23 21:06:33 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: DB SUMMARY 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: DB Session ID: WTA3ABR4WJ88519ST9Z5 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: CURRENT file: CURRENT 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-24T08:11:50.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.error_if_exists: 0 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.create_if_missing: 0 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.paranoid_checks: 1 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.env: 0x55f029b88cc0 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.info_log: 0x55f02ae05c20 2024-04-24T08:11:50.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T08:11:50.569 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.statistics: (nil) 2024-04-24T08:11:50.569 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.use_fsync: 0 2024-04-24T08:11:50.569 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_log_file_size: 0 2024-04-24T08:11:50.569 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T08:11:50.569 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.allow_fallocate: 1 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.use_direct_reads: 0 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.db_log_dir: 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.wal_dir: 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T08:11:50.570 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.write_buffer_manager: 0x55f02add41e0 2024-04-24T08:11:50.571 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.unordered_write: 0 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T08:11:50.572 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.row_cache: None 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.wal_filter: None 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.two_write_queues: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.wal_compression: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.atomic_flush: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T08:11:50.573 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.log_readahead_size: 0 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_background_jobs: 2 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_background_compactions: -1 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_subcompactions: 1 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T08:11:50.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_open_files: -1 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_background_flushes: -1 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Compression algorithms supported: 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kZSTD supported: 0 2024-04-24T08:11:50.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kXpressCompression supported: 0 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kZlibCompression supported: 1 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kSnappyCompression supported: 1 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kLZ4Compression supported: 1 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: kBZip2Compression supported: 0 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T08:11:50.577 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.merge_operator: 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_filter: None 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f02ae05de0) 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout: cache_index_and_filter_blocks: 1 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout: pin_top_level_index_and_filter: 1 2024-04-24T08:11:50.578 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_type: 0 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: data_block_index_type: 0 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_shortening: 1 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: checksum: 4 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: no_block_cache: 0 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache: 0x55f02ae07090 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_name: BinnedLRUCache 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_options: 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: capacity : 536870912 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: num_shard_bits : 4 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: strict_capacity_limit : 0 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: high_pri_pool_ratio: 0.000 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_compressed: (nil) 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: persistent_cache: (nil) 2024-04-24T08:11:50.579 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_size: 4096 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_size_deviation: 10 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_restart_interval: 16 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_block_restart_interval: 1 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: metadata_block_size: 4096 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: partition_filters: 0 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: use_delta_encoding: 1 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: filter_policy: bloomfilter 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: whole_key_filtering: 1 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: verify_compression: 0 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: read_amp_bytes_per_bit: 0 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: format_version: 5 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: enable_index_compression: 1 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_align: 0 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: max_auto_readahead_size: 262144 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: prepopulate_block_cache: 0 2024-04-24T08:11:50.580 INFO:journalctl@ceph.mon.c.smithi177.stdout: initial_auto_readahead_size: 8192 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression: NoCompression 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.num_levels: 7 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T08:11:50.581 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T08:11:50.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T08:11:50.583 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T08:11:50.584 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.inplace_update_support: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.bloom_locality: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.max_successive_merges: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.ttl: 2592000 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T08:11:50.585 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.enable_blob_files: false 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.min_blob_size: 0 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-24T08:11:50.586 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5e532e3c-d011-4678-a55a-1ded993ccbc0 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713946310231199, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713946310231648, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713946310, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5e532e3c-d011-4678-a55a-1ded993ccbc0", "db_session_id": "WTA3ABR4WJ88519ST9Z5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713946310231726, "job": 1, "event": "recovery_finished"} 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f02aef4000 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: DB pointer 0x55f02aede000 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** DB Stats ** 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T08:11:50.587 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T08:11:50.588 INFO:journalctl@ceph.mon.c.smithi177.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Block cache BinnedLRUCache@0x55f02ae07090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-24T08:11:50.589 INFO:journalctl@ceph.mon.c.smithi177.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: using public_addr v2:172.21.15.177:0/0 -> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: starting mon.c rank -1 at public addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] at bind addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(???) e0 preinit fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 bash[33013]: 7230be4a4d0bde41f33bae0674f3b47ab4e2df76f667e8556f6289dafec7de54 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 systemd[1]: Started Ceph mon.c for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).mds e1 new map 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout: e1 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T08:11:50.590 INFO:journalctl@ceph.mon.c.smithi177.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout: legacy client fscid: -1 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout: No filesystems configured 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T08:11:50.591 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Deploying cephadm binary to smithi153 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Added host smithi153 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.592 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.593 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Deploying cephadm binary to smithi177 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Added host smithi177 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.594 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1685016343' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1685016343' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.595 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi146:172.21.15.146=a;smithi153:172.21.15.153=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Saving service mon spec with placement smithi146:172.21.15.146=a;smithi153:172.21.15.153=b;smithi177:172.21.15.177=c;count:3 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: Deploying daemon mon.c on smithi177 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:50.596 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:50 smithi177 ceph-mon[33046]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-24T08:11:51.945 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T08:11:51.945 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T08:11:52.192 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:11:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:11:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: mon.a calling monitor election 2024-04-24T08:11:55.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:55.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: mon.c calling monitor election 2024-04-24T08:11:55.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: monmap epoch 2 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: last_changed 2024-04-24T08:11:50.264850+0000 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: min_mon_release 19 (squid) 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: election_strategy: 1 2024-04-24T08:11:55.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:11:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:11:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: fsmap 2024-04-24T08:11:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:11:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: mgrmap e13: x(active, since 58s) 2024-04-24T08:11:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: overall HEALTH_OK 2024-04-24T08:11:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:55.746 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: mon.a calling monitor election 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: mon.c calling monitor election 2024-04-24T08:11:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: monmap epoch 2 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: last_changed 2024-04-24T08:11:50.264850+0000 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: min_mon_release 19 (squid) 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: election_strategy: 1 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:11:55.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: fsmap 2024-04-24T08:11:55.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:11:55.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: mgrmap e13: x(active, since 58s) 2024-04-24T08:11:55.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: overall HEALTH_OK 2024-04-24T08:11:55.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:55.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:56.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:56.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:11:56.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:56.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:56 smithi177 ceph-mon[33046]: Deploying daemon mon.b on smithi153 2024-04-24T08:11:56.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:56.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.686+0000 7fd6374b7700 1 -- 172.21.15.177:0/3188656939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61c0035c0 msgr2=0x7fd61c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:56.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.686+0000 7fd6374b7700 1 --2- 172.21.15.177:0/3188656939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61c0035c0 0x7fd61c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd630101420 tx=0x7fd62c00feb0 comp rx=0 tx=0).stop 2024-04-24T08:11:56.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.686+0000 7fd6374b7700 1 -- 172.21.15.177:0/3188656939 shutdown_connections 2024-04-24T08:11:56.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.686+0000 7fd6374b7700 1 --2- 172.21.15.177:0/3188656939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61c0035c0 0x7fd61c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:56.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.686+0000 7fd6374b7700 1 -- 172.21.15.177:0/3188656939 >> 172.21.15.177:0/3188656939 conn(0x7fd6300fd020 msgr2=0x7fd6300ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:56.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.686+0000 7fd6374b7700 1 -- 172.21.15.177:0/3188656939 shutdown_connections 2024-04-24T08:11:56.687 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.686+0000 7fd6374b7700 1 -- 172.21.15.177:0/3188656939 wait complete. 2024-04-24T08:11:56.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd6374b7700 1 Processor -- start 2024-04-24T08:11:56.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd6374b7700 1 -- start start 2024-04-24T08:11:56.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd6374b7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 0x7fd630198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:56.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd6374b7700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd630199510 0x7fd630199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:56.688 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd6374b7700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd630109590 con 0x7fd630199510 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd6374b7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd630109410 con 0x7fd61c0035c0 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd635253700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 0x7fd630198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd635253700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 0x7fd630198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:34780/0 (socket says 172.21.15.177:34780) 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd635253700 1 -- 172.21.15.177:0/539690067 learned_addr learned my addr 172.21.15.177:0/539690067 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd635253700 1 -- 172.21.15.177:0/539690067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 msgr2=0x7fd630198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd635253700 1 -- 172.21.15.177:0/539690067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 msgr2=0x7fd630198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd635253700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 0x7fd630198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.687+0000 7fd635253700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 0x7fd630198fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:11:56.689 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.688+0000 7fd634a52700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd630199510 0x7fd630199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:56.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.688+0000 7fd634a52700 1 -- 172.21.15.177:0/539690067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 msgr2=0x7fd630198fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:11:56.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.688+0000 7fd634a52700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 0x7fd630198fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:56.690 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.688+0000 7fd634a52700 1 -- 172.21.15.177:0/539690067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6301a2040 con 0x7fd630199510 2024-04-24T08:11:56.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.689+0000 7fd634a52700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd630199510 0x7fd630199990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd62000b4d0 tx=0x7fd62000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:56.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.689+0000 7fd6267fc700 1 -- 172.21.15.177:0/539690067 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fd620011630 con 0x7fd630199510 2024-04-24T08:11:56.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.689+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6301a2270 con 0x7fd630199510 2024-04-24T08:11:56.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.689+0000 7fd6267fc700 1 -- 172.21.15.177:0/539690067 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd620011cb0 con 0x7fd630199510 2024-04-24T08:11:56.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.689+0000 7fd6267fc700 1 -- 172.21.15.177:0/539690067 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fd620010e00 con 0x7fd630199510 2024-04-24T08:11:56.691 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.689+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6301a27b0 con 0x7fd630199510 2024-04-24T08:11:56.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.690+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd630102c30 con 0x7fd630199510 2024-04-24T08:11:56.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.690+0000 7fd6267fc700 1 -- 172.21.15.177:0/539690067 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fd620018020 con 0x7fd630199510 2024-04-24T08:11:56.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.691+0000 7fd6267fc700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd6140393f0 0x7fd61403b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:56.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.691+0000 7fd6267fc700 1 -- 172.21.15.177:0/539690067 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd62004c3c0 con 0x7fd630199510 2024-04-24T08:11:56.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.691+0000 7fd635253700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd6140393f0 0x7fd61403b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:56.692 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.692+0000 7fd635253700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd6140393f0 0x7fd61403b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00fa60 tx=0x7fd62c005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:56.709 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.708+0000 7fd6267fc700 1 -- 172.21.15.177:0/539690067 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd62001f020 con 0x7fd630199510 2024-04-24T08:11:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:11:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:11:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:56 smithi146 ceph-mon[27805]: Deploying daemon mon.b on smithi153 2024-04-24T08:11:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:11:56.987 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.985+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd63002dfe0 con 0x7fd630199510 2024-04-24T08:11:56.988 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.986+0000 7fd6267fc700 1 -- 172.21.15.177:0/539690067 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fd620010510 con 0x7fd630199510 2024-04-24T08:11:56.988 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:11:56.988 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","modified":"2024-04-24T08:11:50.264850Z","created":"2024-04-24T08:09:44.411147Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T08:11:56.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.988+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd6140393f0 msgr2=0x7fd61403b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:56.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.988+0000 7fd6374b7700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd6140393f0 0x7fd61403b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00fa60 tx=0x7fd62c005c20 comp rx=0 tx=0).stop 2024-04-24T08:11:56.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.988+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd630199510 msgr2=0x7fd630199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:56.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd630199510 0x7fd630199990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd62000b4d0 tx=0x7fd62000b890 comp rx=0 tx=0).stop 2024-04-24T08:11:56.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 shutdown_connections 2024-04-24T08:11:56.991 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd61c0035c0 0x7fd630198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:56.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd6140393f0 0x7fd61403b8b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:56.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 --2- 172.21.15.177:0/539690067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd630199510 0x7fd630199990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:56.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 >> 172.21.15.177:0/539690067 conn(0x7fd6300fd020 msgr2=0x7fd6300fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:56.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 shutdown_connections 2024-04-24T08:11:56.992 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:11:56.989+0000 7fd6374b7700 1 -- 172.21.15.177:0/539690067 wait complete. 2024-04-24T08:11:56.992 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-04-24T08:11:57.520 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:57 smithi177 ceph-mon[33046]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:57.520 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:57 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/539690067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:11:57.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:57 smithi146 ceph-mon[27805]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:57.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:57 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/539690067' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:11:57.743 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:11:57 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:11:57.265+0000 7f86efc00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T08:11:58.548 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T08:11:58.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T08:11:58.790 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:11:59.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:11:59 smithi177 ceph-mon[33046]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:11:59.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:11:59 smithi153 systemd[1]: Started Ceph mon.b for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:11:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:11:59 smithi146 ceph-mon[27805]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:04.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.660+0000 7f8806862700 1 -- 172.21.15.177:0/1662951238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e80035c0 msgr2=0x7f87e8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:04.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.660+0000 7f8806862700 1 --2- 172.21.15.177:0/1662951238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e80035c0 0x7f87e8005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8800101430 tx=0x7f87f000fe60 comp rx=0 tx=0).stop 2024-04-24T08:12:04.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.660+0000 7f8806862700 1 -- 172.21.15.177:0/1662951238 shutdown_connections 2024-04-24T08:12:04.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.661+0000 7f8806862700 1 --2- 172.21.15.177:0/1662951238 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e80035c0 0x7f87e8005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.661+0000 7f8806862700 1 -- 172.21.15.177:0/1662951238 >> 172.21.15.177:0/1662951238 conn(0x7f88000fcff0 msgr2=0x7f88000ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:04.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.661+0000 7f8806862700 1 -- 172.21.15.177:0/1662951238 shutdown_connections 2024-04-24T08:12:04.662 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.661+0000 7f8806862700 1 -- 172.21.15.177:0/1662951238 wait complete. 2024-04-24T08:12:04.664 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.661+0000 7f8806862700 1 Processor -- start 2024-04-24T08:12:04.664 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f8806862700 1 -- start start 2024-04-24T08:12:04.665 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f8806862700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 0x7f8800198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:04.665 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f8806862700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 0x7f880019cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:04.665 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f8806862700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f880019d040 0x7f880019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:04.665 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f8806862700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8800109550 con 0x7f880019d040 2024-04-24T08:12:04.665 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f8806862700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88001093d0 con 0x7f87e80035c0 2024-04-24T08:12:04.665 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f8806862700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88001096d0 con 0x7f8800198770 2024-04-24T08:12:04.665 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f87ff7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 0x7f880019cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f87ff7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 0x7f880019cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:47660/0 (socket says 172.21.15.177:47660) 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f87ff7fe700 1 -- 172.21.15.177:0/3017525636 learned_addr learned my addr 172.21.15.177:0/3017525636 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f87ff7fe700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 msgr2=0x7f880019cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f87ff7fe700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 msgr2=0x7f880019cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.662+0000 7f87ff7fe700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 0x7f880019cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f87ff7fe700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 0x7f880019cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f8804dff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f880019d040 0x7f880019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f87fffff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 0x7f8800198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f87fffff700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 msgr2=0x7f8800198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f87fffff700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 msgr2=0x7f8800198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f87fffff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 0x7f8800198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f87fffff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 0x7f8800198230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:12:04.666 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f8804dff700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 msgr2=0x7f880019cb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f8804dff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 0x7f880019cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f8804dff700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 msgr2=0x7f8800198230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f8804dff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 0x7f8800198230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f8804dff700 1 -- 172.21.15.177:0/3017525636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88001a4db0 con 0x7f880019d040 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.663+0000 7f8804dff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f880019d040 0x7f880019d4f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f87f800bd50 tx=0x7f87f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.664+0000 7f87fd7fa700 1 -- 172.21.15.177:0/3017525636 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f8019070 con 0x7f880019d040 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.664+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88001a4fe0 con 0x7f880019d040 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.664+0000 7f87fd7fa700 1 -- 172.21.15.177:0/3017525636 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87f800cd10 con 0x7f880019d040 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.664+0000 7f87fd7fa700 1 -- 172.21.15.177:0/3017525636 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f80127e0 con 0x7f880019d040 2024-04-24T08:12:04.667 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.664+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88001a5540 con 0x7f880019d040 2024-04-24T08:12:04.668 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.665+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8800064150 con 0x7f880019d040 2024-04-24T08:12:04.668 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.665+0000 7f87fd7fa700 1 -- 172.21.15.177:0/3017525636 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f87f801c070 con 0x7f880019d040 2024-04-24T08:12:04.668 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.665+0000 7f87fd7fa700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f87ec039ae0 0x7f87ec03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:04.668 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.665+0000 7f87fd7fa700 1 -- 172.21.15.177:0/3017525636 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f87f804c0f0 con 0x7f880019d040 2024-04-24T08:12:04.669 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.666+0000 7f87fffff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f87ec039ae0 0x7f87ec03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:04.669 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.667+0000 7f87fffff700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f87ec039ae0 0x7f87ec03bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f87f0000c00 tx=0x7f87f0005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:04.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.673+0000 7f87fd7fa700 1 -- 172.21.15.177:0/3017525636 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87f801ce40 con 0x7f880019d040 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: mon.c calling monitor election 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: mon.a calling monitor election 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: monmap epoch 3 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: last_changed 2024-04-24T08:11:59.462164+0000 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: min_mon_release 19 (squid) 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: election_strategy: 1 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: fsmap 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:12:04.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: mgrmap e13: x(active, since 68s) 2024-04-24T08:12:04.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: overall HEALTH_OK 2024-04-24T08:12:04.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:04.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:04.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:04.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:04.966 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.965+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f88001a58f0 con 0x7f880019d040 2024-04-24T08:12:04.967 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.966+0000 7f87fd7fa700 1 -- 172.21.15.177:0/3017525636 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f87f801bdd0 con 0x7f880019d040 2024-04-24T08:12:04.967 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-24T08:12:04.967 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":3,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","modified":"2024-04-24T08:11:59.462164Z","created":"2024-04-24T08:09:44.411147Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T08:12:04.969 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f87ec039ae0 msgr2=0x7f87ec03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:04.969 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f87ec039ae0 0x7f87ec03bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f87f0000c00 tx=0x7f87f0005de0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.969 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f880019d040 msgr2=0x7f880019d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:04.969 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f880019d040 0x7f880019d4f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f87f800bd50 tx=0x7f87f800be30 comp rx=0 tx=0).stop 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 shutdown_connections 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f87ec039ae0 0x7f87ec03bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f880019d040 0x7f880019d4f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87e80035c0 0x7f8800198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.968+0000 7f8806862700 1 --2- 172.21.15.177:0/3017525636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8800198770 0x7f880019cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.969+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 >> 172.21.15.177:0/3017525636 conn(0x7f88000fcff0 msgr2=0x7f8800105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.969+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 shutdown_connections 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:04.969+0000 7f8806862700 1 -- 172.21.15.177:0/3017525636 wait complete. 2024-04-24T08:12:04.970 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 3 2024-04-24T08:12:04.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: mon.c calling monitor election 2024-04-24T08:12:04.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:12:04.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: mon.a calling monitor election 2024-04-24T08:12:04.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:12:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: monmap epoch 3 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: last_changed 2024-04-24T08:11:59.462164+0000 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: min_mon_release 19 (squid) 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: election_strategy: 1 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:12:04.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: fsmap 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: mgrmap e13: x(active, since 68s) 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: overall HEALTH_OK 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:04.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:05.481 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-24T08:12:05.482 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-24T08:12:05.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:05 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/3017525636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:12:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:05.917 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:12:05.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:05.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:05 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/3017525636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:12:05.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:06.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: mon.c calling monitor election 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: mon.a calling monitor election 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:12:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: monmap epoch 3 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: last_changed 2024-04-24T08:11:59.462164+0000 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: min_mon_release 19 (squid) 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: election_strategy: 1 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: fsmap 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: mgrmap e13: x(active, since 68s) 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: overall HEALTH_OK 2024-04-24T08:12:06.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:06.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:06.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:06.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:06.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:06.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/3017525636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:12:06.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:07.436 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: mon.b calling monitor election 2024-04-24T08:12:07.436 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: Updating smithi146:/etc/ceph/ceph.conf 2024-04-24T08:12:07.436 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: mon.b calling monitor election 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: mon.c calling monitor election 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: mon.a calling monitor election 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: monmap epoch 3 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: last_changed 2024-04-24T08:11:59.462164+0000 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: min_mon_release 19 (squid) 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: election_strategy: 1 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:12:07.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: fsmap 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: mgrmap e13: x(active, since 70s) 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: overall HEALTH_OK 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: mon.b calling monitor election 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: Updating smithi146:/etc/ceph/ceph.conf 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: mon.b calling monitor election 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: mon.c calling monitor election 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: mon.a calling monitor election 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: monmap epoch 3 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: last_changed 2024-04-24T08:11:59.462164+0000 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:12:07.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: min_mon_release 19 (squid) 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: election_strategy: 1 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: fsmap 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: mgrmap e13: x(active, since 70s) 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: overall HEALTH_OK 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: mon.b calling monitor election 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: Updating smithi146:/etc/ceph/ceph.conf 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: mon.b calling monitor election 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: mon.c calling monitor election 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: mon.a calling monitor election 2024-04-24T08:12:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: monmap epoch 3 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: last_changed 2024-04-24T08:11:59.462164+0000 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: created 2024-04-24T08:09:44.411147+0000 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: min_mon_release 19 (squid) 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: election_strategy: 1 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: 0: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.a 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: fsmap 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: mgrmap e13: x(active, since 70s) 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: overall HEALTH_OK 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.889+0000 7ffb0da00700 1 -- 172.21.15.146:0/2672229354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 msgr2=0x7ffb08075270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.890+0000 7ffb0da00700 1 --2- 172.21.15.146:0/2672229354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb08075270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800b0a0 tx=0x7ffaf800b3b0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.890+0000 7ffb0da00700 1 -- 172.21.15.146:0/2672229354 shutdown_connections 2024-04-24T08:12:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.890+0000 7ffb0da00700 1 --2- 172.21.15.146:0/2672229354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb08075270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.890+0000 7ffb0da00700 1 -- 172.21.15.146:0/2672229354 >> 172.21.15.146:0/2672229354 conn(0x7ffb080fdfe0 msgr2=0x7ffb08100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.890+0000 7ffb0da00700 1 -- 172.21.15.146:0/2672229354 shutdown_connections 2024-04-24T08:12:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.890+0000 7ffb0da00700 1 -- 172.21.15.146:0/2672229354 wait complete. 2024-04-24T08:12:07.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.891+0000 7ffb0da00700 1 Processor -- start 2024-04-24T08:12:07.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.891+0000 7ffb0da00700 1 -- start start 2024-04-24T08:12:07.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb0da00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb081a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb0da00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb081a4b70 0x7ffb0819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb0da00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0819ec40 0x7ffb0819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb0da00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffb08113120 con 0x7ffb08074e90 2024-04-24T08:12:07.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb0da00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffb08112fa0 con 0x7ffb081a4b70 2024-04-24T08:12:07.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb0da00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb081132a0 con 0x7ffb0819ec40 2024-04-24T08:12:07.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb081a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb081a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34574/0 (socket says 172.21.15.146:34574) 2024-04-24T08:12:07.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 -- 172.21.15.146:0/922818814 learned_addr learned my addr 172.21.15.146:0/922818814 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffafe000700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb081a4b70 0x7ffb0819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb07400700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0819ec40 0x7ffb0819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 -- 172.21.15.146:0/922818814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0819ec40 msgr2=0x7ffb0819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0819ec40 0x7ffb0819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 -- 172.21.15.146:0/922818814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb081a4b70 msgr2=0x7ffb0819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb081a4b70 0x7ffb0819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb06a00700 1 -- 172.21.15.146:0/922818814 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb081ab3e0 con 0x7ffb08074e90 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.892+0000 7ffb07400700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0819ec40 0x7ffb0819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.893+0000 7ffb06a00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb081a4630 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009510 tx=0x7ffaf8009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.893+0000 7ffaffe00700 1 -- 172.21.15.146:0/922818814 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8022070 con 0x7ffb08074e90 2024-04-24T08:12:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.893+0000 7ffaffe00700 1 -- 172.21.15.146:0/922818814 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaf80050c0 con 0x7ffb08074e90 2024-04-24T08:12:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.893+0000 7ffaffe00700 1 -- 172.21.15.146:0/922818814 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8017040 con 0x7ffb08074e90 2024-04-24T08:12:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.893+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb081ab610 con 0x7ffb08074e90 2024-04-24T08:12:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.893+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb081abb00 con 0x7ffb08074e90 2024-04-24T08:12:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.894+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb080765d0 con 0x7ffb08074e90 2024-04-24T08:12:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.894+0000 7ffaffe00700 1 -- 172.21.15.146:0/922818814 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7ffaf8005600 con 0x7ffb08074e90 2024-04-24T08:12:07.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.895+0000 7ffaffe00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffaec039ae0 0x7ffaec03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.895+0000 7ffaffe00700 1 -- 172.21.15.146:0/922818814 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffaf800f020 con 0x7ffb08074e90 2024-04-24T08:12:07.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.895+0000 7ffafe000700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffaec039ae0 0x7ffaec03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.896+0000 7ffafe000700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffaec039ae0 0x7ffaec03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009ad0 tx=0x7ffaf0009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:07.901+0000 7ffaffe00700 1 -- 172.21.15.146:0/922818814 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffaf800e1d0 con 0x7ffb08074e90 2024-04-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.131+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ffb081abdf0 con 0x7ffb08074e90 2024-04-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.132+0000 7ffaffe00700 1 -- 172.21.15.146:0/922818814 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7ffaf8019dd0 con 0x7ffb08074e90 2024-04-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi146.stdout:# minimal ceph.conf for cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-04-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:12:08.133 INFO:teuthology.orchestra.run.smithi146.stdout: mon_host = [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffaec039ae0 msgr2=0x7ffaec03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffaec039ae0 0x7ffaec03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009ad0 tx=0x7ffaf0009210 comp rx=0 tx=0).stop 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 msgr2=0x7ffb081a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb081a4630 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009510 tx=0x7ffaf8009540 comp rx=0 tx=0).stop 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 shutdown_connections 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb0819ec40 0x7ffb0819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffaec039ae0 0x7ffaec03bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb08074e90 0x7ffb081a4630 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 --2- 172.21.15.146:0/922818814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb081a4b70 0x7ffb0819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 >> 172.21.15.146:0/922818814 conn(0x7ffb080fdfe0 msgr2=0x7ffb080fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 shutdown_connections 2024-04-24T08:12:08.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:08.137+0000 7ffb0da00700 1 -- 172.21.15.146:0/922818814 wait complete. 2024-04-24T08:12:08.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:08 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/922818814' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:08.717 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:08.718 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:08.719 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:08 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/922818814' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:08.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:08.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:08.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:08.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:08 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/922818814' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:08.957 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-24T08:12:08.957 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:12:08.957 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T08:12:09.041 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:12:09.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:12:09.164 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:12:09.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T08:12:09.192 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:12:09.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:12:09.257 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:12:09.257 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T08:12:09.284 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:12:09.284 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:12:09.349 INFO:tasks.cephadm:Adding mgr.x on smithi146 2024-04-24T08:12:09.349 INFO:tasks.cephadm:Adding mgr.y on smithi153 2024-04-24T08:12:09.349 INFO:tasks.cephadm:Adding mgr.z on smithi177 2024-04-24T08:12:09.349 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi146=x;smithi153=y;smithi177=z' 2024-04-24T08:12:09.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:09 smithi153 ceph-mon[32746]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T08:12:09.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:09 smithi153 ceph-mon[32746]: Reconfiguring daemon mon.a on smithi146 2024-04-24T08:12:09.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:09 smithi153 ceph-mon[32746]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:09.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:09 smithi146 ceph-mon[27805]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T08:12:09.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:09 smithi146 ceph-mon[27805]: Reconfiguring daemon mon.a on smithi146 2024-04-24T08:12:09.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:09 smithi146 ceph-mon[27805]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:09.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:09 smithi177 ceph-mon[33046]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T08:12:09.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:09 smithi177 ceph-mon[33046]: Reconfiguring daemon mon.a on smithi146 2024-04-24T08:12:09.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:09 smithi177 ceph-mon[33046]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:09.622 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:12:11.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.038+0000 7f19a9e61700 1 -- 172.21.15.177:0/3395055186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4101590 msgr2=0x7f19a41019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:11.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.038+0000 7f19a9e61700 1 --2- 172.21.15.177:0/3395055186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4101590 0x7f19a41019b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1994009910 tx=0x7f1994009c20 comp rx=0 tx=0).stop 2024-04-24T08:12:11.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.038+0000 7f19a9e61700 1 -- 172.21.15.177:0/3395055186 shutdown_connections 2024-04-24T08:12:11.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.038+0000 7f19a9e61700 1 --2- 172.21.15.177:0/3395055186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4101590 0x7f19a41019b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:11.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.038+0000 7f19a9e61700 1 -- 172.21.15.177:0/3395055186 >> 172.21.15.177:0/3395055186 conn(0x7f19a40fcb30 msgr2=0x7f19a40fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:11.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.038+0000 7f19a9e61700 1 -- 172.21.15.177:0/3395055186 shutdown_connections 2024-04-24T08:12:11.040 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.038+0000 7f19a9e61700 1 -- 172.21.15.177:0/3395055186 wait complete. 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 Processor -- start 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 -- start start 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101590 0x7f19a4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4198440 0x7f19a419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a41988c0 0x7f19a419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f19a41075d0 con 0x7f19a4198440 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f19a4107450 con 0x7f19a4101590 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a9e61700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19a4107750 con 0x7f19a41988c0 2024-04-24T08:12:11.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a3fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a41988c0 0x7f19a419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a3fff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a41988c0 0x7f19a419fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:32792/0 (socket says 172.21.15.177:32792) 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.039+0000 7f19a3fff700 1 -- 172.21.15.177:0/855152066 learned_addr learned my addr 172.21.15.177:0/855152066 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a37fe700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101590 0x7f19a4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a2ffd700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4198440 0x7f19a419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a3fff700 1 -- 172.21.15.177:0/855152066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101590 msgr2=0x7f19a4197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a3fff700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101590 0x7f19a4197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a3fff700 1 -- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4198440 msgr2=0x7f19a419d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a3fff700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4198440 0x7f19a419d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a3fff700 1 -- 172.21.15.177:0/855152066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a41a0310 con 0x7f19a41988c0 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a2ffd700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4198440 0x7f19a419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a3fff700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a41988c0 0x7f19a419fa40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f199c00ba50 tx=0x7f199c00be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a0ff9700 1 -- 172.21.15.177:0/855152066 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c019070 con 0x7f19a41988c0 2024-04-24T08:12:11.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a0ff9700 1 -- 172.21.15.177:0/855152066 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f199c014440 con 0x7f19a41988c0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19a41a0600 con 0x7f19a41988c0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a0ff9700 1 -- 172.21.15.177:0/855152066 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c007590 con 0x7f19a41988c0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.040+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19a41a0bc0 con 0x7f19a41988c0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.041+0000 7f19a0ff9700 1 -- 172.21.15.177:0/855152066 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f199c01c070 con 0x7f19a41988c0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.041+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19a4064150 con 0x7f19a41988c0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.041+0000 7f19a0ff9700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f198c039ae0 0x7f198c03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.041+0000 7f19a0ff9700 1 -- 172.21.15.177:0/855152066 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f199c04bef0 con 0x7f19a41988c0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.042+0000 7f19a37fe700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f198c039ae0 0x7f198c03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:11.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.043+0000 7f19a37fe700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f198c039ae0 0x7f198c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f199400b5c0 tx=0x7f199401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:11.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.047+0000 7f19a0ff9700 1 -- 172.21.15.177:0/855152066 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f199c014910 con 0x7f19a41988c0 2024-04-24T08:12:11.260 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.259+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi146=x;smithi153=y;smithi177=z", "target": ["mon-mgr", ""]}) -- 0x7f19a4062fc0 con 0x7f198c039ae0 2024-04-24T08:12:11.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.272+0000 7f19a0ff9700 1 -- 172.21.15.177:0/855152066 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f19a4062fc0 con 0x7f198c039ae0 2024-04-24T08:12:11.273 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mgr update... 2024-04-24T08:12:11.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f198c039ae0 msgr2=0x7f198c03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:11.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f198c039ae0 0x7f198c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f199400b5c0 tx=0x7f199401a040 comp rx=0 tx=0).stop 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a41988c0 msgr2=0x7f19a419fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a41988c0 0x7f19a419fa40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f199c00ba50 tx=0x7f199c00be10 comp rx=0 tx=0).stop 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 shutdown_connections 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19a41988c0 0x7f19a419fa40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101590 0x7f19a4197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f198c039ae0 0x7f198c03bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 --2- 172.21.15.177:0/855152066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19a4198440 0x7f19a419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 >> 172.21.15.177:0/855152066 conn(0x7f19a40fcb30 msgr2=0x7f19a418ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 shutdown_connections 2024-04-24T08:12:11.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:12:11.275+0000 7f19a9e61700 1 -- 172.21.15.177:0/855152066 wait complete. 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: Reconfiguring mon.b (monmap changed)... 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: Reconfiguring daemon mon.b on smithi153 2024-04-24T08:12:11.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:11 smithi153 ceph-mon[32746]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:11.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:11.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:11.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: Reconfiguring mon.b (monmap changed)... 2024-04-24T08:12:11.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:11.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:11.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:11.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: Reconfiguring daemon mon.b on smithi153 2024-04-24T08:12:11.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:11 smithi146 ceph-mon[27805]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: Reconfiguring mon.b (monmap changed)... 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: Reconfiguring daemon mon.b on smithi153 2024-04-24T08:12:11.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:11 smithi177 ceph-mon[33046]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:11.900 DEBUG:teuthology.orchestra.run.smithi153:mgr.y> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.y.service 2024-04-24T08:12:11.903 DEBUG:teuthology.orchestra.run.smithi177:mgr.z> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.z.service 2024-04-24T08:12:11.905 INFO:tasks.cephadm:Deploying OSDs... 2024-04-24T08:12:11.905 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:12:11.905 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:12:11.964 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:12:11.965 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-04-24T08:12:12.009 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-24 08:11:12.501029988 +0000 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-24 08:05:32.541083758 +0000 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-24 08:05:32.541083758 +0000 2024-04-24T08:12:12.010 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-24 08:05:32.541083758 +0000 2024-04-24T08:12:12.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:12:12.136 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-24T08:12:12.136 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-24T08:12:12.136 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000370176 s, 1.4 MB/s 2024-04-24T08:12:12.140 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:12:12.192 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-04-24T08:12:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:12:12.238 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:12.238 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-24T08:12:12.238 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:12.238 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:12.238 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-24 08:11:12.547029289 +0000 2024-04-24T08:12:12.239 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-24 08:05:33.180074107 +0000 2024-04-24T08:12:12.239 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-24 08:05:33.180074107 +0000 2024-04-24T08:12:12.239 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-24 08:05:33.180074107 +0000 2024-04-24T08:12:12.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:12:12.308 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-24T08:12:12.308 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-24T08:12:12.308 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000374235 s, 1.4 MB/s 2024-04-24T08:12:12.311 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:12:12.360 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-04-24T08:12:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:12:12.403 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:12.403 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-24T08:12:12.403 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:12.404 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:12.404 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-24 08:11:12.585028711 +0000 2024-04-24T08:12:12.404 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-24 08:05:33.807064636 +0000 2024-04-24T08:12:12.404 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-24 08:05:33.807064636 +0000 2024-04-24T08:12:12.404 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-24 08:05:33.807064636 +0000 2024-04-24T08:12:12.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:12:12.508 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-24T08:12:12.508 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-24T08:12:12.508 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000275757 s, 1.9 MB/s 2024-04-24T08:12:12.511 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:12:12.541 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi146=x;smithi153=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:12.541 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: Saving service mgr spec with placement smithi146=x;smithi153=y;smithi177=z;count:3 2024-04-24T08:12:12.541 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.541 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.541 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.541 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:12.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:12.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:12.552 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-04-24T08:12:12.597 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi146=x;smithi153=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:12.597 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: Saving service mgr spec with placement smithi146=x;smithi153=y;smithi177=z;count:3 2024-04-24T08:12:12.598 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.598 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.598 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.598 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:12.598 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:12.598 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-24 08:11:12.623028134 +0000 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-24 08:05:34.421055362 +0000 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-24 08:05:34.421055362 +0000 2024-04-24T08:12:12.633 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-24 08:05:34.421055362 +0000 2024-04-24T08:12:12.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:12:12.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi146=x;smithi153=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:12.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: Saving service mgr spec with placement smithi146=x;smithi153=y;smithi177=z;count:3 2024-04-24T08:12:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:12:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:12:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:12.753 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-04-24T08:12:12.753 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-04-24T08:12:12.753 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000356696 s, 1.4 MB/s 2024-04-24T08:12:12.757 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:12:12.806 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-24T08:12:12.806 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:12:12.824 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:12:12.825 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-04-24T08:12:12.882 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:12:12.882 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:12.882 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-24T08:12:12.882 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:12.883 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:12.883 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-24 08:11:58.928265360 +0000 2024-04-24T08:12:12.883 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-24 08:04:40.521393429 +0000 2024-04-24T08:12:12.883 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-24 08:04:40.521393429 +0000 2024-04-24T08:12:12.883 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-24 08:04:40.521393429 +0000 2024-04-24T08:12:12.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:12:12.949 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-24T08:12:12.949 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-24T08:12:12.949 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00028997 s, 1.8 MB/s 2024-04-24T08:12:12.950 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:12:13.008 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-04-24T08:12:13.065 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-24 08:11:58.928265360 +0000 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-24 08:04:40.857387209 +0000 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-24 08:04:40.857387209 +0000 2024-04-24T08:12:13.066 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-24 08:04:40.857387209 +0000 2024-04-24T08:12:13.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:12:13.132 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-24T08:12:13.132 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-24T08:12:13.132 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000205619 s, 2.5 MB/s 2024-04-24T08:12:13.133 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:12:13.191 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-04-24T08:12:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:12:13.247 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:13.248 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-24T08:12:13.248 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:13.248 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:13.248 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-24 08:11:58.929265342 +0000 2024-04-24T08:12:13.248 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-24 08:04:41.203380805 +0000 2024-04-24T08:12:13.248 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-24 08:04:41.203380805 +0000 2024-04-24T08:12:13.248 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-24 08:04:41.203380805 +0000 2024-04-24T08:12:13.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:12:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-24T08:12:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-24T08:12:13.311 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000246891 s, 2.1 MB/s 2024-04-24T08:12:13.313 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:12:13.370 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-24 08:11:58.929265342 +0000 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-24 08:04:41.540374567 +0000 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-24 08:04:41.540374567 +0000 2024-04-24T08:12:13.426 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-24 08:04:41.540374567 +0000 2024-04-24T08:12:13.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:12:13.489 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-24T08:12:13.489 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-24T08:12:13.489 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000188026 s, 2.7 MB/s 2024-04-24T08:12:13.490 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: Reconfiguring mon.c (monmap changed)... 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: Reconfiguring daemon mon.c on smithi177 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:13.530 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:13.531 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.531 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:13.531 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:12:13.531 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:13.531 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:13.546 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-24T08:12:13.546 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:12:13.561 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:12:13.562 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: Reconfiguring mon.c (monmap changed)... 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: Reconfiguring daemon mon.c on smithi177 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:13.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:13.579 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:12:13.579 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:13.579 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-24T08:12:13.579 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:13.580 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:13.580 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-24 08:11:49.016212226 +0000 2024-04-24T08:12:13.580 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-24 08:04:28.306218373 +0000 2024-04-24T08:12:13.580 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-24 08:04:28.306218373 +0000 2024-04-24T08:12:13.580 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-24 08:04:28.306218373 +0000 2024-04-24T08:12:13.580 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:12:13.642 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-24T08:12:13.642 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-24T08:12:13.643 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000197634 s, 2.6 MB/s 2024-04-24T08:12:13.644 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:12:13.700 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-04-24T08:12:13.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: Reconfiguring mon.c (monmap changed)... 2024-04-24T08:12:13.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: Reconfiguring daemon mon.c on smithi177 2024-04-24T08:12:13.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:13.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:13.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-24 08:11:49.017212198 +0000 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-24 08:04:28.659208536 +0000 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-24 08:04:28.659208536 +0000 2024-04-24T08:12:13.755 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-24 08:04:28.659208536 +0000 2024-04-24T08:12:13.756 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:12:13.817 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-24T08:12:13.817 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-24T08:12:13.817 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000219709 s, 2.3 MB/s 2024-04-24T08:12:13.818 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:12:13.874 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-24 08:11:49.017212198 +0000 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-24 08:04:28.980199589 +0000 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-24 08:04:28.980199589 +0000 2024-04-24T08:12:13.929 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-24 08:04:28.980199589 +0000 2024-04-24T08:12:13.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:12:13.992 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-24T08:12:13.992 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-24T08:12:13.992 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000258772 s, 2.0 MB/s 2024-04-24T08:12:13.993 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:12:14.048 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-04-24T08:12:14.102 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:12:14.102 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:12:14.102 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-24T08:12:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:12:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:12:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-24 08:11:49.018212171 +0000 2024-04-24T08:12:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-24 08:04:29.320190114 +0000 2024-04-24T08:12:14.103 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-24 08:04:29.320190114 +0000 2024-04-24T08:12:14.103 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-24 08:04:29.320190114 +0000 2024-04-24T08:12:14.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:12:14.164 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-24T08:12:14.164 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-24T08:12:14.164 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000191205 s, 2.7 MB/s 2024-04-24T08:12:14.165 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:12:14.220 INFO:tasks.cephadm:Deploying osd.0 on smithi146 with /dev/vg_nvme/lv_4... 2024-04-24T08:12:14.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T08:12:14.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:14 smithi177 ceph-mon[33046]: Deploying daemon mgr.z on smithi177 2024-04-24T08:12:14.620 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:12:14.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:14 smithi153 ceph-mon[32746]: Deploying daemon mgr.z on smithi177 2024-04-24T08:12:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:14 smithi146 ceph-mon[27805]: Deploying daemon mgr.z on smithi177 2024-04-24T08:12:15.288 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 systemd[1]: Starting Ceph mgr.z for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:12:15.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:15 smithi146 ceph-mon[27805]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:15.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:15 smithi177 ceph-mon[33046]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:15.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 podman[34890]: 2024-04-24 08:12:15.214003537 +0000 UTC m=+0.017803853 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:12:15.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 podman[34890]: 2024-04-24 08:12:15.336755097 +0000 UTC m=+0.140555409 container create 78209c103d096f244e29fa401744e682e25b245b73ca8021eae1a3fb567c9fb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:12:15.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 podman[34890]: 2024-04-24 08:12:15.564065412 +0000 UTC m=+0.367865724 container init 78209c103d096f244e29fa401744e682e25b245b73ca8021eae1a3fb567c9fb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-24T08:12:15.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:15 smithi153 ceph-mon[32746]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:15.814 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 podman[34890]: 2024-04-24 08:12:15.567616581 +0000 UTC m=+0.371416896 container start 78209c103d096f244e29fa401744e682e25b245b73ca8021eae1a3fb567c9fb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:12:15.815 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 ceph-mgr[34910]: -- 172.21.15.177:0/1296739644 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a6413dc5a0 con 0x55a6413b0c00 2024-04-24T08:12:15.815 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 bash[34890]: 78209c103d096f244e29fa401744e682e25b245b73ca8021eae1a3fb567c9fb4 2024-04-24T08:12:15.815 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 systemd[1]: Started Ceph mgr.z for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:12:15.815 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:15.724+0000 7fc188692200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:12:16.314 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:15.910+0000 7fc188692200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:12:16.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:16.011+0000 7fc188692200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:12:16.720 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:16.317+0000 7fc188692200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:12:16.720 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:16.402+0000 7fc188692200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:12:16.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: Deploying daemon mgr.y on smithi153 2024-04-24T08:12:16.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:16 smithi146 ceph-mon[27805]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:17.064 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: Deploying daemon mgr.y on smithi153 2024-04-24T08:12:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:16 smithi177 ceph-mon[33046]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:17.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: Deploying daemon mgr.y on smithi153 2024-04-24T08:12:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:16 smithi153 ceph-mon[32746]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:17.524 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T08:12:17.524 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T08:12:17.524 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-04-24T08:12:17.524 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-04-24T08:12:17.525 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.032427 s, 323 MB/s 2024-04-24T08:12:17.525 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-04-24T08:12:17.922 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:17 smithi153 systemd[1]: Starting Ceph mgr.y for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:12:17.953 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:17.604+0000 7fc188692200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:12:17.953 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:17.781+0000 7fc188692200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:12:17.953 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:17.952+0000 7fc188692200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:12:18.208 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:17 smithi153 podman[34114]: 2024-04-24 08:12:17.824891241 +0000 UTC m=+0.018037289 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:12:18.209 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:17 smithi153 podman[34114]: 2024-04-24 08:12:17.922349969 +0000 UTC m=+0.115496034 container create a1ab8f6ee4fb11520898884247d2dabd24e99ff684deb4a647bc1c964364dec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:12:18.209 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 podman[34114]: 2024-04-24 08:12:18.141408898 +0000 UTC m=+0.334554946 container init a1ab8f6ee4fb11520898884247d2dabd24e99ff684deb4a647bc1c964364dec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:12:18.209 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 podman[34114]: 2024-04-24 08:12:18.143960937 +0000 UTC m=+0.337106988 container start a1ab8f6ee4fb11520898884247d2dabd24e99ff684deb4a647bc1c964364dec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-24T08:12:18.209 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 ceph-mgr[34133]: -- 172.21.15.153:0/1118794100 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559375fb65a0 con 0x559375f8b000 2024-04-24T08:12:18.209 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 bash[34114]: a1ab8f6ee4fb11520898884247d2dabd24e99ff684deb4a647bc1c964364dec7 2024-04-24T08:12:18.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2024-04-24T08:12:18.460 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 systemd[1]: Started Ceph mgr.y for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:12:18.460 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:18.299+0000 7fde8c4a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:12:18.536 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:18.236+0000 7fc188692200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:12:18.536 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:18.329+0000 7fc188692200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:12:18.536 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:18.535+0000 7fc188692200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:12:18.695 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:12:18.814 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:18.735+0000 7fc188692200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:12:18.922 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:18.486+0000 7fde8c4a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:12:18.922 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:18.591+0000 7fde8c4a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:12:18.922 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:18 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:18.920+0000 7fde8c4a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:12:19.210 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:19 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:19.009+0000 7fde8c4a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:12:19.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:19 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:19 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:19 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:19 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:19 smithi146 ceph-mon[27805]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:19.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:19 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:19.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:19 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:19 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:19 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:19 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:19 smithi153 ceph-mon[32746]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:19.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:19 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:19.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:19.372+0000 7fc188692200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:12:19.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:19.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-mon[33046]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:19.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:20.064 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:19.813+0000 7fc188692200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:12:20.065 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:19.911+0000 7fc188692200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:12:20.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.541+0000 7f88f3e00700 1 -- 172.21.15.146:0/3312565133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4103520 msgr2=0x7f88f4109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.541+0000 7f88f3e00700 1 --2- 172.21.15.146:0/3312565133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4103520 0x7f88f4109dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b0d0 tx=0x7f88ec00b3e0 comp rx=0 tx=0).stop 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.541+0000 7f88f3e00700 1 -- 172.21.15.146:0/3312565133 shutdown_connections 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.541+0000 7f88f3e00700 1 --2- 172.21.15.146:0/3312565133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4103520 0x7f88f4109dd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.541+0000 7f88f3e00700 1 --2- 172.21.15.146:0/3312565133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4101980 0x7f88f4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.541+0000 7f88f3e00700 1 --2- 172.21.15.146:0/3312565133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88f4102b80 0x7f88f4102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.541+0000 7f88f3e00700 1 -- 172.21.15.146:0/3312565133 >> 172.21.15.146:0/3312565133 conn(0x7f88f40fcf00 msgr2=0x7f88f40ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.542+0000 7f88f3e00700 1 -- 172.21.15.146:0/3312565133 shutdown_connections 2024-04-24T08:12:20.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.542+0000 7f88f3e00700 1 -- 172.21.15.146:0/3312565133 wait complete. 2024-04-24T08:12:20.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 Processor -- start 2024-04-24T08:12:20.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 -- start start 2024-04-24T08:12:20.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4101980 0x7f88f419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:20.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88f4102b80 0x7f88f419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:20.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4103520 0x7f88f41a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:20.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f88f410d5a0 con 0x7f88f4102b80 2024-04-24T08:12:20.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f88f410d420 con 0x7f88f4101980 2024-04-24T08:12:20.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.543+0000 7f88f3e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88f410d720 con 0x7f88f4103520 2024-04-24T08:12:20.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f2a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4101980 0x7f88f419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:20.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f3400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4103520 0x7f88f41a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:20.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f2a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4101980 0x7f88f419ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:46066/0 (socket says 172.21.15.146:46066) 2024-04-24T08:12:20.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f2a00700 1 -- 172.21.15.146:0/2841681015 learned_addr learned my addr 172.21.15.146:0/2841681015 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:12:20.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f2000700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88f4102b80 0x7f88f419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f3400700 1 -- 172.21.15.146:0/2841681015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4101980 msgr2=0x7f88f419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f3400700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4101980 0x7f88f419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f3400700 1 -- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88f4102b80 msgr2=0x7f88f419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f3400700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88f4102b80 0x7f88f419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f3400700 1 -- 172.21.15.146:0/2841681015 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f41a2a40 con 0x7f88f4103520 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f2a00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4101980 0x7f88f419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f2000700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88f4102b80 0x7f88f419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:20.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.544+0000 7f88f3400700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4103520 0x7f88f41a23a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b0d0 tx=0x7f88ec009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:20.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.545+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec021020 con 0x7f88f4103520 2024-04-24T08:12:20.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.545+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88ec004c50 con 0x7f88f4103520 2024-04-24T08:12:20.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.545+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f41a2cd0 con 0x7f88f4103520 2024-04-24T08:12:20.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.545+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec005380 con 0x7f88f4103520 2024-04-24T08:12:20.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.545+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f41a3180 con 0x7f88f4103520 2024-04-24T08:12:20.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.546+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f40697e0 con 0x7f88f4103520 2024-04-24T08:12:20.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.546+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f88ec024020 con 0x7f88f4103520 2024-04-24T08:12:20.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.547+0000 7f88f8a00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f88d8039890 0x7f88d803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:20.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.547+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f88ec00f020 con 0x7f88f4103520 2024-04-24T08:12:20.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.547+0000 7f88f2a00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f88d8039890 0x7f88d803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:20.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.548+0000 7f88f2a00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f88d8039890 0x7f88d803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88e4005950 tx=0x7f88e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:20.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.552+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88ec0115f0 con 0x7f88f4103520 2024-04-24T08:12:20.564 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:20.112+0000 7fc188692200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:12:20.607 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:20.267+0000 7fde8c4a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:12:20.607 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:20.435+0000 7fde8c4a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:12:20.607 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:20.605+0000 7fde8c4a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:12:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:20.791+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f88f40fefd0 con 0x7f88d8039890 2024-04-24T08:12:20.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: Reconfiguring daemon mgr.x on smithi146 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:12:20.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:20.974 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:20.639+0000 7fc188692200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:12:20.974 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:20.729+0000 7fc188692200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: Reconfiguring daemon mgr.x on smithi146 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:20.975 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:12:20.976 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:12:20.976 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:21.125 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: Reconfiguring daemon mgr.x on smithi146 2024-04-24T08:12:21.126 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:21.126 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:12:21.126 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:12:21.126 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:21.185 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:20.890+0000 7fde8c4a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:12:21.185 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:20 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:20.982+0000 7fde8c4a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:12:21.185 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:21 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:21.183+0000 7fde8c4a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:12:21.314 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:20.973+0000 7fc188692200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:12:21.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:21.105+0000 7fc188692200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:12:21.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:21.199+0000 7fc188692200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:12:21.460 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:21 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:21.372+0000 7fde8c4a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:12:21.632 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:21.314+0000 7fc188692200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:12:21.632 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:21.631+0000 7fc188692200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:12:22.382 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:21 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:21.964+0000 7fde8c4a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:22.226+0000 7fc188692200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:12:22.444+0000 7fc188692200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:22.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:22.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:22.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:22 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.601 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:22.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:22.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:22.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:22.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:22 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:22.380+0000 7fde8c4a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:22.479+0000 7fde8c4a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:22.676+0000 7fde8c4a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:22.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:22.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:22.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:22.679 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:22 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:23.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:23.415+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mgrmap(e 14) ==== 92603+0+0 (secure 0 0 0) 0x7f88ec01b420 con 0x7f88f4103520 2024-04-24T08:12:23.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-mon[32746]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:23.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:12:23.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:12:23.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-mon[32746]: Standby manager daemon z started 2024-04-24T08:12:23.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:12:23.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:12:23.461 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:23.194+0000 7fde8c4a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:12:23.461 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:23.285+0000 7fde8c4a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:12:23.638 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:23 smithi146 ceph-mon[27805]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:23.638 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:23 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:12:23.638 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:23 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:12:23.638 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:23 smithi146 ceph-mon[27805]: Standby manager daemon z started 2024-04-24T08:12:23.638 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:23 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:12:23.639 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:23 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:12:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:23 smithi177 ceph-mon[33046]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:23 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:12:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:23 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:12:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:23 smithi177 ceph-mon[33046]: Standby manager daemon z started 2024-04-24T08:12:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:23 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:12:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:23 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/4142964486' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:12:23.863 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:23.525+0000 7fde8c4a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:12:23.863 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:23.656+0000 7fde8c4a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:12:23.863 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:23.749+0000 7fde8c4a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:12:23.863 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:23.861+0000 7fde8c4a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:12:24.417 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:24 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:24.162+0000 7fde8c4a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:12:24.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:24 smithi153 ceph-mon[32746]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-24T08:12:24.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:24 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:12:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:24 smithi146 ceph-mon[27805]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-24T08:12:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:24 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:12:24.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:24 smithi177 ceph-mon[33046]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-24T08:12:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:24 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:12:25.210 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:24 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:24.757+0000 7fde8c4a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:12:25.210 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:12:24 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:12:24.968+0000 7fde8c4a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:12:25.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:25.441+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mon.1 v2:172.21.15.177:3300/0 8 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f88ec005520 con 0x7f88f4103520 2024-04-24T08:12:25.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: Standby manager daemon y started 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1074020792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]: dispatch 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]: dispatch 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]': finished 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T08:12:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:25 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:25.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: Standby manager daemon y started 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1074020792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]: dispatch 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]: dispatch 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]': finished 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T08:12:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:25 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: Standby manager daemon y started 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/142426256' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1074020792' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]: dispatch 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]: dispatch 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1eee291d-ca8f-49ff-8a55-6e95cfb18c9d"}]': finished 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T08:12:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:25 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:26 smithi146 ceph-mon[27805]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-24T08:12:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:12:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:26 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2662672623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:12:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:26 smithi177 ceph-mon[33046]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-24T08:12:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:12:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:26 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2662672623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:12:26.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:26 smithi153 ceph-mon[32746]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-24T08:12:26.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:12:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:26 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2662672623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:12:27.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:27 smithi146 ceph-mon[27805]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:27.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:27 smithi177 ceph-mon[33046]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:27.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:27 smithi153 ceph-mon[32746]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:29.795 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:29 smithi146 ceph-mon[27805]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:29.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:29 smithi177 ceph-mon[33046]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:29.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:29 smithi153 ceph-mon[32746]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:31.782 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:31 smithi146 ceph-mon[27805]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:31.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:31 smithi177 ceph-mon[33046]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:31.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:31 smithi153 ceph-mon[32746]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:33 smithi146 ceph-mon[27805]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:33 smithi177 ceph-mon[33046]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:33.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:33 smithi153 ceph-mon[32746]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:34.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:34 smithi177 ceph-mon[33046]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:34.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:34 smithi153 ceph-mon[32746]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:34 smithi146 ceph-mon[27805]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:35.772 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T08:12:35.772 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:35.772 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:35 smithi146 ceph-mon[27805]: Deploying daemon osd.0 on smithi146 2024-04-24T08:12:35.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T08:12:35.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:35.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:35 smithi177 ceph-mon[33046]: Deploying daemon osd.0 on smithi146 2024-04-24T08:12:35.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T08:12:35.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:35.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:35 smithi153 ceph-mon[32746]: Deploying daemon osd.0 on smithi146 2024-04-24T08:12:36.627 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:36 smithi146 ceph-mon[27805]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:36.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:36 smithi177 ceph-mon[33046]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:36.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:36 smithi153 ceph-mon[32746]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:39.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:39 smithi146 ceph-mon[27805]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:39.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:39 smithi153 ceph-mon[32746]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:39 smithi177 ceph-mon[33046]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:41.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:41 smithi153 ceph-mon[32746]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:41.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:41 smithi146 ceph-mon[27805]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:41.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:41 smithi177 ceph-mon[33046]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:43.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:43 smithi153 ceph-mon[32746]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:43.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:43.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:43.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:43 smithi146 ceph-mon[27805]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:43 smithi177 ceph-mon[33046]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:45.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:45 smithi153 ceph-mon[32746]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:45.726 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:45 smithi146 ceph-mon[27805]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:45.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:45 smithi177 ceph-mon[33046]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: from='osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:12:47.163 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:46 smithi146 ceph-mon[27805]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:12:47.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: from='osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:12:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:46 smithi153 ceph-mon[32746]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:12:47.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.283+0000 7f88f8a00700 1 -- 172.21.15.146:0/2841681015 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f88f40fefd0 con 0x7f88d8039890 2024-04-24T08:12:47.285 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 0 on host 'smithi146' 2024-04-24T08:12:47.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.287+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f88d8039890 msgr2=0x7f88d803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:47.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.287+0000 7f88f3e00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f88d8039890 0x7f88d803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88e4005950 tx=0x7f88e4009260 comp rx=0 tx=0).stop 2024-04-24T08:12:47.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.287+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4103520 msgr2=0x7f88f41a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:47.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.287+0000 7f88f3e00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4103520 0x7f88f41a23a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00b0d0 tx=0x7f88ec009810 comp rx=0 tx=0).stop 2024-04-24T08:12:47.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 shutdown_connections 2024-04-24T08:12:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4103520 0x7f88f41a23a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f88f4101980 0x7f88f419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f88d8039890 0x7f88d803bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 --2- 172.21.15.146:0/2841681015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88f4102b80 0x7f88f419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 >> 172.21.15.146:0/2841681015 conn(0x7f88f40fcf00 msgr2=0x7f88f40fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 shutdown_connections 2024-04-24T08:12:47.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:12:47.288+0000 7f88f3e00700 1 -- 172.21.15.146:0/2841681015 wait complete. 2024-04-24T08:12:47.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:47.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:47.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:47.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:47.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: from='osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:12:47.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:46 smithi177 ceph-mon[33046]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:12:47.989 DEBUG:teuthology.orchestra.run.smithi146:osd.0> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.0.service 2024-04-24T08:12:47.993 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_4... 2024-04-24T08:12:47.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T08:12:48.207 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T08:12:48.207 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: from='osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:12:48.207 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T08:12:48.207 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:48.207 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:12:48.207 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:48.208 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:48.208 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:12:48.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T08:12:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: from='osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:12:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T08:12:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:12:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:48.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T08:12:48.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: from='osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:12:48.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T08:12:48.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:48.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:12:48.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:48.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:48.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:49.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:48 smithi146 ceph-mon[27805]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:12:49.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:48 smithi146 ceph-mon[27805]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T08:12:49.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:49.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:49.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:48 smithi146 ceph-mon[27805]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:49.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:48 smithi177 ceph-mon[33046]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:12:49.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:48 smithi177 ceph-mon[33046]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T08:12:49.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:49.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:49.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:48 smithi177 ceph-mon[33046]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:49.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:48 smithi153 ceph-mon[32746]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:12:49.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:48 smithi153 ceph-mon[32746]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T08:12:49.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:49.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:49.375 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:48 smithi153 ceph-mon[32746]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:12:50.074 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T08:12:50.074 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T08:12:50.074 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-24T08:12:50.074 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-24T08:12:50.074 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117111 s, 895 MB/s 2024-04-24T08:12:50.074 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-24T08:12:50.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:49 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:12:50.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:49 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:12:50.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:49 smithi146 ceph-mon[27805]: osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] boot 2024-04-24T08:12:50.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:49 smithi146 ceph-mon[27805]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T08:12:50.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:50.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:49 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:12:50.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:49 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:12:50.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:49 smithi177 ceph-mon[33046]: osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] boot 2024-04-24T08:12:50.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:49 smithi177 ceph-mon[33046]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T08:12:50.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:50.380 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:49 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:12:50.380 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:49 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:12:50.380 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:49 smithi153 ceph-mon[32746]: osd.0 [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] boot 2024-04-24T08:12:50.380 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:49 smithi153 ceph-mon[32746]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T08:12:50.380 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:12:50.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-04-24T08:12:50.951 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:12:51.311 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:51 smithi153 ceph-mon[32746]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T08:12:51.311 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:51 smithi153 ceph-mon[32746]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:51.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:51 smithi177 ceph-mon[33046]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T08:12:51.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:51 smithi177 ceph-mon[33046]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:51.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:51 smithi146 ceph-mon[27805]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T08:12:51.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:51 smithi146 ceph-mon[27805]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 -- 172.21.15.153:0/2677611563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1046f0 msgr2=0x7fe72c10afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 --2- 172.21.15.153:0/2677611563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1046f0 0x7fe72c10afa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe72400b0d0 tx=0x7fe72400b3e0 comp rx=0 tx=0).stop 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 -- 172.21.15.153:0/2677611563 shutdown_connections 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 --2- 172.21.15.153:0/2677611563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c1046f0 0x7fe72c10afa0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 --2- 172.21.15.153:0/2677611563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c102b50 0x7fe72c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 --2- 172.21.15.153:0/2677611563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c103d50 0x7fe72c1041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 -- 172.21.15.153:0/2677611563 >> 172.21.15.153:0/2677611563 conn(0x7fe72c0fe0f0 msgr2=0x7fe72c100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.502+0000 7fe732c6c700 1 -- 172.21.15.153:0/2677611563 shutdown_connections 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 -- 172.21.15.153:0/2677611563 wait complete. 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 Processor -- start 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 -- start start 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c102b50 0x7fe72c198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:52.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c103d50 0x7fe72c199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c1046f0 0x7fe72c1a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10e7b0 con 0x7fe72c1046f0 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10e630 con 0x7fe72c103d50 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe732c6c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10e930 con 0x7fe72c102b50 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe72bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c103d50 0x7fe72c199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe72bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c103d50 0x7fe72c199230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55058/0 (socket says 172.21.15.153:55058) 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe72bfff700 1 -- 172.21.15.153:0/467524998 learned_addr learned my addr 172.21.15.153:0/467524998 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.503+0000 7fe730a08700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c102b50 0x7fe72c198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe72bfff700 1 -- 172.21.15.153:0/467524998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c102b50 msgr2=0x7fe72c198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe72bfff700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c102b50 0x7fe72c198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe72bfff700 1 -- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c1046f0 msgr2=0x7fe72c1a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:52.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe72bfff700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c1046f0 0x7fe72c1a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe72bfff700 1 -- 172.21.15.153:0/467524998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72c1a0940 con 0x7fe72c103d50 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe730a08700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c102b50 0x7fe72c198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe72bfff700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c103d50 0x7fe72c199230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c9b0 tx=0x7fe71800cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe729ffb700 1 -- 172.21.15.153:0/467524998 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718007b00 con 0x7fe72c103d50 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe729ffb700 1 -- 172.21.15.153:0/467524998 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe718007ca0 con 0x7fe72c103d50 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72c1a0c30 con 0x7fe72c103d50 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe729ffb700 1 -- 172.21.15.153:0/467524998 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718018780 con 0x7fe72c103d50 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.504+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72c1a11f0 con 0x7fe72c103d50 2024-04-24T08:12:52.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.505+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe72c0697e0 con 0x7fe72c103d50 2024-04-24T08:12:52.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.506+0000 7fe729ffb700 1 -- 172.21.15.153:0/467524998 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7fe71800f450 con 0x7fe72c103d50 2024-04-24T08:12:52.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.507+0000 7fe729ffb700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:52.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.507+0000 7fe729ffb700 1 -- 172.21.15.153:0/467524998 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fe7180ce9e0 con 0x7fe72c103d50 2024-04-24T08:12:52.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.507+0000 7fe730a08700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:52.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.508+0000 7fe730a08700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe72c19a1d0 tx=0x7fe720008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:52.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.511+0000 7fe729ffb700 1 -- 172.21.15.153:0/467524998 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe718014030 con 0x7fe72c103d50 2024-04-24T08:12:52.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:12:52.720+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe72c062f30 con 0x7fe71c0a3bb0 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:53.586 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.587 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:12:53.587 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:12:53.587 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:12:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:12:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:12:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:54.641 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:54 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi146 2024-04-24T08:12:54.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:54 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi146 to 17089M 2024-04-24T08:12:54.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:54 smithi153 ceph-mon[32746]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:54.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:54 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi146 2024-04-24T08:12:54.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:54 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi146 to 17089M 2024-04-24T08:12:54.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:54 smithi177 ceph-mon[33046]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:54.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:54 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi146 2024-04-24T08:12:54.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:54 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi146 to 17089M 2024-04-24T08:12:54.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:54 smithi146 ceph-mon[27805]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:12:55.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:55 smithi153 ceph-mon[32746]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:55.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:55 smithi146 ceph-mon[27805]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:55.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:55 smithi177 ceph-mon[33046]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:56.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/689589989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]: dispatch 2024-04-24T08:12:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:56 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]: dispatch 2024-04-24T08:12:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:56 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]': finished 2024-04-24T08:12:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:56 smithi153 ceph-mon[32746]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T08:12:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:12:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/2649212282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:12:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/689589989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]: dispatch 2024-04-24T08:12:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:56 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]: dispatch 2024-04-24T08:12:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:56 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]': finished 2024-04-24T08:12:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:56 smithi146 ceph-mon[27805]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T08:12:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:12:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/2649212282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:12:56.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/689589989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]: dispatch 2024-04-24T08:12:56.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:56 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]: dispatch 2024-04-24T08:12:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:56 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e5912a1-9d40-4f2b-9651-551bc2797a8d"}]': finished 2024-04-24T08:12:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:56 smithi177 ceph-mon[33046]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T08:12:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:12:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/2649212282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:12:57.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:57 smithi153 ceph-mon[32746]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:57.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:57 smithi146 ceph-mon[27805]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:57 smithi177 ceph-mon[33046]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:59.654 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:12:59 smithi153 ceph-mon[32746]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:12:59 smithi146 ceph-mon[27805]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:12:59.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:12:59 smithi177 ceph-mon[33046]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:01.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:01 smithi153 ceph-mon[32746]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:01.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:01 smithi146 ceph-mon[27805]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:01.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:01 smithi177 ceph-mon[33046]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:03.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:03 smithi153 ceph-mon[32746]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:03.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T08:13:03.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:03.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:03 smithi146 ceph-mon[27805]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:03.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T08:13:03.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:03.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:03 smithi177 ceph-mon[33046]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:03.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T08:13:03.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:04.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:04 smithi153 ceph-mon[32746]: Deploying daemon osd.1 on smithi153 2024-04-24T08:13:04.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:04 smithi146 ceph-mon[27805]: Deploying daemon osd.1 on smithi153 2024-04-24T08:13:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:04 smithi177 ceph-mon[33046]: Deploying daemon osd.1 on smithi153 2024-04-24T08:13:05.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:05 smithi153 ceph-mon[32746]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:05.745 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:05 smithi146 ceph-mon[27805]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:05.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:05 smithi177 ceph-mon[33046]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:07.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:07 smithi146 ceph-mon[27805]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:07.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:07 smithi177 ceph-mon[33046]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:07.870 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:07 smithi153 ceph-mon[32746]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:09.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:09 smithi153 ceph-mon[32746]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:09.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:09 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:09.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:09 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:09.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:09 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:09.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:09 smithi146 ceph-mon[27805]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:09.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:09 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:09.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:09 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:09.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:09 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:09.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:09 smithi177 ceph-mon[33046]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:09 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:09 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:09 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:11.604 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:11 smithi153 ceph-mon[32746]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:11.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.702+0000 7fe729ffb700 1 -- 172.21.15.153:0/467524998 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe72c062f30 con 0x7fe71c0a3bb0 2024-04-24T08:13:11.703 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-04-24T08:13:11.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.703+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe71c0a3bb0 msgr2=0x7fe71c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.703+0000 7fe732c6c700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe72c19a1d0 tx=0x7fe720008040 comp rx=0 tx=0).stop 2024-04-24T08:13:11.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.703+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c103d50 msgr2=0x7fe72c199230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.703+0000 7fe732c6c700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c103d50 0x7fe72c199230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c9b0 tx=0x7fe71800cd70 comp rx=0 tx=0).stop 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 shutdown_connections 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe71c0a3bb0 0x7fe71c0a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c1046f0 0x7fe72c1a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c102b50 0x7fe72c198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 --2- 172.21.15.153:0/467524998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe72c103d50 0x7fe72c199230 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 >> 172.21.15.153:0/467524998 conn(0x7fe72c0fe0f0 msgr2=0x7fe72c106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 shutdown_connections 2024-04-24T08:13:11.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:13:11.704+0000 7fe732c6c700 1 -- 172.21.15.153:0/467524998 wait complete. 2024-04-24T08:13:11.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:11 smithi146 ceph-mon[27805]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:11.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:11 smithi177 ceph-mon[33046]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:12.567 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.1.service 2024-04-24T08:13:12.570 INFO:tasks.cephadm:Deploying osd.2 on smithi177 with /dev/vg_nvme/lv_4... 2024-04-24T08:13:12.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T08:13:12.815 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:13:12.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:12.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:12 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:12.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:12 smithi177 ceph-mon[33046]: from='osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:13:12.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:12 smithi177 ceph-mon[33046]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:13:12.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:12 smithi177 ceph-mon[33046]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:12.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:12.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:12 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:12.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:12 smithi153 ceph-mon[32746]: from='osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:13:12.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:12 smithi153 ceph-mon[32746]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:13:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:12 smithi153 ceph-mon[32746]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:12 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:12 smithi146 ceph-mon[27805]: from='osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:13:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:12 smithi146 ceph-mon[27805]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:13:12.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:12 smithi146 ceph-mon[27805]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:13.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:13 smithi146 ceph-mon[27805]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T08:13:13.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:13 smithi146 ceph-mon[27805]: from='osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:13:13.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:13 smithi146 ceph-mon[27805]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T08:13:13.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:13.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:13 smithi146 ceph-mon[27805]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:13:13.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:13.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:13.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:13 smithi177 ceph-mon[33046]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T08:13:13.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:13 smithi177 ceph-mon[33046]: from='osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:13:13.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:13 smithi177 ceph-mon[33046]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T08:13:13.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:13.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:13 smithi177 ceph-mon[33046]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:13:13.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:13.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:13.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:13 smithi153 ceph-mon[32746]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T08:13:13.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:13 smithi153 ceph-mon[32746]: from='osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:13:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:13 smithi153 ceph-mon[32746]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T08:13:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:13 smithi153 ceph-mon[32746]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:13:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:14.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-mon[32746]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:13:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-mon[32746]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T08:13:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-mon[32746]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-mon[32746]: from='osd.1 ' entity='osd.1' 2024-04-24T08:13:14.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1[37340]: 2024-04-24T08:13:14.599+0000 7f1d90160700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T08:13:14.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1[37340]: 2024-04-24T08:13:14.637+0000 7f1d87f4b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:13:14.714 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T08:13:14.714 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T08:13:14.714 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-24T08:13:14.714 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-24T08:13:14.714 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113773 s, 922 MB/s 2024-04-24T08:13:14.714 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-24T08:13:14.981 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:14 smithi177 ceph-mon[33046]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:13:14.981 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:14 smithi177 ceph-mon[33046]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T08:13:14.981 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:14.981 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:14.982 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:14 smithi177 ceph-mon[33046]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:14.982 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:14 smithi177 ceph-mon[33046]: from='osd.1 ' entity='osd.1' 2024-04-24T08:13:14.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:14 smithi146 ceph-mon[27805]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:13:14.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:14 smithi146 ceph-mon[27805]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T08:13:14.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:14.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:14.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:14 smithi146 ceph-mon[27805]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:13:14.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:14 smithi146 ceph-mon[27805]: from='osd.1 ' entity='osd.1' 2024-04-24T08:13:15.317 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-04-24T08:13:15.557 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:13:15.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:15 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:13:15.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:15 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:13:15.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:15.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:15 smithi153 ceph-mon[32746]: osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] boot 2024-04-24T08:13:15.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:15 smithi153 ceph-mon[32746]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T08:13:15.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:15.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:15 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:13:15.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:15 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:13:15.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:15.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:15 smithi177 ceph-mon[33046]: osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] boot 2024-04-24T08:13:15.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:15 smithi177 ceph-mon[33046]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T08:13:15.893 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:15.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:15 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:13:15.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:15 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:13:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:15 smithi146 ceph-mon[27805]: osd.1 [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] boot 2024-04-24T08:13:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:15 smithi146 ceph-mon[27805]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T08:13:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1524415452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 msgr2=0x7fc7c810acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1524415452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c810acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b0a0 tx=0x7fc7c000b3b0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1524415452 shutdown_connections 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1524415452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c810acb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1524415452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c81035e0 0x7fc7c81059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1524415452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7c8105f10 0x7fc7c8108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1524415452 >> 172.21.15.177:0/1524415452 conn(0x7fc7c80fcff0 msgr2=0x7fc7c80ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1524415452 shutdown_connections 2024-04-24T08:13:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.118+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1524415452 wait complete. 2024-04-24T08:13:17.121 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 Processor -- start 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 -- start start 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c81035e0 0x7fc7c8198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7c8105f10 0x7fc7c8199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c81a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c810e800 con 0x7fc7c8105f10 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c810e680 con 0x7fc7c81035e0 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7d07d1700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c810e980 con 0x7fc7c8108840 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c81a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c81a0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:45986/0 (socket says 172.21.15.177:45986) 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 -- 172.21.15.177:0/1027162815 learned_addr learned my addr 172.21.15.177:0/1027162815 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ce56d700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c81035e0 0x7fc7c8198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 -- 172.21.15.177:0/1027162815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c81035e0 msgr2=0x7fc7c8198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.122 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c81035e0 0x7fc7c8198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 -- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7c8105f10 msgr2=0x7fc7c8199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7c8105f10 0x7fc7c8199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 -- 172.21.15.177:0/1027162815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c81a0a20 con 0x7fc7c8108840 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.119+0000 7fc7ced6e700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c81a0380 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000d010 tx=0x7fc7c0004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.120+0000 7fc7bf7fe700 1 -- 172.21.15.177:0/1027162815 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0021070 con 0x7fc7c8108840 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.120+0000 7fc7bf7fe700 1 -- 172.21.15.177:0/1027162815 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c001d930 con 0x7fc7c8108840 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.120+0000 7fc7bf7fe700 1 -- 172.21.15.177:0/1027162815 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00055a0 con 0x7fc7c8108840 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.120+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c81a0cb0 con 0x7fc7c8108840 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.120+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c81a11e0 con 0x7fc7c8108840 2024-04-24T08:13:17.123 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.121+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c80697e0 con 0x7fc7c8108840 2024-04-24T08:13:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.122+0000 7fc7bf7fe700 1 -- 172.21.15.177:0/1027162815 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7fc7c0024070 con 0x7fc7c8108840 2024-04-24T08:13:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.122+0000 7fc7bf7fe700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7b40a3bb0 0x7fc7b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.122+0000 7fc7bf7fe700 1 -- 172.21.15.177:0/1027162815 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fc7c000f020 con 0x7fc7c8108840 2024-04-24T08:13:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.123+0000 7fc7ce56d700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7b40a3bb0 0x7fc7b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.124 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.123+0000 7fc7ce56d700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7b40a3bb0 0x7fc7b40a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009a10 tx=0x7fc7c4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.127 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.127+0000 7fc7bf7fe700 1 -- 172.21.15.177:0/1027162815 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7c002b020 con 0x7fc7c8108840 2024-04-24T08:13:17.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:17.334+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc7c8062f30 con 0x7fc7b40a3bb0 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi153 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:17.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:17.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:17.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:17.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:17.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi153 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:17.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi153 2024-04-24T08:13:17.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:13:17.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi153 to 17088M 2024-04-24T08:13:17.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:17.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:17.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:17.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:17.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:17.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:17.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:17.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:18.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:18 smithi177 ceph-mon[33046]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T08:13:18.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:18 smithi177 ceph-mon[33046]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:13:18.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:18 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:13:18.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:18 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:13:18.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:18 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:18.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:18 smithi153 ceph-mon[32746]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T08:13:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:18 smithi153 ceph-mon[32746]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:13:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:18 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:13:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:18 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:13:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:18 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:18.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:18 smithi146 ceph-mon[27805]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T08:13:18.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:18 smithi146 ceph-mon[27805]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:13:18.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:13:18.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:13:18.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:19.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:19 smithi177 ceph-mon[33046]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:19.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:19 smithi153 ceph-mon[32746]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:19.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:19 smithi146 ceph-mon[27805]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:20.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:20 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/3044768690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]: dispatch 2024-04-24T08:13:20.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:20 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]: dispatch 2024-04-24T08:13:20.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:20 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]': finished 2024-04-24T08:13:20.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:20 smithi177 ceph-mon[33046]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T08:13:20.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:20.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:20 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/3044768690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]: dispatch 2024-04-24T08:13:20.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:20 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]: dispatch 2024-04-24T08:13:20.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:20 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]': finished 2024-04-24T08:13:20.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:20 smithi153 ceph-mon[32746]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T08:13:20.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:20.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:20 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/3044768690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]: dispatch 2024-04-24T08:13:20.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:20 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]: dispatch 2024-04-24T08:13:20.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:20 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a25edc87-3db2-4652-82ee-c406cbe5dd64"}]': finished 2024-04-24T08:13:20.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:20 smithi146 ceph-mon[27805]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T08:13:20.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:21.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:21 smithi153 ceph-mon[32746]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:21 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/3303944905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:13:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:21 smithi146 ceph-mon[27805]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:21 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/3303944905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:13:21.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:21 smithi177 ceph-mon[33046]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:21.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:21 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/3303944905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:13:23.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:23 smithi153 ceph-mon[32746]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:23.731 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:23 smithi177 ceph-mon[33046]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:23 smithi146 ceph-mon[27805]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:25.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:25 smithi153 ceph-mon[32746]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:25.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:25 smithi146 ceph-mon[27805]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:25.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:25 smithi177 ceph-mon[33046]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:27.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:27 smithi153 ceph-mon[32746]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:27.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T08:13:27.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:27.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:27 smithi146 ceph-mon[27805]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:27.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T08:13:27.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:27.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:27 smithi177 ceph-mon[33046]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T08:13:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:28 smithi177 ceph-mon[33046]: Deploying daemon osd.2 on smithi177 2024-04-24T08:13:28.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:28 smithi153 ceph-mon[32746]: Deploying daemon osd.2 on smithi177 2024-04-24T08:13:28.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:28 smithi146 ceph-mon[27805]: Deploying daemon osd.2 on smithi177 2024-04-24T08:13:29.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:29 smithi153 ceph-mon[32746]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:29.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:29 smithi146 ceph-mon[27805]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:29.787 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:29 smithi177 ceph-mon[33046]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:31.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:31 smithi153 ceph-mon[32746]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:31.719 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:31 smithi177 ceph-mon[33046]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:31 smithi146 ceph-mon[27805]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:33.543 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:33 smithi177 ceph-mon[33046]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:33.543 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:33.543 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:33.543 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:33.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:33 smithi153 ceph-mon[32746]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:33.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:33.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:33.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:33 smithi146 ceph-mon[27805]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:33.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:33.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:35 smithi153 ceph-mon[32746]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:35 smithi146 ceph-mon[27805]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:35.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:35.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:35 smithi177 ceph-mon[33046]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:36.229 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 2 on host 'smithi177' 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.227+0000 7fc7bf7fe700 1 -- 172.21.15.177:0/1027162815 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc7c8062f30 con 0x7fc7b40a3bb0 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7b40a3bb0 msgr2=0x7fc7b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7b40a3bb0 0x7fc7b40a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009a10 tx=0x7fc7c4009190 comp rx=0 tx=0).stop 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 msgr2=0x7fc7c81a0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c81a0380 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000d010 tx=0x7fc7c0004e40 comp rx=0 tx=0).stop 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 shutdown_connections 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7c8108840 0x7fc7c81a0380 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7c81035e0 0x7fc7c8198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7b40a3bb0 0x7fc7b40a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 --2- 172.21.15.177:0/1027162815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7c8105f10 0x7fc7c8199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 >> 172.21.15.177:0/1027162815 conn(0x7fc7c80fcff0 msgr2=0x7fc7c8103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:36.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 shutdown_connections 2024-04-24T08:13:36.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:13:36.229+0000 7fc7d07d1700 1 -- 172.21.15.177:0/1027162815 wait complete. 2024-04-24T08:13:36.827 DEBUG:teuthology.orchestra.run.smithi177:osd.2> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.2.service 2024-04-24T08:13:36.829 INFO:tasks.cephadm:Deploying osd.3 on smithi146 with /dev/vg_nvme/lv_3... 2024-04-24T08:13:36.829 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T08:13:37.259 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:13:37.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:37 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:37.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:37 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:37.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:37 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:37.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:37 smithi153 ceph-mon[32746]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:37.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:37 smithi153 ceph-mon[32746]: from='osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:13:37.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:37 smithi153 ceph-mon[32746]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:13:37.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:37 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:37.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:37 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:37.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:37 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:37.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:37 smithi146 ceph-mon[27805]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:37.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:37 smithi146 ceph-mon[27805]: from='osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:13:37.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:37 smithi146 ceph-mon[27805]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:13:37.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:37 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:37.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:37 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:37.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:37 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:37.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:37 smithi177 ceph-mon[33046]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:37.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:37 smithi177 ceph-mon[33046]: from='osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:13:37.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:37 smithi177 ceph-mon[33046]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:13:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:38 smithi177 ceph-mon[33046]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T08:13:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:38 smithi177 ceph-mon[33046]: from='osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:13:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:38 smithi177 ceph-mon[33046]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T08:13:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:38 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:38 smithi177 ceph-mon[33046]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:13:38.316 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:38 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2[38174]: 2024-04-24T08:13:38.202+0000 7efeb4029700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T08:13:38.316 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:38 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2[38174]: 2024-04-24T08:13:38.208+0000 7efeafe1c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:13:38.412 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:38 smithi146 ceph-mon[27805]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T08:13:38.413 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:38 smithi146 ceph-mon[27805]: from='osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:13:38.413 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:38 smithi146 ceph-mon[27805]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T08:13:38.413 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:38 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:38.413 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:38 smithi146 ceph-mon[27805]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:13:38.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:38 smithi153 ceph-mon[32746]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T08:13:38.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:38 smithi153 ceph-mon[32746]: from='osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:13:38.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:38 smithi153 ceph-mon[32746]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T08:13:38.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:38 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:38.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:38 smithi153 ceph-mon[32746]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:13:39.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:39 smithi177 ceph-mon[33046]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:13:39.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:39 smithi177 ceph-mon[33046]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T08:13:39.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:39.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:39.404 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:39 smithi177 ceph-mon[33046]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:39.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:39 smithi146 ceph-mon[27805]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:13:39.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:39 smithi146 ceph-mon[27805]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T08:13:39.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:39.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:39.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:39 smithi146 ceph-mon[27805]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:39.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:39 smithi153 ceph-mon[32746]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:13:39.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:39 smithi153 ceph-mon[32746]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T08:13:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:39 smithi153 ceph-mon[32746]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:13:39.896 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T08:13:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T08:13:39.897 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-04-24T08:13:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-04-24T08:13:39.897 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0343637 s, 305 MB/s 2024-04-24T08:13:39.897 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-04-24T08:13:40.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] boot 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:40.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:40.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] boot 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:13:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:40.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:40.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2024-04-24T08:13:40.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:13:40.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:13:40.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: osd.2 [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] boot 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:41.129 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:13:41.275 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:41 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi177 2024-04-24T08:13:41.276 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:41 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-24T08:13:41.276 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:41 smithi177 ceph-mon[33046]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T08:13:41.276 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:41 smithi177 ceph-mon[33046]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:41.276 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:41 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:13:41.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:41 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi177 2024-04-24T08:13:41.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:41 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-24T08:13:41.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:41 smithi146 ceph-mon[27805]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T08:13:41.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:41 smithi146 ceph-mon[27805]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:41.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:41 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:13:41.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:41 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi177 2024-04-24T08:13:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:41 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-24T08:13:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:41 smithi153 ceph-mon[32746]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T08:13:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:41 smithi153 ceph-mon[32746]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:41 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:13:42.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T08:13:42.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:42 smithi146 ceph-mon[27805]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T08:13:42.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:42 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:13:42.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:42 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T08:13:42.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:42 smithi177 ceph-mon[33046]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T08:13:42.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:42 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:13:42.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:42 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T08:13:42.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:42 smithi153 ceph-mon[32746]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T08:13:42.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:42 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:13:42.939 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46965]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:13:42.940 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46965]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:42.940 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46965]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:42.940 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46965]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:42.940 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46969]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:13:42.940 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46969]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:42.940 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46969]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:42.940 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:13:42 smithi146 sudo[46969]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.210 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:13:43.211 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.211 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41035]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.211 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41035]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.211 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41038]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:13:43.211 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.211 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41038]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.211 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:13:42 smithi153 sudo[41038]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.228+0000 7fe28b000700 1 -- 172.21.15.146:0/2497666137 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284103c90 msgr2=0x7fe2841040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.228+0000 7fe28b000700 1 --2- 172.21.15.146:0/2497666137 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284103c90 0x7fe2841040f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe2700099c0 tx=0x7fe270009cd0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.228+0000 7fe28b000700 1 -- 172.21.15.146:0/2497666137 shutdown_connections 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.228+0000 7fe28b000700 1 --2- 172.21.15.146:0/2497666137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe284104630 0x7fe28410aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.228+0000 7fe28b000700 1 --2- 172.21.15.146:0/2497666137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284102a90 0x7fe284102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.228+0000 7fe28b000700 1 --2- 172.21.15.146:0/2497666137 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284103c90 0x7fe2841040f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.228+0000 7fe28b000700 1 -- 172.21.15.146:0/2497666137 >> 172.21.15.146:0/2497666137 conn(0x7fe2840fe030 msgr2=0x7fe284100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.229+0000 7fe28b000700 1 -- 172.21.15.146:0/2497666137 shutdown_connections 2024-04-24T08:13:43.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.229+0000 7fe28b000700 1 -- 172.21.15.146:0/2497666137 wait complete. 2024-04-24T08:13:43.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.230+0000 7fe28b000700 1 Processor -- start 2024-04-24T08:13:43.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.230+0000 7fe28b000700 1 -- start start 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe28b000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284102a90 0x7fe284198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe28b000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284103c90 0x7fe2841991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe28b000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe284104630 0x7fe2841a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe28b000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe28410e650 con 0x7fe284102a90 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe28b000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe28410e4d0 con 0x7fe284103c90 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe28b000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe28410e7d0 con 0x7fe284104630 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284103c90 0x7fe2841991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284103c90 0x7fe2841991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39464/0 (socket says 172.21.15.146:39464) 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 -- 172.21.15.146:0/1709195149 learned_addr learned my addr 172.21.15.146:0/1709195149 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289c00700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284102a90 0x7fe284198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe28a600700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe284104630 0x7fe2841a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 -- 172.21.15.146:0/1709195149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe284104630 msgr2=0x7fe2841a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe284104630 0x7fe2841a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 -- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284102a90 msgr2=0x7fe284198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284102a90 0x7fe284198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.231+0000 7fe289200700 1 -- 172.21.15.146:0/1709195149 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2841a08b0 con 0x7fe284103c90 2024-04-24T08:13:43.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.232+0000 7fe289200700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284103c90 0x7fe2841991a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe27000b580 tx=0x7fe27000bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.254+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270009e60 con 0x7fe284103c90 2024-04-24T08:13:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.254+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe270003e10 con 0x7fe284103c90 2024-04-24T08:13:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.254+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700173f0 con 0x7fe284103c90 2024-04-24T08:13:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.254+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2841a0b40 con 0x7fe284103c90 2024-04-24T08:13:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.254+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2841a0f40 con 0x7fe284103c90 2024-04-24T08:13:43.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.256+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2840697e0 con 0x7fe284103c90 2024-04-24T08:13:43.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.258+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7fe27000f680 con 0x7fe284103c90 2024-04-24T08:13:43.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.260+0000 7fe282000700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe26c0a3c30 0x7fe26c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.260+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7fe2700d3180 con 0x7fe284103c90 2024-04-24T08:13:43.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.260+0000 7fe289c00700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe26c0a3c30 0x7fe26c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.261+0000 7fe289c00700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe26c0a3c30 0x7fe26c0a60f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe28419a140 tx=0x7fe278009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.265+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe270013070 con 0x7fe284103c90 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42218]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42218]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42218]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42218]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42221]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42221]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42224]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42224]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42224]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42224]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42227]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42227]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42227]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 sudo[42227]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46977]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46977]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46977]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46977]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46980]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46980]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46980]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 sudo[46980]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.495 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.496 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:43.506+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe284062f30 con 0x7fe26c0a3c30 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41041]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41041]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41041]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41041]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41044]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41044]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41044]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 sudo[41044]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:13:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:43.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:43.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:13:43.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:13:43.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:13:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:44 smithi177 ceph-mon[33046]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:44 smithi177 ceph-mon[33046]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:44 smithi177 ceph-mon[33046]: osdmap e22: 3 total, 3 up, 3 in 2024-04-24T08:13:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:44 smithi177 ceph-mon[33046]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:13:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:44 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:13:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:44 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:13:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:44 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:44.604 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:44 smithi146 ceph-mon[27805]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:44.604 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:44 smithi146 ceph-mon[27805]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:44.604 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:44 smithi146 ceph-mon[27805]: osdmap e22: 3 total, 3 up, 3 in 2024-04-24T08:13:44.605 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:44 smithi146 ceph-mon[27805]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:13:44.605 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:44 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:13:44.605 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:44 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:13:44.605 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:44 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:44.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:44 smithi153 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:13:44.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:44 smithi153 ceph-mon[32746]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:13:44.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:44 smithi153 ceph-mon[32746]: osdmap e22: 3 total, 3 up, 3 in 2024-04-24T08:13:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:44 smithi153 ceph-mon[32746]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:13:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:44 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:13:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:44 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:13:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:44 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:45.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:13:45.294+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe270099b00 con 0x7fe284103c90 2024-04-24T08:13:45.478 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:45 smithi146 ceph-mon[27805]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:45.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:45 smithi177 ceph-mon[33046]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:45.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:45 smithi153 ceph-mon[32746]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:46.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:46 smithi146 ceph-mon[27805]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T08:13:46.564 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:46 smithi177 ceph-mon[33046]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T08:13:46.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:46 smithi153 ceph-mon[32746]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T08:13:47.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:47 smithi177 ceph-mon[33046]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:47.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:47 smithi153 ceph-mon[32746]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:47.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:47 smithi146 ceph-mon[27805]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:48.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:48 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2333856292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]: dispatch 2024-04-24T08:13:48.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:48 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]: dispatch 2024-04-24T08:13:48.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:48 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]': finished 2024-04-24T08:13:48.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:48 smithi177 ceph-mon[33046]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T08:13:48.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:13:48.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:48 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/364496553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:13:48.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:48 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2333856292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]: dispatch 2024-04-24T08:13:48.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:48 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]: dispatch 2024-04-24T08:13:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:48 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]': finished 2024-04-24T08:13:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:48 smithi153 ceph-mon[32746]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T08:13:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:13:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:48 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/364496553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:13:48.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:48 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2333856292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]: dispatch 2024-04-24T08:13:48.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:48 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]: dispatch 2024-04-24T08:13:48.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:48 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0de0716-915a-4554-bb92-b04edb82af4b"}]': finished 2024-04-24T08:13:48.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:48 smithi146 ceph-mon[27805]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T08:13:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:13:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:48 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/364496553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:13:49.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:49 smithi153 ceph-mon[32746]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:49.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:49 smithi146 ceph-mon[27805]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:49 smithi177 ceph-mon[33046]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:51.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:51 smithi153 ceph-mon[32746]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:51.711 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:51 smithi177 ceph-mon[33046]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:51 smithi146 ceph-mon[27805]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:53.677 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:53 smithi146 ceph-mon[27805]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:53.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:53 smithi153 ceph-mon[32746]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:53.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:53 smithi177 ceph-mon[33046]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:55.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:55 smithi146 ceph-mon[27805]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:55.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:55 smithi153 ceph-mon[32746]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:55.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:55 smithi177 ceph-mon[33046]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:57.701 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:57 smithi146 ceph-mon[27805]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:57.701 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T08:13:57.701 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:57.701 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:57 smithi146 ceph-mon[27805]: Deploying daemon osd.3 on smithi146 2024-04-24T08:13:57.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:57 smithi153 ceph-mon[32746]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:57.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T08:13:57.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:57.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:57 smithi153 ceph-mon[32746]: Deploying daemon osd.3 on smithi146 2024-04-24T08:13:57.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:57 smithi177 ceph-mon[33046]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T08:13:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:57 smithi177 ceph-mon[33046]: Deploying daemon osd.3 on smithi146 2024-04-24T08:13:59.669 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:13:59 smithi146 ceph-mon[27805]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:59.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:13:59 smithi153 ceph-mon[32746]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:13:59.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:13:59 smithi177 ceph-mon[33046]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:01.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:01 smithi153 ceph-mon[32746]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:01.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:01 smithi146 ceph-mon[27805]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:01.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:01 smithi177 ceph-mon[33046]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:03.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:03 smithi153 ceph-mon[32746]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:03.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:03 smithi146 ceph-mon[27805]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:03.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:03 smithi177 ceph-mon[33046]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:05.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:05 smithi153 ceph-mon[32746]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:05.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:05.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:05.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:05.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:05 smithi146 ceph-mon[27805]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:05.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:05.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:05.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:05 smithi177 ceph-mon[33046]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:07.698 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:07 smithi146 ceph-mon[27805]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:07.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:07 smithi153 ceph-mon[32746]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:07.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:07 smithi177 ceph-mon[33046]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:09.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:09.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:09.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:09.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: from='osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:14:09.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:08 smithi153 ceph-mon[32746]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:14:09.226 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.226 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.226 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:09.227 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:09.227 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.227 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:09.227 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: from='osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:14:09.227 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:08 smithi146 ceph-mon[27805]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: from='osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:14:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:08 smithi177 ceph-mon[33046]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:14:09.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.347+0000 7fe282000700 1 -- 172.21.15.146:0/1709195149 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe284062f30 con 0x7fe26c0a3c30 2024-04-24T08:14:09.349 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 3 on host 'smithi146' 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe26c0a3c30 msgr2=0x7fe26c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe26c0a3c30 0x7fe26c0a60f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe28419a140 tx=0x7fe278009260 comp rx=0 tx=0).stop 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284103c90 msgr2=0x7fe2841991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284103c90 0x7fe2841991a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe27000b580 tx=0x7fe27000bee0 comp rx=0 tx=0).stop 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 shutdown_connections 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe284104630 0x7fe2841a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe26c0a3c30 0x7fe26c0a60f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe284102a90 0x7fe284198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 --2- 172.21.15.146:0/1709195149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe284103c90 0x7fe2841991a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 >> 172.21.15.146:0/1709195149 conn(0x7fe2840fe030 msgr2=0x7fe284107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 shutdown_connections 2024-04-24T08:14:09.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:14:09.351+0000 7fe28b000700 1 -- 172.21.15.146:0/1709195149 wait complete. 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: from='osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:10.131 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:09 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:10.201 DEBUG:teuthology.orchestra.run.smithi146:osd.3> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.3.service 2024-04-24T08:14:10.206 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_3... 2024-04-24T08:14:10.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T08:14:10.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T08:14:10.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: from='osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:14:10.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T08:14:10.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:10.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:14:10.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:10.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:10.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:09 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: from='osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:10.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:09 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:10.452 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:14:10.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:14:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: osdmap e25: 4 total, 3 up, 4 in 2024-04-24T08:14:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] boot 2024-04-24T08:14:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T08:14:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:11.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:14:11.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: osdmap e25: 4 total, 3 up, 4 in 2024-04-24T08:14:11.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:11.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:11.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:11.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] boot 2024-04-24T08:14:11.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T08:14:11.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:11.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:14:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: osdmap e25: 4 total, 3 up, 4 in 2024-04-24T08:14:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:14:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: osd.3 [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] boot 2024-04-24T08:14:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T08:14:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:14:12.441 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T08:14:12.442 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T08:14:12.442 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-24T08:14:12.442 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-24T08:14:12.442 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116138 s, 903 MB/s 2024-04-24T08:14:12.442 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-24T08:14:12.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:12 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:14:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:12 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:14:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:12 smithi177 ceph-mon[33046]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T08:14:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:12 smithi177 ceph-mon[33046]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:12.828 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:12 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:14:12.828 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:12 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:14:12.828 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:12 smithi153 ceph-mon[32746]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T08:14:12.828 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:12 smithi153 ceph-mon[32746]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:12.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:12 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:14:12.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:12 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:14:12.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:12 smithi146 ceph-mon[27805]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T08:14:12.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:12 smithi146 ceph-mon[27805]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:13.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-04-24T08:14:13.419 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:14:13.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:13 smithi153 ceph-mon[32746]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T08:14:13.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:13 smithi177 ceph-mon[33046]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T08:14:13.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:13 smithi146 ceph-mon[27805]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T08:14:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:14 smithi146 ceph-mon[27805]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T08:14:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:14 smithi146 ceph-mon[27805]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:14 smithi177 ceph-mon[33046]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T08:14:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:14 smithi177 ceph-mon[33046]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:14.817 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:14 smithi153 ceph-mon[32746]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T08:14:14.817 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:14 smithi153 ceph-mon[32746]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.034+0000 7f70780e0700 1 -- 172.21.15.153:0/935460637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70701088e0 msgr2=0x7f707010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.034+0000 7f70780e0700 1 --2- 172.21.15.153:0/935460637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70701088e0 0x7f707010acd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f706800b0a0 tx=0x7f706800b3b0 comp rx=0 tx=0).stop 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 -- 172.21.15.153:0/935460637 shutdown_connections 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 --2- 172.21.15.153:0/935460637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70701088e0 0x7f707010acd0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 --2- 172.21.15.153:0/935460637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7070103680 0x7f7070105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 --2- 172.21.15.153:0/935460637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7070105fb0 0x7f70701083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 -- 172.21.15.153:0/935460637 >> 172.21.15.153:0/935460637 conn(0x7f70700fd010 msgr2=0x7f70700ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 -- 172.21.15.153:0/935460637 shutdown_connections 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 -- 172.21.15.153:0/935460637 wait complete. 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 Processor -- start 2024-04-24T08:14:15.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.035+0000 7f70780e0700 1 -- start start 2024-04-24T08:14:15.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f70780e0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7070103680 0x7f7070198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f70780e0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070105fb0 0x7f7070199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f70780e0700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70701088e0 0x7f70701a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f70780e0700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f707010e780 con 0x7f70701088e0 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f70780e0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f707010e600 con 0x7f7070105fb0 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f70780e0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f707010e900 con 0x7f7070103680 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070105fb0 0x7f7070199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070105fb0 0x7f7070199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49530/0 (socket says 172.21.15.153:49530) 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 -- 172.21.15.153:0/3767416582 learned_addr learned my addr 172.21.15.153:0/3767416582 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f7075e7c700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7070103680 0x7f7070198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707667d700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70701088e0 0x7f70701a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 -- 172.21.15.153:0/3767416582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7070103680 msgr2=0x7f7070198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7070103680 0x7f7070198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:15.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 -- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70701088e0 msgr2=0x7f70701a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70701088e0 0x7f70701a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 -- 172.21.15.153:0/3767416582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70701a09a0 con 0x7f7070105fb0 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707667d700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70701088e0 0x7f70701a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f707567b700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070105fb0 0x7f7070199280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f706000c920 tx=0x7f706000cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f7066ffd700 1 -- 172.21.15.153:0/3767416582 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060007b00 con 0x7f7070105fb0 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f7066ffd700 1 -- 172.21.15.153:0/3767416582 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7060004d10 con 0x7f7070105fb0 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70701a0c90 con 0x7f7070105fb0 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.036+0000 7f7066ffd700 1 -- 172.21.15.153:0/3767416582 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060005680 con 0x7f7070105fb0 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.037+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70701a1180 con 0x7f7070105fb0 2024-04-24T08:14:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.037+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70700697e0 con 0x7f7070105fb0 2024-04-24T08:14:15.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.039+0000 7f7066ffd700 1 -- 172.21.15.153:0/3767416582 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7060004750 con 0x7f7070105fb0 2024-04-24T08:14:15.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.039+0000 7f7066ffd700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f705c0a3c40 0x7f705c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:15.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.040+0000 7f7066ffd700 1 -- 172.21.15.153:0/3767416582 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3669+0+0 (secure 0 0 0) 0x7f70600ced90 con 0x7f7070105fb0 2024-04-24T08:14:15.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.040+0000 7f7075e7c700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f705c0a3c40 0x7f705c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:15.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.040+0000 7f7075e7c700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f705c0a3c40 0x7f705c0a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f706c009a10 tx=0x7f706c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:15.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.044+0000 7f7066ffd700 1 -- 172.21.15.153:0/3767416582 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7060014030 con 0x7f7070105fb0 2024-04-24T08:14:15.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:15.253+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7070062f30 con 0x7f705c0a3c40 2024-04-24T08:14:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: osdmap e30: 4 total, 4 up, 4 in 2024-04-24T08:14:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi146 2024-04-24T08:14:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi146 to 8544M 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:14:15.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:15 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: osdmap e30: 4 total, 4 up, 4 in 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi146 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi146 to 8544M 2024-04-24T08:14:15.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:15.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:15.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:14:15.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:14:15.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:14:15.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:15 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: osdmap e30: 4 total, 4 up, 4 in 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi146 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi146 to 8544M 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:15.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:15.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:15.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:14:15.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:14:15.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:14:15.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:15 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:16.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:16 smithi153 ceph-mon[32746]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-24T08:14:16.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:16 smithi146 ceph-mon[27805]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-24T08:14:17.064 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:16 smithi177 ceph-mon[33046]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-24T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:18 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/2871421722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]: dispatch 2024-04-24T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:18 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]: dispatch 2024-04-24T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:18 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]': finished 2024-04-24T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:18 smithi153 ceph-mon[32746]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:18 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:18 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/2871421722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]: dispatch 2024-04-24T08:14:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:18 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]: dispatch 2024-04-24T08:14:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:18 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]': finished 2024-04-24T08:14:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:18 smithi177 ceph-mon[33046]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T08:14:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:18 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:18.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:18 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/2871421722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]: dispatch 2024-04-24T08:14:18.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:18 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]: dispatch 2024-04-24T08:14:18.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:18 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f96f1d5-f2a7-4b27-8f7e-1d440866acd3"}]': finished 2024-04-24T08:14:18.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:18 smithi146 ceph-mon[27805]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T08:14:18.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:19.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:19 smithi177 ceph-mon[33046]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-24T08:14:19.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:19 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/589875958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:14:19.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:19 smithi153 ceph-mon[32746]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-24T08:14:19.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:19 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/589875958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:14:19.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:19 smithi146 ceph-mon[27805]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-24T08:14:19.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:19 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/589875958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:14:21.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:21 smithi153 ceph-mon[32746]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-24T08:14:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:21 smithi146 ceph-mon[27805]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-24T08:14:21.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:21 smithi177 ceph-mon[33046]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-24T08:14:23.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:23 smithi153 ceph-mon[32746]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:14:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:23 smithi146 ceph-mon[27805]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:14:23.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:23 smithi177 ceph-mon[33046]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:14:25.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:25 smithi153 ceph-mon[32746]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T08:14:25.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:25 smithi146 ceph-mon[27805]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T08:14:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:25 smithi177 ceph-mon[33046]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T08:14:26.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T08:14:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:26 smithi153 ceph-mon[32746]: Deploying daemon osd.4 on smithi153 2024-04-24T08:14:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T08:14:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:26 smithi146 ceph-mon[27805]: Deploying daemon osd.4 on smithi153 2024-04-24T08:14:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T08:14:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:26 smithi177 ceph-mon[33046]: Deploying daemon osd.4 on smithi153 2024-04-24T08:14:27.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:27 smithi153 ceph-mon[32746]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:27.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:27 smithi146 ceph-mon[27805]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:27 smithi177 ceph-mon[33046]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:29.563 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:29 smithi153 ceph-mon[32746]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:29.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:29 smithi146 ceph-mon[27805]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:29.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:29 smithi177 ceph-mon[33046]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:31.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:31 smithi153 ceph-mon[32746]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:31 smithi146 ceph-mon[27805]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:31 smithi177 ceph-mon[33046]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:33.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:33 smithi153 ceph-mon[32746]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:33 smithi146 ceph-mon[27805]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:33.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:33 smithi177 ceph-mon[33046]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:34.398 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-04-24T08:14:34.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.396+0000 7f7066ffd700 1 -- 172.21.15.153:0/3767416582 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7070062f30 con 0x7f705c0a3c40 2024-04-24T08:14:34.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f705c0a3c40 msgr2=0x7f705c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:34.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f705c0a3c40 0x7f705c0a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f706c009a10 tx=0x7f706c009190 comp rx=0 tx=0).stop 2024-04-24T08:14:34.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070105fb0 msgr2=0x7f7070199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:34.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070105fb0 0x7f7070199280 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f706000c920 tx=0x7f706000cc30 comp rx=0 tx=0).stop 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 shutdown_connections 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f705c0a3c40 0x7f705c0a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70701088e0 0x7f70701a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7070103680 0x7f7070198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 --2- 172.21.15.153:0/3767416582 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7070105fb0 0x7f7070199280 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.399+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 >> 172.21.15.153:0/3767416582 conn(0x7f70700fd010 msgr2=0x7f7070103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.400+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 shutdown_connections 2024-04-24T08:14:34.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:14:34.400+0000 7f70780e0700 1 -- 172.21.15.153:0/3767416582 wait complete. 2024-04-24T08:14:35.031 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.4.service 2024-04-24T08:14:35.034 INFO:tasks.cephadm:Deploying osd.5 on smithi177 with /dev/vg_nvme/lv_3... 2024-04-24T08:14:35.034 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T08:14:35.279 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:14:35.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:35 smithi177 ceph-mon[33046]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:35.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:35 smithi177 ceph-mon[33046]: from='osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:14:35.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:35 smithi177 ceph-mon[33046]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:14:35.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.612 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:35 smithi153 ceph-mon[32746]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:35 smithi153 ceph-mon[32746]: from='osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:14:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:35 smithi153 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:14:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:35 smithi146 ceph-mon[27805]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:35.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:35 smithi146 ceph-mon[27805]: from='osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:14:35.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:35 smithi146 ceph-mon[27805]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:14:35.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:35.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:36.680 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:36 smithi153 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T08:14:36.680 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:36 smithi153 ceph-mon[32746]: from='osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:14:36.680 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:36 smithi153 ceph-mon[32746]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T08:14:36.680 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:36.680 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:36 smithi153 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:14:36.738 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:36 smithi177 ceph-mon[33046]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T08:14:36.738 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:36 smithi177 ceph-mon[33046]: from='osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:14:36.738 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:36 smithi177 ceph-mon[33046]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T08:14:36.738 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:36.738 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:36 smithi177 ceph-mon[33046]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:14:36.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:36 smithi146 ceph-mon[27805]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T08:14:36.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:36 smithi146 ceph-mon[27805]: from='osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:14:36.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:36 smithi146 ceph-mon[27805]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T08:14:36.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:36.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:36 smithi146 ceph-mon[27805]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:14:37.429 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T08:14:37.429 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T08:14:37.430 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-24T08:14:37.430 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-24T08:14:37.430 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112401 s, 933 MB/s 2024-04-24T08:14:37.430 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 2024-04-24T08:14:37.430 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-24T08:14:37.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-mon[32746]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:14:37.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-mon[32746]: osdmap e33: 5 total, 4 up, 5 in 2024-04-24T08:14:37.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-mon[32746]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:37.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:37.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-mon[32746]: from='osd.4 ' entity='osd.4' 2024-04-24T08:14:37.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:37.711 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:14:37.316+0000 7f43e6583700 -1 osd.4 0 waiting for initial osdmap 2024-04-24T08:14:37.711 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:14:37 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:14:37.474+0000 7f43de36e700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:14:37.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:37 smithi146 ceph-mon[27805]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:14:37.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:37 smithi146 ceph-mon[27805]: osdmap e33: 5 total, 4 up, 5 in 2024-04-24T08:14:37.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:37 smithi146 ceph-mon[27805]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:37.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:37 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:37.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:37 smithi146 ceph-mon[27805]: from='osd.4 ' entity='osd.4' 2024-04-24T08:14:37.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:37 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:37.753 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:37 smithi177 ceph-mon[33046]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:14:37.753 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:37 smithi177 ceph-mon[33046]: osdmap e33: 5 total, 4 up, 5 in 2024-04-24T08:14:37.753 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:37 smithi177 ceph-mon[33046]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:37.753 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:37 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:37.753 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:37 smithi177 ceph-mon[33046]: from='osd.4 ' entity='osd.4' 2024-04-24T08:14:37.753 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:37 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:38.081 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-04-24T08:14:38.325 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:14:38.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:38 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:14:38.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:38 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:14:38.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:38 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:38.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:38 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:14:38.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:38 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:14:38.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:38 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:38.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:38 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:14:38.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:38 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:14:38.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:38 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:39.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:39.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] boot 2024-04-24T08:14:39.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T08:14:39.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:39.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi153 2024-04-24T08:14:39.528 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:39.529 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:39.530 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] boot 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi153 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:39.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:39.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:39.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:39.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 -- 172.21.15.177:0/3799543347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb4103e20 msgr2=0x7effb4104280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:39.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 --2- 172.21.15.177:0/3799543347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb4103e20 0x7effb4104280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7effa8009990 tx=0x7effa8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:14:39.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 -- 172.21.15.177:0/3799543347 shutdown_connections 2024-04-24T08:14:39.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 --2- 172.21.15.177:0/3799543347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effb41047c0 0x7effb410b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:39.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 --2- 172.21.15.177:0/3799543347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effb4100e40 0x7effb4101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:39.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 --2- 172.21.15.177:0/3799543347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb4103e20 0x7effb4104280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:39.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 -- 172.21.15.177:0/3799543347 >> 172.21.15.177:0/3799543347 conn(0x7effb40fc7a0 msgr2=0x7effb40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:39.734 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 -- 172.21.15.177:0/3799543347 shutdown_connections 2024-04-24T08:14:39.734 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.732+0000 7effb8e86700 1 -- 172.21.15.177:0/3799543347 wait complete. 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 Processor -- start 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 -- start start 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effb4100e40 0x7effb4198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effb4103e20 0x7effb41992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb41047c0 0x7effb41a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7effb410e7f0 con 0x7effb4100e40 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7effb410e670 con 0x7effb4103e20 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb8e86700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7effb410e970 con 0x7effb41047c0 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb2d9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb41047c0 0x7effb41a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb2d9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb41047c0 0x7effb41a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:57538/0 (socket says 172.21.15.177:57538) 2024-04-24T08:14:39.736 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb2d9d700 1 -- 172.21.15.177:0/1796726710 learned_addr learned my addr 172.21.15.177:0/1796726710 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.733+0000 7effb1d9b700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effb4103e20 0x7effb41992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb2d9d700 1 -- 172.21.15.177:0/1796726710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effb4103e20 msgr2=0x7effb41992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb2d9d700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effb4103e20 0x7effb41992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb2d9d700 1 -- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effb4100e40 msgr2=0x7effb4198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb2d9d700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effb4100e40 0x7effb4198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb2d9d700 1 -- 172.21.15.177:0/1796726710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb41a0a00 con 0x7effb41047c0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb2d9d700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb41047c0 0x7effb41a0360 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7effac00ef60 tx=0x7effac00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effa37fe700 1 -- 172.21.15.177:0/1796726710 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac0092e0 con 0x7effb41047c0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effa37fe700 1 -- 172.21.15.177:0/1796726710 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effac004bb0 con 0x7effb41047c0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb41a0c90 con 0x7effb41047c0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effa37fe700 1 -- 172.21.15.177:0/1796726710 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac004280 con 0x7effb41047c0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.734+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb41a11c0 con 0x7effb41047c0 2024-04-24T08:14:39.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.735+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb40697e0 con 0x7effb41047c0 2024-04-24T08:14:39.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.736+0000 7effa37fe700 1 -- 172.21.15.177:0/1796726710 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7effac01b030 con 0x7effb41047c0 2024-04-24T08:14:39.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.737+0000 7effa37fe700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7eff9c0a3cc0 0x7eff9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:39.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.737+0000 7effa37fe700 1 -- 172.21.15.177:0/1796726710 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7effac0cf790 con 0x7effb41047c0 2024-04-24T08:14:39.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.737+0000 7effb259c700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7eff9c0a3cc0 0x7eff9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:39.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.738+0000 7effb259c700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7eff9c0a3cc0 0x7eff9c0a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7effb406b060 tx=0x7effa4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:39.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.741+0000 7effa37fe700 1 -- 172.21.15.177:0/1796726710 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effac022020 con 0x7effb41047c0 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: osd.4 [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] boot 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi153 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi153 to 8544M 2024-04-24T08:14:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:39.947 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:39.946+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7effb4062f30 con 0x7eff9c0a3cc0 2024-04-24T08:14:40.874 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:40 smithi177 ceph-mon[33046]: osdmap e35: 5 total, 5 up, 5 in 2024-04-24T08:14:40.874 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:40 smithi177 ceph-mon[33046]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:14:40.874 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:14:40.874 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:14:40.874 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:40.875 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:40 smithi177 ceph-mon[33046]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:41.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:40 smithi153 ceph-mon[32746]: osdmap e35: 5 total, 5 up, 5 in 2024-04-24T08:14:41.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:40 smithi153 ceph-mon[32746]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:14:41.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:14:41.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:14:41.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:41.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:40 smithi153 ceph-mon[32746]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:41.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:40 smithi146 ceph-mon[27805]: osdmap e35: 5 total, 5 up, 5 in 2024-04-24T08:14:41.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:40 smithi146 ceph-mon[27805]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:14:41.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:14:41.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:14:41.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:41.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:40 smithi146 ceph-mon[27805]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:43.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:43 smithi153 ceph-mon[32746]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:43 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/1409193025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]: dispatch 2024-04-24T08:14:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:43 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]: dispatch 2024-04-24T08:14:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:43 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]': finished 2024-04-24T08:14:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:43 smithi153 ceph-mon[32746]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T08:14:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:14:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:43 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/269146819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:14:43.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:43 smithi146 ceph-mon[27805]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:43 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/1409193025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]: dispatch 2024-04-24T08:14:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:43 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]: dispatch 2024-04-24T08:14:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:43 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]': finished 2024-04-24T08:14:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:43 smithi146 ceph-mon[27805]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T08:14:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:14:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:43 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/269146819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:14:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:43 smithi177 ceph-mon[33046]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:43 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/1409193025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]: dispatch 2024-04-24T08:14:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:43 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]: dispatch 2024-04-24T08:14:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:43 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad461bd7-3a6c-47cc-aa4c-39793fc77bc6"}]': finished 2024-04-24T08:14:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:43 smithi177 ceph-mon[33046]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T08:14:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:14:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:43 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/269146819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:14:45.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:45 smithi153 ceph-mon[32746]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:45 smithi146 ceph-mon[27805]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:45.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:45 smithi177 ceph-mon[33046]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:47.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:47 smithi153 ceph-mon[32746]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:47.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:47 smithi146 ceph-mon[27805]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:47.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:47 smithi177 ceph-mon[33046]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:49.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:49 smithi153 ceph-mon[32746]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:49.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:49 smithi146 ceph-mon[27805]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:49 smithi177 ceph-mon[33046]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:50.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:50 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T08:14:50.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:50 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:50.732 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T08:14:50.732 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:50 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:50.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:50 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T08:14:50.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:50 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:51.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:51 smithi153 ceph-mon[32746]: Deploying daemon osd.5 on smithi177 2024-04-24T08:14:51.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:51 smithi153 ceph-mon[32746]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:51.729 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:51 smithi177 ceph-mon[33046]: Deploying daemon osd.5 on smithi177 2024-04-24T08:14:51.729 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:51 smithi177 ceph-mon[33046]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:51 smithi146 ceph-mon[27805]: Deploying daemon osd.5 on smithi177 2024-04-24T08:14:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:51 smithi146 ceph-mon[27805]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:53.554 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:53 smithi177 ceph-mon[33046]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:53 smithi146 ceph-mon[27805]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:53.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:53 smithi153 ceph-mon[32746]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:55.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:55 smithi146 ceph-mon[27805]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:55.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:55 smithi177 ceph-mon[33046]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:55.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:55 smithi153 ceph-mon[32746]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:55.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:55 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:55 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:55 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:57.732 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:57 smithi177 ceph-mon[33046]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:57.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:57 smithi146 ceph-mon[27805]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:57.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:57 smithi153 ceph-mon[32746]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:58.879 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 5 on host 'smithi177' 2024-04-24T08:14:58.879 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.877+0000 7effa37fe700 1 -- 172.21.15.177:0/1796726710 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7effb4062f30 con 0x7eff9c0a3cc0 2024-04-24T08:14:58.881 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7eff9c0a3cc0 msgr2=0x7eff9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7eff9c0a3cc0 0x7eff9c0a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7effb406b060 tx=0x7effa4006d20 comp rx=0 tx=0).stop 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb41047c0 msgr2=0x7effb41a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb41047c0 0x7effb41a0360 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7effac00ef60 tx=0x7effac00ef90 comp rx=0 tx=0).stop 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 shutdown_connections 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7effb41047c0 0x7effb41a0360 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7eff9c0a3cc0 0x7eff9c0a6180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effb4100e40 0x7effb4198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 --2- 172.21.15.177:0/1796726710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7effb4103e20 0x7effb41992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 >> 172.21.15.177:0/1796726710 conn(0x7effb40fc7a0 msgr2=0x7effb41070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 shutdown_connections 2024-04-24T08:14:58.882 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:14:58.879+0000 7effb8e86700 1 -- 172.21.15.177:0/1796726710 wait complete. 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.070 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.071 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:14:59.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:59.506 DEBUG:teuthology.orchestra.run.smithi177:osd.5> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.5.service 2024-04-24T08:14:59.509 INFO:tasks.cephadm:Deploying osd.6 on smithi146 with /dev/vg_nvme/lv_2... 2024-04-24T08:14:59.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:15:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:59 smithi177 ceph-mon[33046]: from='osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:15:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:14:59 smithi177 ceph-mon[33046]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:15:00.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:59 smithi146 ceph-mon[27805]: from='osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:15:00.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:14:59 smithi146 ceph-mon[27805]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:15:00.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:59 smithi153 ceph-mon[32746]: from='osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:15:00.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:14:59 smithi153 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:15:01.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:01 smithi146 ceph-mon[27805]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T08:15:01.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:01 smithi146 ceph-mon[27805]: from='osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:15:01.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:01 smithi146 ceph-mon[27805]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T08:15:01.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:01 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:01.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:01 smithi146 ceph-mon[27805]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:15:01.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:01 smithi146 ceph-mon[27805]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:15:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:01 smithi177 ceph-mon[33046]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T08:15:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:01 smithi177 ceph-mon[33046]: from='osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:15:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:01 smithi177 ceph-mon[33046]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T08:15:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:01 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:01 smithi177 ceph-mon[33046]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:15:01.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:01 smithi177 ceph-mon[33046]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:15:01.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:01 smithi153 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T08:15:01.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:01 smithi153 ceph-mon[32746]: from='osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:15:01.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:01 smithi153 ceph-mon[32746]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T08:15:01.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:01 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:01.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:01 smithi153 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:15:01.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:01 smithi153 ceph-mon[32746]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:15:02.060 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:02 smithi177 ceph-mon[33046]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:15:02.060 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:02 smithi177 ceph-mon[33046]: osdmap e38: 6 total, 5 up, 6 in 2024-04-24T08:15:02.060 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:02.060 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:02 smithi177 ceph-mon[33046]: from='osd.5 ' entity='osd.5' 2024-04-24T08:15:02.060 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:02 smithi177 ceph-mon[33046]: osdmap e39: 6 total, 5 up, 6 in 2024-04-24T08:15:02.060 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:02.060 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:15:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:15:01.904+0000 7f791a83c700 -1 osd.5 0 waiting for initial osdmap 2024-04-24T08:15:02.060 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:15:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:15:02.058+0000 7f7912627700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:15:02.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:02 smithi153 ceph-mon[32746]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:15:02.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:02 smithi153 ceph-mon[32746]: osdmap e38: 6 total, 5 up, 6 in 2024-04-24T08:15:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:02 smithi153 ceph-mon[32746]: from='osd.5 ' entity='osd.5' 2024-04-24T08:15:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:02 smithi153 ceph-mon[32746]: osdmap e39: 6 total, 5 up, 6 in 2024-04-24T08:15:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:02 smithi146 ceph-mon[27805]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:15:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:02 smithi146 ceph-mon[27805]: osdmap e38: 6 total, 5 up, 6 in 2024-04-24T08:15:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:02 smithi146 ceph-mon[27805]: from='osd.5 ' entity='osd.5' 2024-04-24T08:15:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:02 smithi146 ceph-mon[27805]: osdmap e39: 6 total, 5 up, 6 in 2024-04-24T08:15:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:02.885 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T08:15:02.885 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T08:15:02.885 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-04-24T08:15:02.885 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-04-24T08:15:02.885 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0359111 s, 292 MB/s 2024-04-24T08:15:02.885 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi177 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] boot 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T08:15:03.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi177 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:03.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] boot 2024-04-24T08:15:03.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T08:15:03.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:03.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi177 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: osd.5 [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] boot 2024-04-24T08:15:03.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T08:15:03.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:15:03.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2024-04-24T08:15:04.194 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:15:05.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:05 smithi177 ceph-mon[33046]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T08:15:05.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:05 smithi177 ceph-mon[33046]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:05.319 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:05 smithi146 ceph-mon[27805]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T08:15:05.319 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:05 smithi146 ceph-mon[27805]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:05.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:05 smithi153 ceph-mon[32746]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T08:15:05.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:05 smithi153 ceph-mon[32746]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 -- 172.21.15.146:0/1780320214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10a390 msgr2=0x7fc84c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 --2- 172.21.15.146:0/1780320214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10a390 0x7fc84c10a7f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8380099c0 tx=0x7fc838009cd0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 -- 172.21.15.146:0/1780320214 shutdown_connections 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 --2- 172.21.15.146:0/1780320214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c10ad30 0x7fc84c1115e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 --2- 172.21.15.146:0/1780320214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 0x7fc84c1095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 --2- 172.21.15.146:0/1780320214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10a390 0x7fc84c10a7f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 -- 172.21.15.146:0/1780320214 >> 172.21.15.146:0/1780320214 conn(0x7fc84c078160 msgr2=0x7fc84c07a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 -- 172.21.15.146:0/1780320214 shutdown_connections 2024-04-24T08:15:06.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.128+0000 7fc851400700 1 -- 172.21.15.146:0/1780320214 wait complete. 2024-04-24T08:15:06.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.129+0000 7fc851400700 1 Processor -- start 2024-04-24T08:15:06.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc851400700 1 -- start start 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc851400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 0x7fc84c19f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc851400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c10a390 0x7fc84c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc851400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10ad30 0x7fc84c1a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc851400700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c114da0 con 0x7fc84c10ad30 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc851400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c114c20 con 0x7fc84c10a390 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc851400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c114f20 con 0x7fc84c109190 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc84a800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 0x7fc84c19f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc84a800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 0x7fc84c19f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:36246/0 (socket says 172.21.15.146:36246) 2024-04-24T08:15:06.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc84a800700 1 -- 172.21.15.146:0/711517161 learned_addr learned my addr 172.21.15.146:0/711517161 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:15:06.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.130+0000 7fc84b200700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10ad30 0x7fc84c1a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc849e00700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c10a390 0x7fc84c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc84a800700 1 -- 172.21.15.146:0/711517161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c10a390 msgr2=0x7fc84c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc84a800700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c10a390 0x7fc84c19f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc84a800700 1 -- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10ad30 msgr2=0x7fc84c1a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc84a800700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10ad30 0x7fc84c1a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc84a800700 1 -- 172.21.15.146:0/711517161 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c1a6fb0 con 0x7fc84c109190 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc849e00700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c10a390 0x7fc84c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc84b200700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10ad30 0x7fc84c1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.131+0000 7fc84a800700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 0x7fc84c19f350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00e9a0 tx=0x7fc83c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.132+0000 7fc843400700 1 -- 172.21.15.146:0/711517161 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c004220 con 0x7fc84c109190 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.132+0000 7fc843400700 1 -- 172.21.15.146:0/711517161 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc83c0043c0 con 0x7fc84c109190 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.132+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c1a72a0 con 0x7fc84c109190 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.132+0000 7fc843400700 1 -- 172.21.15.146:0/711517161 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c003df0 con 0x7fc84c109190 2024-04-24T08:15:06.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.132+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c1a7830 con 0x7fc84c109190 2024-04-24T08:15:06.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.133+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc84c0697e0 con 0x7fc84c109190 2024-04-24T08:15:06.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.135+0000 7fc843400700 1 -- 172.21.15.146:0/711517161 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc83c00c810 con 0x7fc84c109190 2024-04-24T08:15:06.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.136+0000 7fc843400700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc8300a3d40 0x7fc8300a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.136+0000 7fc843400700 1 -- 172.21.15.146:0/711517161 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4533+0+0 (secure 0 0 0) 0x7fc83c0cf700 con 0x7fc84c109190 2024-04-24T08:15:06.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.136+0000 7fc849e00700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc8300a3d40 0x7fc8300a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.137+0000 7fc849e00700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc8300a3d40 0x7fc8300a6200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc838000c00 tx=0x7fc838005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.140+0000 7fc843400700 1 -- 172.21.15.146:0/711517161 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc83c09cf00 con 0x7fc84c109190 2024-04-24T08:15:06.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:06 smithi146 ceph-mon[27805]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T08:15:06.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:06 smithi146 ceph-mon[27805]: osdmap e43: 6 total, 6 up, 6 in 2024-04-24T08:15:06.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:06 smithi177 ceph-mon[33046]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T08:15:06.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:06 smithi177 ceph-mon[33046]: osdmap e43: 6 total, 6 up, 6 in 2024-04-24T08:15:06.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:06.375+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc84c062f30 con 0x7fc8300a3d40 2024-04-24T08:15:06.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:06 smithi153 ceph-mon[32746]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T08:15:06.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:06 smithi153 ceph-mon[32746]: osdmap e43: 6 total, 6 up, 6 in 2024-04-24T08:15:07.290 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:07 smithi146 ceph-mon[27805]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:15:07.290 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:07 smithi146 ceph-mon[27805]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:15:07.290 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:15:07.290 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:15:07.290 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:07.290 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:07 smithi146 ceph-mon[27805]: osdmap e44: 6 total, 6 up, 6 in 2024-04-24T08:15:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:07 smithi177 ceph-mon[33046]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:15:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:07 smithi177 ceph-mon[33046]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:15:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:15:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:15:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:07 smithi177 ceph-mon[33046]: osdmap e44: 6 total, 6 up, 6 in 2024-04-24T08:15:07.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:07 smithi153 ceph-mon[32746]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:15:07.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:07 smithi153 ceph-mon[32746]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:15:07.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:15:07.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:15:07.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:07.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:07 smithi153 ceph-mon[32746]: osdmap e44: 6 total, 6 up, 6 in 2024-04-24T08:15:09.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:09 smithi153 ceph-mon[32746]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-24T08:15:09.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:09 smithi146 ceph-mon[27805]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-24T08:15:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:09 smithi177 ceph-mon[33046]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-04-24T08:15:10.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:10 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3654090056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]: dispatch 2024-04-24T08:15:10.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:10 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]: dispatch 2024-04-24T08:15:10.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:10 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]': finished 2024-04-24T08:15:10.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:10 smithi153 ceph-mon[32746]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T08:15:10.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:10.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:10 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3654090056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]: dispatch 2024-04-24T08:15:10.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:10 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]: dispatch 2024-04-24T08:15:10.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:10 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]': finished 2024-04-24T08:15:10.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:10 smithi146 ceph-mon[27805]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T08:15:10.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:10 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3654090056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]: dispatch 2024-04-24T08:15:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:10 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]: dispatch 2024-04-24T08:15:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:10 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3257f43-689c-4fa9-b182-d9783ef1aa31"}]': finished 2024-04-24T08:15:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:10 smithi177 ceph-mon[33046]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T08:15:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:11.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:11 smithi153 ceph-mon[32746]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-24T08:15:11.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:11 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/904570788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:15:11.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:11 smithi146 ceph-mon[27805]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-24T08:15:11.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:11 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/904570788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:15:11.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:11 smithi177 ceph-mon[33046]: pgmap v162: 1 pgs: 1 peering; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-24T08:15:11.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:11 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/904570788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:15:13.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:13 smithi153 ceph-mon[32746]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-24T08:15:13.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:13 smithi146 ceph-mon[27805]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-24T08:15:13.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:13 smithi177 ceph-mon[33046]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-24T08:15:15.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:15 smithi146 ceph-mon[27805]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:15 smithi177 ceph-mon[33046]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:15.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:15 smithi153 ceph-mon[32746]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:17.708 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:17 smithi146 ceph-mon[27805]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:17 smithi177 ceph-mon[33046]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:17.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:17 smithi153 ceph-mon[32746]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:19.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:19 smithi146 ceph-mon[27805]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:19.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:19 smithi177 ceph-mon[33046]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:19.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:19 smithi153 ceph-mon[32746]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:20.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T08:15:20.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:20.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T08:15:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:20.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T08:15:20.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:21.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:21 smithi146 ceph-mon[27805]: Deploying daemon osd.6 on smithi146 2024-04-24T08:15:21.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:21 smithi146 ceph-mon[27805]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:21.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:21 smithi177 ceph-mon[33046]: Deploying daemon osd.6 on smithi146 2024-04-24T08:15:21.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:21 smithi177 ceph-mon[33046]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:21.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:21 smithi153 ceph-mon[32746]: Deploying daemon osd.6 on smithi146 2024-04-24T08:15:21.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:21 smithi153 ceph-mon[32746]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:23 smithi146 ceph-mon[27805]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:23 smithi177 ceph-mon[33046]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:23.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:23 smithi153 ceph-mon[32746]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:25.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:25 smithi146 ceph-mon[27805]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:25 smithi177 ceph-mon[33046]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:25.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:25 smithi153 ceph-mon[32746]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:27.581 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:27 smithi146 ceph-mon[27805]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:27.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:27 smithi153 ceph-mon[32746]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:27.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:27 smithi177 ceph-mon[33046]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:28.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:28.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:28 smithi146 ceph-mon[27805]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:28 smithi177 ceph-mon[33046]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:29.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:29.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:29.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:29.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:28 smithi153 ceph-mon[32746]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:31.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:31 smithi153 ceph-mon[32746]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:31 smithi146 ceph-mon[27805]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:31.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:31.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:31 smithi177 ceph-mon[33046]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:32.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.567+0000 7fc843400700 1 -- 172.21.15.146:0/711517161 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc84c062f30 con 0x7fc8300a3d40 2024-04-24T08:15:32.568 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 6 on host 'smithi146' 2024-04-24T08:15:32.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc8300a3d40 msgr2=0x7fc8300a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc8300a3d40 0x7fc8300a6200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc838000c00 tx=0x7fc838005fb0 comp rx=0 tx=0).stop 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 msgr2=0x7fc84c19f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 0x7fc84c19f350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00e9a0 tx=0x7fc83c00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 shutdown_connections 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc8300a3d40 0x7fc8300a6200 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c10ad30 0x7fc84c1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c109190 0x7fc84c19f350 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 --2- 172.21.15.146:0/711517161 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c10a390 0x7fc84c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 >> 172.21.15.146:0/711517161 conn(0x7fc84c078160 msgr2=0x7fc84c10d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:32.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 shutdown_connections 2024-04-24T08:15:32.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:15:32.571+0000 7fc851400700 1 -- 172.21.15.146:0/711517161 wait complete. 2024-04-24T08:15:32.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:32 smithi153 ceph-mon[32746]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:15:32.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:32 smithi146 ceph-mon[27805]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:15:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:32 smithi177 ceph-mon[33046]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:15:33.475 DEBUG:teuthology.orchestra.run.smithi146:osd.6> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.6.service 2024-04-24T08:15:33.481 INFO:tasks.cephadm:Deploying osd.7 on smithi153 with /dev/vg_nvme/lv_2... 2024-04-24T08:15:33.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T08:15:33.728 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:15:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T08:15:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: osdmap e46: 7 total, 6 up, 7 in 2024-04-24T08:15:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:15:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:33.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: osdmap e46: 7 total, 6 up, 7 in 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:33.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:33.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T08:15:33.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: osdmap e46: 7 total, 6 up, 7 in 2024-04-24T08:15:33.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:15:33.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:33.851 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:33.851 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:33.851 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:34.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:34 smithi146 ceph-mon[27805]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:15:34.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:34 smithi146 ceph-mon[27805]: osdmap e47: 7 total, 6 up, 7 in 2024-04-24T08:15:34.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:34 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:34.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:34 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:34 smithi177 ceph-mon[33046]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:15:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:34 smithi177 ceph-mon[33046]: osdmap e47: 7 total, 6 up, 7 in 2024-04-24T08:15:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:34 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:34 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:34.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:34 smithi153 ceph-mon[32746]: from='osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:15:34.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:34 smithi153 ceph-mon[32746]: osdmap e47: 7 total, 6 up, 7 in 2024-04-24T08:15:34.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:34 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:34.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:34 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:35.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:35 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:15:35.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:35 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:15:35.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:35 smithi153 ceph-mon[32746]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:35.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:35 smithi153 ceph-mon[32746]: osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] boot 2024-04-24T08:15:35.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:35 smithi153 ceph-mon[32746]: osdmap e48: 7 total, 7 up, 7 in 2024-04-24T08:15:35.759 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:35.759 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T08:15:35.759 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T08:15:35.759 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-24T08:15:35.760 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-24T08:15:35.760 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120148 s, 873 MB/s 2024-04-24T08:15:35.760 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-24T08:15:35.776 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:35 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:15:35.776 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:35 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:15:35.776 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:35 smithi146 ceph-mon[27805]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:35.776 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:35 smithi146 ceph-mon[27805]: osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] boot 2024-04-24T08:15:35.776 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:35 smithi146 ceph-mon[27805]: osdmap e48: 7 total, 7 up, 7 in 2024-04-24T08:15:35.776 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:35 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:15:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:35 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:15:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:35 smithi177 ceph-mon[33046]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:15:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:35 smithi177 ceph-mon[33046]: osd.6 [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] boot 2024-04-24T08:15:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:35 smithi177 ceph-mon[33046]: osdmap e48: 7 total, 7 up, 7 in 2024-04-24T08:15:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:15:36.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-04-24T08:15:36.628 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:15:36.659 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:36 smithi153 ceph-mon[32746]: osdmap e49: 7 total, 7 up, 7 in 2024-04-24T08:15:36.680 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:36 smithi146 ceph-mon[27805]: osdmap e49: 7 total, 7 up, 7 in 2024-04-24T08:15:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:36 smithi177 ceph-mon[33046]: osdmap e49: 7 total, 7 up, 7 in 2024-04-24T08:15:37.608 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:37 smithi153 ceph-mon[32746]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:37.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:37 smithi177 ceph-mon[33046]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:37.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:37 smithi146 ceph-mon[27805]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 -- 172.21.15.153:0/256112301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa241035e0 msgr2=0x7faa241059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 --2- 172.21.15.153:0/256112301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa241035e0 0x7faa241059d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faa20009990 tx=0x7faa20009ca0 comp rx=0 tx=0).stop 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 -- 172.21.15.153:0/256112301 shutdown_connections 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 --2- 172.21.15.153:0/256112301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa24108840 0x7faa2410acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 --2- 172.21.15.153:0/256112301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa241035e0 0x7faa241059d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 --2- 172.21.15.153:0/256112301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa24105f10 0x7faa24108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 -- 172.21.15.153:0/256112301 >> 172.21.15.153:0/256112301 conn(0x7faa240fcff0 msgr2=0x7faa240ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 -- 172.21.15.153:0/256112301 shutdown_connections 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 -- 172.21.15.153:0/256112301 wait complete. 2024-04-24T08:15:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 Processor -- start 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 -- start start 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.156+0000 7faa2b797700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa241035e0 0x7faa24198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa2b797700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa24105f10 0x7faa24199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa2b797700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa24108840 0x7faa241a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa2b797700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faa2410e7a0 con 0x7faa24105f10 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa2b797700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faa2410e620 con 0x7faa24108840 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa2b797700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa2410e920 con 0x7faa241035e0 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa24108840 0x7faa241a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa24108840 0x7faa241a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58928/0 (socket says 172.21.15.153:58928) 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 -- 172.21.15.153:0/293650827 learned_addr learned my addr 172.21.15.153:0/293650827 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29533700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa241035e0 0x7faa24198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa28d32700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa24105f10 0x7faa24199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 -- 172.21.15.153:0/293650827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa241035e0 msgr2=0x7faa24198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:38.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa241035e0 0x7faa24198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 -- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa24105f10 msgr2=0x7faa24199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa24105f10 0x7faa24199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 -- 172.21.15.153:0/293650827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa241a0960 con 0x7faa24108840 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa28d32700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa24105f10 0x7faa24199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa29d34700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa24108840 0x7faa241a0290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00b9e0 tx=0x7faa1c00bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa1a7fc700 1 -- 172.21.15.153:0/293650827 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c019070 con 0x7faa24108840 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa241a0c50 con 0x7faa24108840 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa1a7fc700 1 -- 172.21.15.153:0/293650827 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa1c00ce10 con 0x7faa24108840 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa1a7fc700 1 -- 172.21.15.153:0/293650827 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c012360 con 0x7faa24108840 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.157+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa241a1140 con 0x7faa24108840 2024-04-24T08:15:38.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.158+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa240697e0 con 0x7faa24108840 2024-04-24T08:15:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.159+0000 7faa1a7fc700 1 -- 172.21.15.153:0/293650827 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7faa1c01c070 con 0x7faa24108840 2024-04-24T08:15:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.160+0000 7faa1a7fc700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faa100a3c40 0x7faa100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.160+0000 7faa1a7fc700 1 -- 172.21.15.153:0/293650827 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4965+0+0 (secure 0 0 0) 0x7faa1c0cec30 con 0x7faa24108840 2024-04-24T08:15:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.160+0000 7faa29533700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faa100a3c40 0x7faa100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:38.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.161+0000 7faa29533700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faa100a3c40 0x7faa100a6100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa20003890 tx=0x7faa20003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.164+0000 7faa1a7fc700 1 -- 172.21.15.153:0/293650827 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa1c09c320 con 0x7faa24108840 2024-04-24T08:15:38.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:38.371+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7faa24062f30 con 0x7faa100a3c40 2024-04-24T08:15:38.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:38 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:15:38.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:38 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:15:38.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:38 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:38 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:15:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:38 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:15:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:38 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:38.871 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:38 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:15:38.871 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:38 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:15:38.871 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:38 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:39.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:39.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:39 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:15:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:39.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:39 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:39 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:40 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi146 2024-04-24T08:15:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:40 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi146 to 5696M 2024-04-24T08:15:40.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:40 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi146 2024-04-24T08:15:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:40 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi146 to 5696M 2024-04-24T08:15:40.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:40 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi146 2024-04-24T08:15:40.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:40 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi146 to 5696M 2024-04-24T08:15:41.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:41 smithi153 ceph-mon[32746]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:41.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:41 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/3905594199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]: dispatch 2024-04-24T08:15:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:41 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]: dispatch 2024-04-24T08:15:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:41 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]': finished 2024-04-24T08:15:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:41 smithi153 ceph-mon[32746]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T08:15:41.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:41 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:15:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:41 smithi177 ceph-mon[33046]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:41 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/3905594199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]: dispatch 2024-04-24T08:15:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:41 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]: dispatch 2024-04-24T08:15:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:41 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]': finished 2024-04-24T08:15:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:41 smithi177 ceph-mon[33046]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T08:15:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:41 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:15:41.895 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:41 smithi146 ceph-mon[27805]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:41.895 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:41 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/3905594199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]: dispatch 2024-04-24T08:15:41.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:41 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]: dispatch 2024-04-24T08:15:41.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:41 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b062222-8a11-47ad-9389-ad3a5b1ed19d"}]': finished 2024-04-24T08:15:41.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:41 smithi146 ceph-mon[27805]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T08:15:41.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:41 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:15:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:42 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/2228965855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:15:42.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:42 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/2228965855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:15:42.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:42 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/2228965855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:15:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:43 smithi177 ceph-mon[33046]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:43.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:43 smithi153 ceph-mon[32746]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:43.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:43 smithi146 ceph-mon[27805]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:45.626 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:45 smithi153 ceph-mon[32746]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:45.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:45 smithi177 ceph-mon[33046]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:45.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:45 smithi146 ceph-mon[27805]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:47.653 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:47 smithi153 ceph-mon[32746]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:47 smithi177 ceph-mon[33046]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:47.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:47 smithi146 ceph-mon[27805]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:48 smithi177 ceph-mon[33046]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:48.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:48 smithi153 ceph-mon[32746]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:48.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:48 smithi146 ceph-mon[27805]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T08:15:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:49 smithi177 ceph-mon[33046]: Deploying daemon osd.7 on smithi153 2024-04-24T08:15:49.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T08:15:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:49 smithi153 ceph-mon[32746]: Deploying daemon osd.7 on smithi153 2024-04-24T08:15:49.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T08:15:49.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:49.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:49 smithi146 ceph-mon[27805]: Deploying daemon osd.7 on smithi153 2024-04-24T08:15:50.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:50 smithi153 ceph-mon[32746]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:50.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:50 smithi177 ceph-mon[33046]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:50.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:50 smithi146 ceph-mon[27805]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:53.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:53 smithi153 ceph-mon[32746]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:53 smithi146 ceph-mon[27805]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:53 smithi177 ceph-mon[33046]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:55.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:55.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:55.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:55 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:55.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:55 smithi177 ceph-mon[33046]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:55.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:55 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:55.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:55 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:55.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:55 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:55.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:55 smithi153 ceph-mon[32746]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:55.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:55 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:55 smithi146 ceph-mon[27805]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:57.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:57 smithi153 ceph-mon[32746]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:57.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:57.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:57.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:57 smithi146 ceph-mon[27805]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:57.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:57.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:57.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:57 smithi177 ceph-mon[33046]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:58.105 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 7 on host 'smithi153' 2024-04-24T08:15:58.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.103+0000 7faa1a7fc700 1 -- 172.21.15.153:0/293650827 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faa24062f30 con 0x7faa100a3c40 2024-04-24T08:15:58.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faa100a3c40 msgr2=0x7faa100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faa100a3c40 0x7faa100a6100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa20003890 tx=0x7faa20003960 comp rx=0 tx=0).stop 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa24108840 msgr2=0x7faa241a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa24108840 0x7faa241a0290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00b9e0 tx=0x7faa1c00bcf0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 shutdown_connections 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa24108840 0x7faa241a0290 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa241035e0 0x7faa24198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faa100a3c40 0x7faa100a6100 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 --2- 172.21.15.153:0/293650827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa24105f10 0x7faa24199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 >> 172.21.15.153:0/293650827 conn(0x7faa240fcff0 msgr2=0x7faa24103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 shutdown_connections 2024-04-24T08:15:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:15:58.106+0000 7faa2b797700 1 -- 172.21.15.153:0/293650827 wait complete. 2024-04-24T08:15:58.740 DEBUG:teuthology.orchestra.run.smithi153:osd.7> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.7.service 2024-04-24T08:15:58.742 INFO:tasks.cephadm:Deploying osd.8 on smithi177 with /dev/vg_nvme/lv_2... 2024-04-24T08:15:58.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T08:15:58.987 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:15:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:59 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:59 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:59 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:59 smithi177 ceph-mon[33046]: from='osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:15:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:59 smithi177 ceph-mon[33046]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:15:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:15:59 smithi177 ceph-mon[33046]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:59.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:59 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:59.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:59 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:59.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:59 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:59.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:59 smithi153 ceph-mon[32746]: from='osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:15:59.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:59 smithi153 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:15:59.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:15:59 smithi153 ceph-mon[32746]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:15:59.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:59 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:59.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:59 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:15:59.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:59 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:59.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:59 smithi146 ceph-mon[27805]: from='osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:15:59.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:59 smithi146 ceph-mon[27805]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:15:59.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:15:59 smithi146 ceph-mon[27805]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:16:00.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:00 smithi153 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T08:16:00.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:00 smithi153 ceph-mon[32746]: from='osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:16:00.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:00 smithi153 ceph-mon[32746]: osdmap e51: 8 total, 7 up, 8 in 2024-04-24T08:16:00.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:00 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:00.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:00 smithi153 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:16:00.082 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:16:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:16:00.069+0000 7f5a30845700 -1 osd.7 0 waiting for initial osdmap 2024-04-24T08:16:00.082 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:16:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:16:00.080+0000 7f5a2c638700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:16:00.326 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:00 smithi177 ceph-mon[33046]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T08:16:00.326 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:00 smithi177 ceph-mon[33046]: from='osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:16:00.326 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:00 smithi177 ceph-mon[33046]: osdmap e51: 8 total, 7 up, 8 in 2024-04-24T08:16:00.326 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:00 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:00.326 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:00 smithi177 ceph-mon[33046]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:16:00.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:00 smithi146 ceph-mon[27805]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T08:16:00.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:00 smithi146 ceph-mon[27805]: from='osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:16:00.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:00 smithi146 ceph-mon[27805]: osdmap e51: 8 total, 7 up, 8 in 2024-04-24T08:16:00.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:00 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:00.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:00 smithi146 ceph-mon[27805]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:16:01.043 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T08:16:01.043 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T08:16:01.043 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-24T08:16:01.043 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-24T08:16:01.043 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111232 s, 943 MB/s 2024-04-24T08:16:01.043 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-24T08:16:01.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:01 smithi153 ceph-mon[32746]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:16:01.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:01 smithi153 ceph-mon[32746]: osdmap e52: 8 total, 7 up, 8 in 2024-04-24T08:16:01.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:01 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:01.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:01 smithi153 ceph-mon[32746]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:16:01.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:01 smithi153 ceph-mon[32746]: osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] boot 2024-04-24T08:16:01.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:01 smithi153 ceph-mon[32746]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T08:16:01.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:01 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:01 smithi177 ceph-mon[33046]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:16:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:01 smithi177 ceph-mon[33046]: osdmap e52: 8 total, 7 up, 8 in 2024-04-24T08:16:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:01 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:01 smithi177 ceph-mon[33046]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:16:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:01 smithi177 ceph-mon[33046]: osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] boot 2024-04-24T08:16:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:01 smithi177 ceph-mon[33046]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T08:16:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:01 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:01.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:01 smithi146 ceph-mon[27805]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:16:01.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:01 smithi146 ceph-mon[27805]: osdmap e52: 8 total, 7 up, 8 in 2024-04-24T08:16:01.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:01 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:01.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:01 smithi146 ceph-mon[27805]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:16:01.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:01 smithi146 ceph-mon[27805]: osd.7 [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] boot 2024-04-24T08:16:01.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:01 smithi146 ceph-mon[27805]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T08:16:01.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:01 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:16:01.663 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-04-24T08:16:01.904 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:16:02.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:16:02.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:16:02.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T08:16:02.295 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:02.296 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:02 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:16:02.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:02.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:02 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:02.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:02 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:03 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi153 2024-04-24T08:16:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:03 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-24T08:16:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:03 smithi177 ceph-mon[33046]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:03 smithi177 ceph-mon[33046]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T08:16:03.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:03 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi153 2024-04-24T08:16:03.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:03 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-24T08:16:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:03 smithi153 ceph-mon[32746]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:03.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:03 smithi153 ceph-mon[32746]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T08:16:03.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:03 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi153 2024-04-24T08:16:03.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:03 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi153 to 5696M 2024-04-24T08:16:03.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:03 smithi146 ceph-mon[27805]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:03.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:03 smithi146 ceph-mon[27805]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T08:16:03.506 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 -- 172.21.15.177:0/2691949699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18103d50 msgr2=0x7f4c181041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:03.506 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/2691949699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18103d50 0x7f4c181041b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009990 tx=0x7f4c04009ca0 comp rx=0 tx=0).stop 2024-04-24T08:16:03.506 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 -- 172.21.15.177:0/2691949699 shutdown_connections 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/2691949699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c181046f0 0x7f4c1810afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/2691949699 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c18102b50 0x7f4c18102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/2691949699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18103d50 0x7f4c181041b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 -- 172.21.15.177:0/2691949699 >> 172.21.15.177:0/2691949699 conn(0x7f4c180fe0f0 msgr2=0x7f4c18100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 -- 172.21.15.177:0/2691949699 shutdown_connections 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.503+0000 7f4c1ef28700 1 -- 172.21.15.177:0/2691949699 wait complete. 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 Processor -- start 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 -- start start 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18102b50 0x7f4c18198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c18103d50 0x7f4c18199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c181046f0 0x7f4c181a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1810e7b0 con 0x7f4c18103d50 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1810e630 con 0x7f4c181046f0 2024-04-24T08:16:03.507 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ef28700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1810e930 con 0x7f4c18102b50 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ccc4700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18102b50 0x7f4c18198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ccc4700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18102b50 0x7f4c18198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:49008/0 (socket says 172.21.15.177:49008) 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.504+0000 7f4c1ccc4700 1 -- 172.21.15.177:0/192553038 learned_addr learned my addr 172.21.15.177:0/192553038 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1d4c5700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c181046f0 0x7f4c181a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ccc4700 1 -- 172.21.15.177:0/192553038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c181046f0 msgr2=0x7f4c181a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ccc4700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c181046f0 0x7f4c181a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ccc4700 1 -- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c18103d50 msgr2=0x7f4c18199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ccc4700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c18103d50 0x7f4c18199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ccc4700 1 -- 172.21.15.177:0/192553038 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c181a0a20 con 0x7f4c18102b50 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ccc4700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18102b50 0x7f4c18198dc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00d9b0 tx=0x7f4c0c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c15ffb700 1 -- 172.21.15.177:0/192553038 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c017070 con 0x7f4c18102b50 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c15ffb700 1 -- 172.21.15.177:0/192553038 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c0c009e10 con 0x7f4c18102b50 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c15ffb700 1 -- 172.21.15.177:0/192553038 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c00f620 con 0x7f4c18102b50 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c181a0d10 con 0x7f4c18102b50 2024-04-24T08:16:03.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.505+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c181a1220 con 0x7f4c18102b50 2024-04-24T08:16:03.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.506+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c1802dfe0 con 0x7f4c18102b50 2024-04-24T08:16:03.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.507+0000 7f4c15ffb700 1 -- 172.21.15.177:0/192553038 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4c0c009930 con 0x7f4c18102b50 2024-04-24T08:16:03.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.508+0000 7f4c15ffb700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c080a3cc0 0x7f4c080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:03.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.508+0000 7f4c15ffb700 1 -- 172.21.15.177:0/192553038 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5426+0+0 (secure 0 0 0) 0x7f4c0c0cf3f0 con 0x7f4c18102b50 2024-04-24T08:16:03.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.508+0000 7f4c17fff700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c080a3cc0 0x7f4c080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:03.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.509+0000 7f4c17fff700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c080a3cc0 0x7f4c080a6180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4c04005f50 tx=0x7f4c04005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:03.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.513+0000 7f4c15ffb700 1 -- 172.21.15.177:0/192553038 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c0c09c8f0 con 0x7f4c18102b50 2024-04-24T08:16:03.718 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:03.717+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4c18062f30 con 0x7f4c080a3cc0 2024-04-24T08:16:04.663 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:04 smithi177 ceph-mon[33046]: osdmap e56: 8 total, 8 up, 8 in 2024-04-24T08:16:04.663 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:04 smithi177 ceph-mon[33046]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:16:04.663 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:16:04.663 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:16:04.663 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:04.663 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:04 smithi177 ceph-mon[33046]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:04.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:04 smithi153 ceph-mon[32746]: osdmap e56: 8 total, 8 up, 8 in 2024-04-24T08:16:04.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:04 smithi153 ceph-mon[32746]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:16:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:16:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:16:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:04 smithi153 ceph-mon[32746]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:04 smithi146 ceph-mon[27805]: osdmap e56: 8 total, 8 up, 8 in 2024-04-24T08:16:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:04 smithi146 ceph-mon[27805]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:16:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:16:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:16:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:04 smithi146 ceph-mon[27805]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:05 smithi177 ceph-mon[33046]: osdmap e57: 8 total, 8 up, 8 in 2024-04-24T08:16:05.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:05 smithi153 ceph-mon[32746]: osdmap e57: 8 total, 8 up, 8 in 2024-04-24T08:16:05.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:05 smithi146 ceph-mon[27805]: osdmap e57: 8 total, 8 up, 8 in 2024-04-24T08:16:06.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:06 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/359687942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]: dispatch 2024-04-24T08:16:06.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:06 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]: dispatch 2024-04-24T08:16:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:06 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]': finished 2024-04-24T08:16:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:06 smithi153 ceph-mon[32746]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T08:16:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:06 smithi153 ceph-mon[32746]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:06.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:06 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/359687942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]: dispatch 2024-04-24T08:16:06.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:06 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]: dispatch 2024-04-24T08:16:06.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:06 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]': finished 2024-04-24T08:16:06.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:06 smithi146 ceph-mon[27805]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T08:16:06.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:06.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:06 smithi146 ceph-mon[27805]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:06 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/359687942' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]: dispatch 2024-04-24T08:16:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:06 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]: dispatch 2024-04-24T08:16:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:06 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30f8bd53-eba7-4942-8c3d-1f327abe42fa"}]': finished 2024-04-24T08:16:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:06 smithi177 ceph-mon[33046]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T08:16:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:06 smithi177 ceph-mon[33046]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:07.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:07 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/3105812251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:16:07.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:07 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/3105812251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:16:08.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:07 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/3105812251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:16:08.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:08 smithi153 ceph-mon[32746]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:08.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:08 smithi146 ceph-mon[27805]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:09.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:08 smithi177 ceph-mon[33046]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:11.705 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:11 smithi177 ceph-mon[33046]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:11.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:11 smithi153 ceph-mon[32746]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:11.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:11 smithi146 ceph-mon[27805]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:13.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:13 smithi153 ceph-mon[32746]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:13.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:13 smithi146 ceph-mon[27805]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:13.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:13 smithi177 ceph-mon[33046]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:14.719 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T08:16:14.719 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T08:16:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:14.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T08:16:14.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:15.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:15 smithi146 ceph-mon[27805]: Deploying daemon osd.8 on smithi177 2024-04-24T08:16:15.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:15 smithi146 ceph-mon[27805]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:15.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:15 smithi177 ceph-mon[33046]: Deploying daemon osd.8 on smithi177 2024-04-24T08:16:15.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:15 smithi177 ceph-mon[33046]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:15.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:15 smithi153 ceph-mon[32746]: Deploying daemon osd.8 on smithi177 2024-04-24T08:16:15.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:15 smithi153 ceph-mon[32746]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:17.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:17 smithi146 ceph-mon[27805]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:17.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:17 smithi177 ceph-mon[33046]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:17.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:17 smithi153 ceph-mon[32746]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:19.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:19 smithi146 ceph-mon[27805]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:19.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:19 smithi177 ceph-mon[33046]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:19.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:19 smithi153 ceph-mon[32746]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:20 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:20.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:20 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:20.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:20.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:20.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:20 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:21.782 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:21 smithi177 ceph-mon[33046]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:21.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:21 smithi153 ceph-mon[32746]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:21.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:21 smithi146 ceph-mon[27805]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:23.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.355+0000 7f4c15ffb700 1 -- 172.21.15.177:0/192553038 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4c18062f30 con 0x7f4c080a3cc0 2024-04-24T08:16:23.356 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 8 on host 'smithi177' 2024-04-24T08:16:23.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c080a3cc0 msgr2=0x7f4c080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:23.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c080a3cc0 0x7f4c080a6180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4c04005f50 tx=0x7f4c04005e80 comp rx=0 tx=0).stop 2024-04-24T08:16:23.358 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18102b50 msgr2=0x7f4c18198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18102b50 0x7f4c18198dc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00d9b0 tx=0x7f4c0c00dcc0 comp rx=0 tx=0).stop 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 shutdown_connections 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c181046f0 0x7f4c181a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c18102b50 0x7f4c18198dc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c080a3cc0 0x7f4c080a6180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 --2- 172.21.15.177:0/192553038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c18103d50 0x7f4c18199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 >> 172.21.15.177:0/192553038 conn(0x7f4c180fe0f0 msgr2=0x7f4c181070a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 shutdown_connections 2024-04-24T08:16:23.359 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:16:23.357+0000 7f4c1ef28700 1 -- 172.21.15.177:0/192553038 wait complete. 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:23.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:16:23.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:23 smithi177 ceph-mon[33046]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:16:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:23 smithi146 ceph-mon[27805]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:16:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:23 smithi153 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:16:24.027 DEBUG:teuthology.orchestra.run.smithi177:osd.8> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.8.service 2024-04-24T08:16:24.029 INFO:tasks.cephadm:Deploying osd.9 on smithi146 with /dev/vg_nvme/lv_1... 2024-04-24T08:16:24.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T08:16:24.469 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:16:25.659 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:25 smithi146 ceph-mon[27805]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T08:16:25.660 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:25 smithi146 ceph-mon[27805]: from='osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:16:25.660 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:25 smithi146 ceph-mon[27805]: osdmap e59: 9 total, 8 up, 9 in 2024-04-24T08:16:25.660 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:25 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:25.660 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:25 smithi146 ceph-mon[27805]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:16:25.660 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:25 smithi146 ceph-mon[27805]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:25.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:25 smithi153 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T08:16:25.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:25 smithi153 ceph-mon[32746]: from='osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:16:25.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:25 smithi153 ceph-mon[32746]: osdmap e59: 9 total, 8 up, 9 in 2024-04-24T08:16:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:25 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:25 smithi153 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:16:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:25 smithi153 ceph-mon[32746]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:25 smithi177 ceph-mon[33046]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T08:16:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:25 smithi177 ceph-mon[33046]: from='osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:16:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:25 smithi177 ceph-mon[33046]: osdmap e59: 9 total, 8 up, 9 in 2024-04-24T08:16:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:25 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:25 smithi177 ceph-mon[33046]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:16:25.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:25 smithi177 ceph-mon[33046]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:26.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:26 smithi146 ceph-mon[27805]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:16:26.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:26 smithi146 ceph-mon[27805]: osdmap e60: 9 total, 8 up, 9 in 2024-04-24T08:16:26.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:26 smithi146 ceph-mon[27805]: osdmap e61: 9 total, 8 up, 9 in 2024-04-24T08:16:26.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:26 smithi146 ceph-mon[27805]: from='osd.8 ' entity='osd.8' 2024-04-24T08:16:26.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-mon[33046]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:16:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-mon[33046]: osdmap e60: 9 total, 8 up, 9 in 2024-04-24T08:16:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-mon[33046]: osdmap e61: 9 total, 8 up, 9 in 2024-04-24T08:16:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-mon[33046]: from='osd.8 ' entity='osd.8' 2024-04-24T08:16:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:16:26.245+0000 7fa575711700 -1 osd.8 0 waiting for initial osdmap 2024-04-24T08:16:26.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:16:26 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:16:26.362+0000 7fa56d4fc700 -1 osd.8 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:16:26.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:26 smithi153 ceph-mon[32746]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:16:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:26 smithi153 ceph-mon[32746]: osdmap e60: 9 total, 8 up, 9 in 2024-04-24T08:16:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:26 smithi153 ceph-mon[32746]: osdmap e61: 9 total, 8 up, 9 in 2024-04-24T08:16:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:26 smithi153 ceph-mon[32746]: from='osd.8 ' entity='osd.8' 2024-04-24T08:16:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:27.412 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T08:16:27.412 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T08:16:27.412 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-04-24T08:16:27.412 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-04-24T08:16:27.412 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0321119 s, 327 MB/s 2024-04-24T08:16:27.412 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-04-24T08:16:27.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:16:27.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:16:27.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:27.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] boot 2024-04-24T08:16:27.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T08:16:27.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:27.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:27.613 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.717 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] boot 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.718 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.719 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:27.719 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:27.719 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:16:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:16:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:16:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: osd.8 [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] boot 2024-04-24T08:16:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T08:16:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:16:27.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:28.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_1 2024-04-24T08:16:28.628 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:28 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi177 2024-04-24T08:16:28.628 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:28 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-24T08:16:28.628 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:28 smithi153 ceph-mon[32746]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T08:16:28.640 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:28 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi177 2024-04-24T08:16:28.640 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:28 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-24T08:16:28.640 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:28 smithi146 ceph-mon[27805]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T08:16:28.748 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:16:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:28 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi177 2024-04-24T08:16:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:28 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-24T08:16:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:28 smithi177 ceph-mon[33046]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T08:16:29.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:29 smithi146 ceph-mon[27805]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:29.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:29 smithi146 ceph-mon[27805]: osdmap e64: 9 total, 9 up, 9 in 2024-04-24T08:16:29.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:29 smithi153 ceph-mon[32746]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:29.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:29 smithi153 ceph-mon[32746]: osdmap e64: 9 total, 9 up, 9 in 2024-04-24T08:16:29.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:29 smithi177 ceph-mon[33046]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:29 smithi177 ceph-mon[33046]: osdmap e64: 9 total, 9 up, 9 in 2024-04-24T08:16:30.689 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:30 smithi146 ceph-mon[27805]: osdmap e65: 9 total, 9 up, 9 in 2024-04-24T08:16:30.689 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:30 smithi146 ceph-mon[27805]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.862+0000 7f976fc00700 1 -- 172.21.15.146:0/2579738533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680774a0 msgr2=0x7f97680778c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.862+0000 7f976fc00700 1 --2- 172.21.15.146:0/2579738533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680774a0 0x7f97680778c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f975c009990 tx=0x7f975c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.862+0000 7f976fc00700 1 -- 172.21.15.146:0/2579738533 shutdown_connections 2024-04-24T08:16:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.862+0000 7f976fc00700 1 --2- 172.21.15.146:0/2579738533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680790b0 0x7f9768079930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.862+0000 7f976fc00700 1 --2- 172.21.15.146:0/2579738533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680774a0 0x7f97680778c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.862+0000 7f976fc00700 1 --2- 172.21.15.146:0/2579738533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97680786f0 0x7f9768078b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.862+0000 7f976fc00700 1 -- 172.21.15.146:0/2579738533 >> 172.21.15.146:0/2579738533 conn(0x7f9768100340 msgr2=0x7f9768102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.863+0000 7f976fc00700 1 -- 172.21.15.146:0/2579738533 shutdown_connections 2024-04-24T08:16:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.863+0000 7f976fc00700 1 -- 172.21.15.146:0/2579738533 wait complete. 2024-04-24T08:16:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 Processor -- start 2024-04-24T08:16:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 -- start start 2024-04-24T08:16:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680774a0 0x7f976819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97680786f0 0x7f976819f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680790b0 0x7f97681a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9768114de0 con 0x7f97680774a0 2024-04-24T08:16:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9768114c60 con 0x7f97680790b0 2024-04-24T08:16:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.864+0000 7f976fc00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9768114f60 con 0x7f97680786f0 2024-04-24T08:16:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680774a0 0x7f976819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f9767e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97680786f0 0x7f976819f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680774a0 0x7f976819f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:42268/0 (socket says 172.21.15.146:42268) 2024-04-24T08:16:30.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 -- 172.21.15.146:0/4069903955 learned_addr learned my addr 172.21.15.146:0/4069903955 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:16:30.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976dc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680790b0 0x7f97681a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 -- 172.21.15.146:0/4069903955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97680786f0 msgr2=0x7f976819f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97680786f0 0x7f976819f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 -- 172.21.15.146:0/4069903955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680790b0 msgr2=0x7f97681a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680790b0 0x7f97681a6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 -- 172.21.15.146:0/4069903955 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97681a6fe0 con 0x7f97680774a0 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976dc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680790b0 0x7f97681a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.865+0000 7f976d200700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680774a0 0x7f976819f380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f975c0095c0 tx=0x7f975c00bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.866+0000 7f9765600700 1 -- 172.21.15.146:0/4069903955 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c026070 con 0x7f97680774a0 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.866+0000 7f9765600700 1 -- 172.21.15.146:0/4069903955 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f975c003d70 con 0x7f97680774a0 2024-04-24T08:16:30.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.866+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97681a7270 con 0x7f97680774a0 2024-04-24T08:16:30.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.866+0000 7f9765600700 1 -- 172.21.15.146:0/4069903955 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c021410 con 0x7f97680774a0 2024-04-24T08:16:30.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.866+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97681a7720 con 0x7f97680774a0 2024-04-24T08:16:30.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.868+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97680697e0 con 0x7f97680774a0 2024-04-24T08:16:30.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.870+0000 7f9765600700 1 -- 172.21.15.146:0/4069903955 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f975c029070 con 0x7f97680774a0 2024-04-24T08:16:30.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.871+0000 7f9765600700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97500a3cc0 0x7f97500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.871+0000 7f9765600700 1 -- 172.21.15.146:0/4069903955 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5829+0+0 (secure 0 0 0) 0x7f975c0d0850 con 0x7f97680774a0 2024-04-24T08:16:30.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.871+0000 7f9767e00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97500a3cc0 0x7f97500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.872+0000 7f9767e00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97500a3cc0 0x7f97500a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9754009c60 tx=0x7f9754009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:30.879+0000 7f9765600700 1 -- 172.21.15.146:0/4069903955 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f975c05dff0 con 0x7f97680774a0 2024-04-24T08:16:30.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:30 smithi153 ceph-mon[32746]: osdmap e65: 9 total, 9 up, 9 in 2024-04-24T08:16:30.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:30 smithi153 ceph-mon[32746]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:31.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:30 smithi177 ceph-mon[33046]: osdmap e65: 9 total, 9 up, 9 in 2024-04-24T08:16:31.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:30 smithi177 ceph-mon[33046]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:31.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:31.117+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9768062f30 con 0x7f97500a3cc0 2024-04-24T08:16:31.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:31 smithi153 ceph-mon[32746]: osdmap e66: 9 total, 9 up, 9 in 2024-04-24T08:16:31.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:31 smithi153 ceph-mon[32746]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:16:31.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:16:31.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:16:31.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:31 smithi146 ceph-mon[27805]: osdmap e66: 9 total, 9 up, 9 in 2024-04-24T08:16:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:31 smithi146 ceph-mon[27805]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:16:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:16:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:16:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:32.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:31 smithi177 ceph-mon[33046]: osdmap e66: 9 total, 9 up, 9 in 2024-04-24T08:16:32.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:31 smithi177 ceph-mon[33046]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:16:32.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:16:32.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:16:32.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:32.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:32 smithi146 ceph-mon[27805]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-24T08:16:32.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:32 smithi153 ceph-mon[32746]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-24T08:16:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:32 smithi177 ceph-mon[33046]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-24T08:16:35.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:35 smithi153 ceph-mon[32746]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:16:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:35 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/296455642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]: dispatch 2024-04-24T08:16:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:35 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]: dispatch 2024-04-24T08:16:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:35 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]': finished 2024-04-24T08:16:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:35 smithi153 ceph-mon[32746]: osdmap e67: 10 total, 9 up, 10 in 2024-04-24T08:16:35.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:35 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:35.731 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:35 smithi146 ceph-mon[27805]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:16:35.731 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:35 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/296455642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]: dispatch 2024-04-24T08:16:35.731 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:35 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]: dispatch 2024-04-24T08:16:35.731 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:35 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]': finished 2024-04-24T08:16:35.731 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:35 smithi146 ceph-mon[27805]: osdmap e67: 10 total, 9 up, 10 in 2024-04-24T08:16:35.731 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:35 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:35 smithi177 ceph-mon[33046]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:16:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:35 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/296455642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]: dispatch 2024-04-24T08:16:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:35 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]: dispatch 2024-04-24T08:16:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:35 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f"}]': finished 2024-04-24T08:16:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:35 smithi177 ceph-mon[33046]: osdmap e67: 10 total, 9 up, 10 in 2024-04-24T08:16:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:35 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:36 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4285934442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:16:36.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:36 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4285934442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:16:36.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:36 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4285934442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:16:37.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:37 smithi177 ceph-mon[33046]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-24T08:16:37.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:37 smithi153 ceph-mon[32746]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-24T08:16:37.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:37 smithi146 ceph-mon[27805]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-24T08:16:39.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:39 smithi153 ceph-mon[32746]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:16:39.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:39 smithi177 ceph-mon[33046]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:16:39.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:39 smithi146 ceph-mon[27805]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:16:41.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:41 smithi146 ceph-mon[27805]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T08:16:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:41 smithi177 ceph-mon[33046]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T08:16:41.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:41 smithi153 ceph-mon[32746]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-24T08:16:43.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:43 smithi146 ceph-mon[27805]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:43 smithi177 ceph-mon[33046]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:43.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:43 smithi153 ceph-mon[32746]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:45.804 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:45 smithi146 ceph-mon[27805]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:45.805 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:45 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T08:16:45.805 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:45 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:45 smithi177 ceph-mon[33046]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:45 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T08:16:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:45 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:45.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:45 smithi153 ceph-mon[32746]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:45.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:45 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T08:16:45.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:45 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:16:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:46 smithi146 ceph-mon[27805]: Deploying daemon osd.9 on smithi146 2024-04-24T08:16:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:46 smithi177 ceph-mon[33046]: Deploying daemon osd.9 on smithi146 2024-04-24T08:16:46.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:46 smithi153 ceph-mon[32746]: Deploying daemon osd.9 on smithi146 2024-04-24T08:16:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:47 smithi177 ceph-mon[33046]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:47.841 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:47 smithi146 ceph-mon[27805]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:47.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:47 smithi153 ceph-mon[32746]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:48.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:48 smithi153 ceph-mon[32746]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:48.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:48 smithi146 ceph-mon[27805]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:49.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:48 smithi177 ceph-mon[33046]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:51.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:51 smithi146 ceph-mon[27805]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:51 smithi177 ceph-mon[33046]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:51.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:51 smithi153 ceph-mon[32746]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:53 smithi146 ceph-mon[27805]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:53 smithi177 ceph-mon[33046]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:53.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:53 smithi153 ceph-mon[32746]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:53.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:53.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:53.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:55.584 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:55 smithi146 ceph-mon[27805]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:55 smithi177 ceph-mon[33046]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:55.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:55 smithi153 ceph-mon[32746]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:57.624 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:57 smithi146 ceph-mon[27805]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:57.624 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:57 smithi146 ceph-mon[27805]: from='osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:16:57.624 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:57 smithi146 ceph-mon[27805]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:16:57.624 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:57.625 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:57.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:57 smithi153 ceph-mon[32746]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:57.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:57 smithi153 ceph-mon[32746]: from='osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:16:57.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:57 smithi153 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:16:57.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:57.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:57 smithi177 ceph-mon[33046]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:57 smithi177 ceph-mon[33046]: from='osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:16:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:57 smithi177 ceph-mon[33046]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:16:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:58.272 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 9 on host 'smithi146' 2024-04-24T08:16:58.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.270+0000 7f9765600700 1 -- 172.21.15.146:0/4069903955 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9768062f30 con 0x7f97500a3cc0 2024-04-24T08:16:58.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.273+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97500a3cc0 msgr2=0x7f97500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.273+0000 7f976fc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97500a3cc0 0x7f97500a6180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9754009c60 tx=0x7f9754009260 comp rx=0 tx=0).stop 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.273+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680774a0 msgr2=0x7f976819f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.273+0000 7f976fc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680774a0 0x7f976819f380 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f975c0095c0 tx=0x7f975c00bb90 comp rx=0 tx=0).stop 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 shutdown_connections 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97680790b0 0x7f97681a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97500a3cc0 0x7f97500a6180 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97680774a0 0x7f976819f380 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 --2- 172.21.15.146:0/4069903955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97680786f0 0x7f976819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 >> 172.21.15.146:0/4069903955 conn(0x7f9768100340 msgr2=0x7f9768101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 shutdown_connections 2024-04-24T08:16:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:16:58.274+0000 7f976fc00700 1 -- 172.21.15.146:0/4069903955 wait complete. 2024-04-24T08:16:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T08:16:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: from='osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:16:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: osdmap e68: 10 total, 9 up, 10 in 2024-04-24T08:16:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:16:58.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:58.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:58.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:58 smithi146 ceph-mon[27805]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: from='osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: osdmap e68: 10 total, 9 up, 10 in 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:58 smithi177 ceph-mon[33046]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:16:58.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T08:16:58.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: from='osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:16:58.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: osdmap e68: 10 total, 9 up, 10 in 2024-04-24T08:16:58.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:58.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-04-24T08:16:58.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:58.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:16:58.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:58 smithi153 ceph-mon[32746]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-04-24T08:16:59.109 DEBUG:teuthology.orchestra.run.smithi146:osd.9> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.9.service 2024-04-24T08:16:59.113 INFO:tasks.cephadm:Deploying osd.10 on smithi153 with /dev/vg_nvme/lv_1... 2024-04-24T08:16:59.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T08:16:59.350 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:16:59.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:59 smithi146 ceph-mon[27805]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:59.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:59 smithi146 ceph-mon[27805]: osdmap e69: 10 total, 9 up, 10 in 2024-04-24T08:16:59.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:59 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:59.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:16:59 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:59.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:16:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:16:59.564+0000 7f5bb0800700 -1 osd.9 0 waiting for initial osdmap 2024-04-24T08:16:59.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:16:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:16:59.601+0000 7f5ba6600700 -1 osd.9 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:16:59.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:59 smithi153 ceph-mon[32746]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:59.786 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:59 smithi153 ceph-mon[32746]: osdmap e69: 10 total, 9 up, 10 in 2024-04-24T08:16:59.786 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:59 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:59.786 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:16:59 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:59.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:59 smithi177 ceph-mon[33046]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:16:59.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:59 smithi177 ceph-mon[33046]: osdmap e69: 10 total, 9 up, 10 in 2024-04-24T08:16:59.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:59 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:16:59.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:16:59 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:00 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:17:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:00 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:17:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:00 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:00 smithi146 ceph-mon[27805]: from='osd.9 ' entity='osd.9' 2024-04-24T08:17:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:00 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:17:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:00 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:17:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:00 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:00 smithi153 ceph-mon[32746]: from='osd.9 ' entity='osd.9' 2024-04-24T08:17:00.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:00 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:17:00.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:00 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:17:00.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:00 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:00.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:00 smithi177 ceph-mon[33046]: from='osd.9 ' entity='osd.9' 2024-04-24T08:17:01.360 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T08:17:01.360 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T08:17:01.360 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-24T08:17:01.361 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-24T08:17:01.361 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01183 s, 886 MB/s 2024-04-24T08:17:01.361 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-24T08:17:01.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:01 smithi177 ceph-mon[33046]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:17:01.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:01 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:01.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:01 smithi177 ceph-mon[33046]: osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] boot 2024-04-24T08:17:01.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:01 smithi177 ceph-mon[33046]: osdmap e70: 10 total, 10 up, 10 in 2024-04-24T08:17:01.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:01 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:01.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:01 smithi153 ceph-mon[32746]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:17:01.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:01 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:01.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:01 smithi153 ceph-mon[32746]: osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] boot 2024-04-24T08:17:01.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:01 smithi153 ceph-mon[32746]: osdmap e70: 10 total, 10 up, 10 in 2024-04-24T08:17:01.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:01 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:01.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-04-24T08:17:01.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:01 smithi146 ceph-mon[27805]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:17:01.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:01 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:01.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:01 smithi146 ceph-mon[27805]: osd.9 [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] boot 2024-04-24T08:17:01.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:01 smithi146 ceph-mon[27805]: osdmap e70: 10 total, 10 up, 10 in 2024-04-24T08:17:01.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:01 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:17:02.210 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:17:02.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:02 smithi153 ceph-mon[32746]: osdmap e71: 10 total, 10 up, 10 in 2024-04-24T08:17:02.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:02 smithi153 ceph-mon[32746]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:02.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:02 smithi146 ceph-mon[27805]: osdmap e71: 10 total, 10 up, 10 in 2024-04-24T08:17:02.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:02 smithi146 ceph-mon[27805]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:03.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:02 smithi177 ceph-mon[33046]: osdmap e71: 10 total, 10 up, 10 in 2024-04-24T08:17:03.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:02 smithi177 ceph-mon[33046]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/3564497871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1810aa50 msgr2=0x7fbd1810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/3564497871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1810aa50 0x7fbd1810cec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000b0a0 tx=0x7fbd1000b3b0 comp rx=0 tx=0).stop 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/3564497871 shutdown_connections 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/3564497871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd1810aa50 0x7fbd1810cec0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/3564497871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd18076aa0 0x7fbd18074f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/3564497871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd180754d0 0x7fbd18075950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/3564497871 >> 172.21.15.153:0/3564497871 conn(0x7fbd180fe0f0 msgr2=0x7fbd18100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/3564497871 shutdown_connections 2024-04-24T08:17:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/3564497871 wait complete. 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 Processor -- start 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.792+0000 7fbd1fb1c700 1 -- start start 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd180754d0 0x7fbd1819aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18076aa0 0x7fbd1819b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd1810aa50 0x7fbd181a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbd18110a40 con 0x7fbd1810aa50 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd181108c0 con 0x7fbd18076aa0 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbd18110bc0 con 0x7fbd180754d0 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18076aa0 0x7fbd1819b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18076aa0 0x7fbd1819b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42550/0 (socket says 172.21.15.153:42550) 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 -- 172.21.15.153:0/2128645351 learned_addr learned my addr 172.21.15.153:0/2128645351 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d8b8700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd180754d0 0x7fbd1819aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:03.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 -- 172.21.15.153:0/2128645351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd180754d0 msgr2=0x7fbd1819aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd180754d0 0x7fbd1819aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 -- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd1810aa50 msgr2=0x7fbd181a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd1810aa50 0x7fbd181a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 -- 172.21.15.153:0/2128645351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd181a2c50 con 0x7fbd18076aa0 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d8b8700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd180754d0 0x7fbd1819aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1d0b7700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18076aa0 0x7fbd1819b530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800b530 tx=0x7fbd0800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd0effd700 1 -- 172.21.15.153:0/2128645351 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd08015020 con 0x7fbd18076aa0 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd0effd700 1 -- 172.21.15.153:0/2128645351 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd08011d10 con 0x7fbd18076aa0 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd181a2f40 con 0x7fbd18076aa0 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd0effd700 1 -- 172.21.15.153:0/2128645351 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0800f2e0 con 0x7fbd18076aa0 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.793+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd181a3470 con 0x7fbd18076aa0 2024-04-24T08:17:03.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.794+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd180697e0 con 0x7fbd18076aa0 2024-04-24T08:17:03.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.795+0000 7fbd0effd700 1 -- 172.21.15.153:0/2128645351 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fbd08018020 con 0x7fbd18076aa0 2024-04-24T08:17:03.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.796+0000 7fbd0effd700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbd040a3c40 0x7fbd040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:03.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.796+0000 7fbd0effd700 1 -- 172.21.15.153:0/2128645351 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6261+0+0 (secure 0 0 0) 0x7fbd080cf890 con 0x7fbd18076aa0 2024-04-24T08:17:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.796+0000 7fbd1d8b8700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbd040a3c40 0x7fbd040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:03.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.797+0000 7fbd1d8b8700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbd040a3c40 0x7fbd040a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd14007900 tx=0x7fbd14008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:03.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:03.800+0000 7fbd0effd700 1 -- 172.21.15.153:0/2128645351 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd0801f020 con 0x7fbd18076aa0 2024-04-24T08:17:04.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:04.009+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbd18062f30 con 0x7fbd040a3c40 2024-04-24T08:17:04.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:17:04.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:17:04.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:04.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:17:04.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:17:04.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:04.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:17:04.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:17:04.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:05.270 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi146 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi146 to 4272M 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:05.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:05.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:05.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:05.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:05 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:17:05.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi146 2024-04-24T08:17:05.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi146 to 4272M 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:05.494 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:05 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:17:05.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi146 2024-04-24T08:17:05.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi146 to 4272M 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:05.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:05.567 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:05 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:07 smithi177 ceph-mon[33046]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:07 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/330214557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]: dispatch 2024-04-24T08:17:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:07 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]: dispatch 2024-04-24T08:17:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:07 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]': finished 2024-04-24T08:17:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:07 smithi177 ceph-mon[33046]: osdmap e72: 11 total, 10 up, 11 in 2024-04-24T08:17:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:07 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:07 smithi177 ceph-mon[33046]: from='client.? 172.21.15.153:0/901347232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:17:07.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:07 smithi153 ceph-mon[32746]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:07 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/330214557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]: dispatch 2024-04-24T08:17:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:07 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]: dispatch 2024-04-24T08:17:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:07 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]': finished 2024-04-24T08:17:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:07 smithi153 ceph-mon[32746]: osdmap e72: 11 total, 10 up, 11 in 2024-04-24T08:17:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:07 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:07 smithi153 ceph-mon[32746]: from='client.? 172.21.15.153:0/901347232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:17:07.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:07 smithi146 ceph-mon[27805]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:07.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:07 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/330214557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]: dispatch 2024-04-24T08:17:07.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:07 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]: dispatch 2024-04-24T08:17:07.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:07 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7cb8a28-80cc-473d-91d1-dda23d2790af"}]': finished 2024-04-24T08:17:07.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:07 smithi146 ceph-mon[27805]: osdmap e72: 11 total, 10 up, 11 in 2024-04-24T08:17:07.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:07 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:07.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:07 smithi146 ceph-mon[27805]: from='client.? 172.21.15.153:0/901347232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:17:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:09 smithi177 ceph-mon[33046]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:09.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:09 smithi153 ceph-mon[32746]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:09.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:09 smithi146 ceph-mon[27805]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:11.795 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:11 smithi146 ceph-mon[27805]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:11.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:11 smithi177 ceph-mon[33046]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:11.867 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:11 smithi153 ceph-mon[32746]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:13.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:13 smithi177 ceph-mon[33046]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:13.908 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:13 smithi153 ceph-mon[32746]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:13.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:13 smithi146 ceph-mon[27805]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:14.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T08:17:14.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T08:17:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T08:17:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:15.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:15 smithi153 ceph-mon[32746]: Deploying daemon osd.10 on smithi153 2024-04-24T08:17:15.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:15 smithi153 ceph-mon[32746]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:15 smithi177 ceph-mon[33046]: Deploying daemon osd.10 on smithi153 2024-04-24T08:17:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:15 smithi177 ceph-mon[33046]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:15.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:15 smithi146 ceph-mon[27805]: Deploying daemon osd.10 on smithi153 2024-04-24T08:17:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:15 smithi146 ceph-mon[27805]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:17.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:17 smithi153 ceph-mon[32746]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:17 smithi177 ceph-mon[33046]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:17.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:17 smithi146 ceph-mon[27805]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:19.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:19 smithi153 ceph-mon[32746]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:19.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:19 smithi177 ceph-mon[33046]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:19.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:19 smithi146 ceph-mon[27805]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:21.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:21.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:21.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:21.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:21 smithi153 ceph-mon[32746]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:21.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:21.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:21.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:21.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:21 smithi146 ceph-mon[27805]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:21 smithi177 ceph-mon[33046]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:23.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:23 smithi153 ceph-mon[32746]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:23 smithi146 ceph-mon[27805]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:23.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:23 smithi177 ceph-mon[33046]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:24.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.144+0000 7fbd0effd700 1 -- 172.21.15.153:0/2128645351 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fbd18062f30 con 0x7fbd040a3c40 2024-04-24T08:17:24.145 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 10 on host 'smithi153' 2024-04-24T08:17:24.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbd040a3c40 msgr2=0x7fbd040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:24.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbd040a3c40 0x7fbd040a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd14007900 tx=0x7fbd14008040 comp rx=0 tx=0).stop 2024-04-24T08:17:24.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18076aa0 msgr2=0x7fbd1819b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:24.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18076aa0 0x7fbd1819b530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800b530 tx=0x7fbd0800b8f0 comp rx=0 tx=0).stop 2024-04-24T08:17:24.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 shutdown_connections 2024-04-24T08:17:24.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbd040a3c40 0x7fbd040a6100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:24.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd1810aa50 0x7fbd181a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:24.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd180754d0 0x7fbd1819aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:24.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 --2- 172.21.15.153:0/2128645351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd18076aa0 0x7fbd1819b530 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:24.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.146+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 >> 172.21.15.153:0/2128645351 conn(0x7fbd180fe0f0 msgr2=0x7fbd181069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:24.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.147+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 shutdown_connections 2024-04-24T08:17:24.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-24T08:17:24.147+0000 7fbd1fb1c700 1 -- 172.21.15.153:0/2128645351 wait complete. 2024-04-24T08:17:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:24 smithi146 ceph-mon[27805]: from='osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:17:24.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:24 smithi146 ceph-mon[27805]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:17:24.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:24 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:24.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:24 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:24.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:24 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:24 smithi177 ceph-mon[33046]: from='osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:17:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:24 smithi177 ceph-mon[33046]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:17:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:24 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:24 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:24 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:24 smithi153 ceph-mon[32746]: from='osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:17:24.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:24 smithi153 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:17:24.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:24 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:24.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:24 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:24.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:24 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:25.011 DEBUG:teuthology.orchestra.run.smithi153:osd.10> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.10.service 2024-04-24T08:17:25.013 INFO:tasks.cephadm:Deploying osd.11 on smithi177 with /dev/vg_nvme/lv_1... 2024-04-24T08:17:25.013 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T08:17:25.259 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:17:25.633 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:25 smithi177 ceph-mon[33046]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:25.633 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:25 smithi177 ceph-mon[33046]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T08:17:25.633 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:25 smithi177 ceph-mon[33046]: from='osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:17:25.633 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:25 smithi177 ceph-mon[33046]: osdmap e73: 11 total, 10 up, 11 in 2024-04-24T08:17:25.633 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:25 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:25.633 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:25 smithi177 ceph-mon[33046]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:17:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-mon[32746]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T08:17:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-mon[32746]: from='osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:17:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-mon[32746]: osdmap e73: 11 total, 10 up, 11 in 2024-04-24T08:17:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:25.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:17:25.711 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:17:25.495+0000 7ff6493ad700 -1 osd.10 0 waiting for initial osdmap 2024-04-24T08:17:25.711 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:17:25 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:17:25.509+0000 7ff6451a0700 -1 osd.10 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:17:25.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:25 smithi146 ceph-mon[27805]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:25.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:25 smithi146 ceph-mon[27805]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T08:17:25.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:25 smithi146 ceph-mon[27805]: from='osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:17:25.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:25 smithi146 ceph-mon[27805]: osdmap e73: 11 total, 10 up, 11 in 2024-04-24T08:17:25.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:25 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:25.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:25 smithi146 ceph-mon[27805]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-24T08:17:26.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:26 smithi153 ceph-mon[32746]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:17:26.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:26 smithi153 ceph-mon[32746]: osdmap e74: 11 total, 10 up, 11 in 2024-04-24T08:17:26.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:26.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:26 smithi153 ceph-mon[32746]: osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] boot 2024-04-24T08:17:26.535 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:26 smithi153 ceph-mon[32746]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T08:17:26.536 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:26.619 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:26 smithi177 ceph-mon[33046]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:17:26.619 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:26 smithi177 ceph-mon[33046]: osdmap e74: 11 total, 10 up, 11 in 2024-04-24T08:17:26.619 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:26.619 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:26 smithi177 ceph-mon[33046]: osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] boot 2024-04-24T08:17:26.619 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:26 smithi177 ceph-mon[33046]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T08:17:26.619 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:26.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:26 smithi146 ceph-mon[27805]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-24T08:17:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:26 smithi146 ceph-mon[27805]: osdmap e74: 11 total, 10 up, 11 in 2024-04-24T08:17:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:26 smithi146 ceph-mon[27805]: osd.10 [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] boot 2024-04-24T08:17:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:26 smithi146 ceph-mon[27805]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T08:17:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:17:27.224 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T08:17:27.224 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T08:17:27.224 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-24T08:17:27.224 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-24T08:17:27.224 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110498 s, 949 MB/s 2024-04-24T08:17:27.224 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-24T08:17:27.606 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:27 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:17:27.606 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:27 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:17:27.606 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:27 smithi177 ceph-mon[33046]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:27.606 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:27 smithi177 ceph-mon[33046]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T08:17:27.900 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-04-24T08:17:27.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:27 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:17:27.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:27 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:17:27.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:27 smithi153 ceph-mon[32746]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:27.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:27 smithi153 ceph-mon[32746]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T08:17:27.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:27 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:17:27.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:27 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:17:27.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:27 smithi146 ceph-mon[27805]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:17:27.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:27 smithi146 ceph-mon[27805]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T08:17:28.140 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:17:28.954 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: osdmap e77: 11 total, 11 up, 11 in 2024-04-24T08:17:28.954 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi153 2024-04-24T08:17:28.954 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:28 smithi177 ceph-mon[33046]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:28.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: osdmap e77: 11 total, 11 up, 11 in 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi153 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:28 smithi153 ceph-mon[32746]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: osdmap e77: 11 total, 11 up, 11 in 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi153 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi153 to 4272M 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:28.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:28.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:28.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:28 smithi146 ceph-mon[27805]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 -- 172.21.15.177:0/1570938237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d081035f0 msgr2=0x7f6d08109ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1570938237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d081035f0 0x7f6d08109ea0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000b0a0 tx=0x7f6d0000b3b0 comp rx=0 tx=0).stop 2024-04-24T08:17:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 -- 172.21.15.177:0/1570938237 shutdown_connections 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1570938237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d081035f0 0x7f6d08109ea0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1570938237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d08101a50 0x7f6d08101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1570938237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d08102c50 0x7f6d081030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 -- 172.21.15.177:0/1570938237 >> 172.21.15.177:0/1570938237 conn(0x7f6d080fcff0 msgr2=0x7f6d080ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 -- 172.21.15.177:0/1570938237 shutdown_connections 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.743+0000 7f6d0e254700 1 -- 172.21.15.177:0/1570938237 wait complete. 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 Processor -- start 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 -- start start 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d08101a50 0x7f6d0810fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d08102c50 0x7f6d0810e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d081035f0 0x7f6d0810e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810d6b0 con 0x7f6d08101a50 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810d530 con 0x7f6d081035f0 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d0e254700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0810d830 con 0x7f6d08102c50 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d08102c50 0x7f6d0810e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d08102c50 0x7f6d0810e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:53308/0 (socket says 172.21.15.177:53308) 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 -- 172.21.15.177:0/1301587880 learned_addr learned my addr 172.21.15.177:0/1301587880 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d07fff700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d081035f0 0x7f6d0810e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 -- 172.21.15.177:0/1301587880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d081035f0 msgr2=0x7f6d0810e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d081035f0 0x7f6d0810e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 -- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d08101a50 msgr2=0x7f6d0810fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d08101a50 0x7f6d0810fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 -- 172.21.15.177:0/1301587880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0810f030 con 0x7f6d08102c50 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6cfffff700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d08102c50 0x7f6d0810e240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000d950 tx=0x7f6cf000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.744+0000 7f6d057fa700 1 -- 172.21.15.177:0/1301587880 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0016070 con 0x7f6d08102c50 2024-04-24T08:17:29.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.745+0000 7f6d057fa700 1 -- 172.21.15.177:0/1301587880 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf0009e20 con 0x7f6d08102c50 2024-04-24T08:17:29.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.745+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d081a2f50 con 0x7f6d08102c50 2024-04-24T08:17:29.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.745+0000 7f6d057fa700 1 -- 172.21.15.177:0/1301587880 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000f620 con 0x7f6d08102c50 2024-04-24T08:17:29.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.745+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d081a3480 con 0x7f6d08102c50 2024-04-24T08:17:29.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.746+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d08069810 con 0x7f6d08102c50 2024-04-24T08:17:29.749 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.747+0000 7f6d057fa700 1 -- 172.21.15.177:0/1301587880 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6cf0019070 con 0x7f6d08102c50 2024-04-24T08:17:29.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.747+0000 7f6d057fa700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6cf40a3cc0 0x7f6cf40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:29.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.747+0000 7f6d057fa700 1 -- 172.21.15.177:0/1301587880 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6694+0+0 (secure 0 0 0) 0x7f6cf00cf8b0 con 0x7f6d08102c50 2024-04-24T08:17:29.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.748+0000 7f6d077fe700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6cf40a3cc0 0x7f6cf40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:29.750 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.748+0000 7f6d077fe700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6cf40a3cc0 0x7f6cf40a6180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6d08102ab0 tx=0x7f6cf8006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:29.752 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.752+0000 7f6d057fa700 1 -- 172.21.15.177:0/1301587880 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d08069810 con 0x7f6d08102c50 2024-04-24T08:17:29.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:29 smithi177 ceph-mon[33046]: osdmap e78: 11 total, 11 up, 11 in 2024-04-24T08:17:29.958 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:29.957+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6d080fdd50 con 0x7f6cf40a3cc0 2024-04-24T08:17:29.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:29 smithi153 ceph-mon[32746]: osdmap e78: 11 total, 11 up, 11 in 2024-04-24T08:17:29.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:29 smithi146 ceph-mon[27805]: osdmap e78: 11 total, 11 up, 11 in 2024-04-24T08:17:30.868 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:30 smithi177 ceph-mon[33046]: osdmap e79: 11 total, 11 up, 11 in 2024-04-24T08:17:30.868 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:30 smithi177 ceph-mon[33046]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:17:30.868 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:30 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:17:30.869 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:30 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:17:30.869 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:30 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:30.869 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:30 smithi177 ceph-mon[33046]: pgmap v266: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:30.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:30 smithi153 ceph-mon[32746]: osdmap e79: 11 total, 11 up, 11 in 2024-04-24T08:17:30.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:30 smithi153 ceph-mon[32746]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:17:30.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:30 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:17:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:30 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:17:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:30 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:30 smithi153 ceph-mon[32746]: pgmap v266: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:30.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:30 smithi146 ceph-mon[27805]: osdmap e79: 11 total, 11 up, 11 in 2024-04-24T08:17:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:30 smithi146 ceph-mon[27805]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:17:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:30 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:17:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:30 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:17:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:30 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:30 smithi146 ceph-mon[27805]: pgmap v266: 1 pgs: 1 remapped+peering; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:33 smithi177 ceph-mon[33046]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-24T08:17:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:33 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/2285693928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]: dispatch 2024-04-24T08:17:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:33 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]: dispatch 2024-04-24T08:17:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:33 smithi177 ceph-mon[33046]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]': finished 2024-04-24T08:17:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:33 smithi177 ceph-mon[33046]: osdmap e80: 12 total, 11 up, 12 in 2024-04-24T08:17:33.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:33 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:33.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:33 smithi177 ceph-mon[33046]: from='client.? 172.21.15.177:0/3054434400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:17:33.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:33 smithi153 ceph-mon[32746]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-24T08:17:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:33 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/2285693928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]: dispatch 2024-04-24T08:17:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:33 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]: dispatch 2024-04-24T08:17:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:33 smithi153 ceph-mon[32746]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]': finished 2024-04-24T08:17:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:33 smithi153 ceph-mon[32746]: osdmap e80: 12 total, 11 up, 12 in 2024-04-24T08:17:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:33 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:33 smithi153 ceph-mon[32746]: from='client.? 172.21.15.177:0/3054434400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:17:33.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:33 smithi146 ceph-mon[27805]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-24T08:17:33.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:33 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/2285693928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]: dispatch 2024-04-24T08:17:33.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:33 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]: dispatch 2024-04-24T08:17:33.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:33 smithi146 ceph-mon[27805]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a48d77c-4350-46bf-8fdf-49827d3957c4"}]': finished 2024-04-24T08:17:33.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:33 smithi146 ceph-mon[27805]: osdmap e80: 12 total, 11 up, 12 in 2024-04-24T08:17:33.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:33 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:33.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:33 smithi146 ceph-mon[27805]: from='client.? 172.21.15.177:0/3054434400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:17:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:35 smithi177 ceph-mon[33046]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:17:35.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:35 smithi153 ceph-mon[32746]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:17:35.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:35 smithi146 ceph-mon[27805]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:17:37.786 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:37 smithi177 ceph-mon[33046]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-24T08:17:37.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:37 smithi153 ceph-mon[32746]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-24T08:17:37.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:37 smithi146 ceph-mon[27805]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-24T08:17:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:39 smithi177 ceph-mon[33046]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T08:17:39.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:39 smithi153 ceph-mon[32746]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T08:17:39.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:39 smithi146 ceph-mon[27805]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-24T08:17:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T08:17:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:40.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T08:17:40.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:40.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T08:17:40.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:41.647 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:41 smithi177 ceph-mon[33046]: Deploying daemon osd.11 on smithi177 2024-04-24T08:17:41.647 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:41 smithi177 ceph-mon[33046]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:17:41.894 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:41 smithi146 ceph-mon[27805]: Deploying daemon osd.11 on smithi177 2024-04-24T08:17:41.894 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:41 smithi146 ceph-mon[27805]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:17:41.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:41 smithi153 ceph-mon[32746]: Deploying daemon osd.11 on smithi177 2024-04-24T08:17:41.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:41 smithi153 ceph-mon[32746]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:17:43.799 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:43 smithi177 ceph-mon[33046]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:43.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:43 smithi153 ceph-mon[32746]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:43.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:43 smithi146 ceph-mon[27805]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:45 smithi177 ceph-mon[33046]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:45.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:45 smithi153 ceph-mon[32746]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:45.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:45 smithi146 ceph-mon[27805]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:46.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:46.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:46.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:46.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:46 smithi153 ceph-mon[32746]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:46.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:46.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:46.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:46.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:46 smithi146 ceph-mon[27805]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:47.036 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:47.036 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:47.036 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:47.036 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:46 smithi177 ceph-mon[33046]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:49.396 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 11 on host 'smithi177' 2024-04-24T08:17:49.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.394+0000 7f6d057fa700 1 -- 172.21.15.177:0/1301587880 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6d080fdd50 con 0x7f6cf40a3cc0 2024-04-24T08:17:49.398 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6cf40a3cc0 msgr2=0x7f6cf40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.398 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6cf40a3cc0 0x7f6cf40a6180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6d08102ab0 tx=0x7f6cf8006d20 comp rx=0 tx=0).stop 2024-04-24T08:17:49.398 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d08102c50 msgr2=0x7f6d0810e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.398 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d08102c50 0x7f6d0810e240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000d950 tx=0x7f6cf000dc60 comp rx=0 tx=0).stop 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 shutdown_connections 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d081035f0 0x7f6d0810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6cf40a3cc0 0x7f6cf40a6180 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d08101a50 0x7f6d0810fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 --2- 172.21.15.177:0/1301587880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d08102c50 0x7f6d0810e240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 >> 172.21.15.177:0/1301587880 conn(0x7f6d080fcff0 msgr2=0x7f6d080fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 shutdown_connections 2024-04-24T08:17:49.399 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:17:49.397+0000 7f6d0e254700 1 -- 172.21.15.177:0/1301587880 wait complete. 2024-04-24T08:17:49.733 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:49.733 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.734 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.734 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:49.734 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:49.734 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.734 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.734 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.734 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:49 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:49 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:49 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:17:50.055 DEBUG:teuthology.orchestra.run.smithi177:osd.11> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.11.service 2024-04-24T08:17:50.057 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-24T08:17:50.058 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-24T08:17:50.495 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:17:50.810 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:50 smithi146 ceph-mon[27805]: from='osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:17:50.810 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:50 smithi146 ceph-mon[27805]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:17:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:50 smithi177 ceph-mon[33046]: from='osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:17:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:50 smithi177 ceph-mon[33046]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:17:50.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:50 smithi153 ceph-mon[32746]: from='osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:17:50.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:50 smithi153 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:17:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-mon[33046]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-mon[33046]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T08:17:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-mon[33046]: from='osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:17:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-mon[33046]: osdmap e81: 12 total, 11 up, 12 in 2024-04-24T08:17:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:51.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-mon[33046]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:17:51.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:17:51.524+0000 7f91485ac700 -1 osd.11 0 waiting for initial osdmap 2024-04-24T08:17:51.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:17:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:17:51.543+0000 7f914439f700 -1 osd.11 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:17:51.912 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:51 smithi146 ceph-mon[27805]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:51.912 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:51 smithi146 ceph-mon[27805]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T08:17:51.912 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:51 smithi146 ceph-mon[27805]: from='osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:17:51.913 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:51 smithi146 ceph-mon[27805]: osdmap e81: 12 total, 11 up, 12 in 2024-04-24T08:17:51.913 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:51 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:51.913 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:51 smithi146 ceph-mon[27805]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:17:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:51 smithi153 ceph-mon[32746]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:51 smithi153 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T08:17:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:51 smithi153 ceph-mon[32746]: from='osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:17:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:51 smithi153 ceph-mon[32746]: osdmap e81: 12 total, 11 up, 12 in 2024-04-24T08:17:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:51 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:51 smithi153 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-24T08:17:52.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:52 smithi146 ceph-mon[27805]: purged_snaps scrub starts 2024-04-24T08:17:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:52 smithi146 ceph-mon[27805]: purged_snaps scrub ok 2024-04-24T08:17:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:52 smithi146 ceph-mon[27805]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:17:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:52 smithi146 ceph-mon[27805]: osdmap e82: 12 total, 11 up, 12 in 2024-04-24T08:17:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:52 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:52 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:52.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.760+0000 7f5327800700 1 -- 172.21.15.146:0/3715249107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 msgr2=0x7f53201041e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.760+0000 7f5327800700 1 --2- 172.21.15.146:0/3715249107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 0x7f53201041e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f530c009990 tx=0x7f530c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.761+0000 7f5327800700 1 -- 172.21.15.146:0/3715249107 shutdown_connections 2024-04-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.761+0000 7f5327800700 1 --2- 172.21.15.146:0/3715249107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5320104720 0x7f532010afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.761+0000 7f5327800700 1 --2- 172.21.15.146:0/3715249107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5320102b80 0x7f5320102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.761+0000 7f5327800700 1 --2- 172.21.15.146:0/3715249107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 0x7f53201041e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.761+0000 7f5327800700 1 -- 172.21.15.146:0/3715249107 >> 172.21.15.146:0/3715249107 conn(0x7f53200fe140 msgr2=0x7f5320100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.761+0000 7f5327800700 1 -- 172.21.15.146:0/3715249107 shutdown_connections 2024-04-24T08:17:52.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.761+0000 7f5327800700 1 -- 172.21.15.146:0/3715249107 wait complete. 2024-04-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.762+0000 7f5327800700 1 Processor -- start 2024-04-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.762+0000 7f5327800700 1 -- start start 2024-04-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5327800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5320102b80 0x7f5320076890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5327800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 0x7f5320074ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5327800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5320104720 0x7f5320075420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5327800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f532010e790 con 0x7f5320102b80 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5327800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f532010e610 con 0x7f5320104720 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5327800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f532010e910 con 0x7f5320103d80 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5324e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5320102b80 0x7f5320076890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f531fe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 0x7f5320074ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5324e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5320102b80 0x7f5320076890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:42726/0 (socket says 172.21.15.146:42726) 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.763+0000 7f5324e00700 1 -- 172.21.15.146:0/4238040210 learned_addr learned my addr 172.21.15.146:0/4238040210 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:17:52.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f5325800700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5320104720 0x7f5320075420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f531fe00700 1 -- 172.21.15.146:0/4238040210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5320104720 msgr2=0x7f5320075420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f531fe00700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5320104720 0x7f5320075420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f531fe00700 1 -- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5320102b80 msgr2=0x7f5320076890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f531fe00700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5320102b80 0x7f5320076890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f531fe00700 1 -- 172.21.15.146:0/4238040210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5320075cd0 con 0x7f5320103d80 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f531fe00700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 0x7f5320074ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f530c00b950 tx=0x7f530c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f531d600700 1 -- 172.21.15.146:0/4238040210 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c026070 con 0x7f5320103d80 2024-04-24T08:17:52.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.764+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53201a2eb0 con 0x7f5320103d80 2024-04-24T08:17:52.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.765+0000 7f531d600700 1 -- 172.21.15.146:0/4238040210 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530c0218f0 con 0x7f5320103d80 2024-04-24T08:17:52.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.765+0000 7f531d600700 1 -- 172.21.15.146:0/4238040210 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c00fe60 con 0x7f5320103d80 2024-04-24T08:17:52.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.765+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53201a32a0 con 0x7f5320103d80 2024-04-24T08:17:52.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.766+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53200697e0 con 0x7f5320103d80 2024-04-24T08:17:52.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.768+0000 7f531d600700 1 -- 172.21.15.146:0/4238040210 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f530c029070 con 0x7f5320103d80 2024-04-24T08:17:52.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.769+0000 7f531d600700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53080a3cc0 0x7f53080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.769+0000 7f531d600700 1 -- 172.21.15.146:0/4238040210 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f530c0d0d50 con 0x7f5320103d80 2024-04-24T08:17:52.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.769+0000 7f5324e00700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53080a3cc0 0x7f53080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.770+0000 7f5324e00700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53080a3cc0 0x7f53080a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5314005950 tx=0x7f5314009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:52.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:52.773+0000 7f531d600700 1 -- 172.21.15.146:0/4238040210 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f530c013070 con 0x7f5320103d80 2024-04-24T08:17:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:52 smithi177 ceph-mon[33046]: purged_snaps scrub starts 2024-04-24T08:17:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:52 smithi177 ceph-mon[33046]: purged_snaps scrub ok 2024-04-24T08:17:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:52 smithi177 ceph-mon[33046]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:17:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:52 smithi177 ceph-mon[33046]: osdmap e82: 12 total, 11 up, 12 in 2024-04-24T08:17:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:52 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:52 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:52.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:52 smithi153 ceph-mon[32746]: purged_snaps scrub starts 2024-04-24T08:17:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:52 smithi153 ceph-mon[32746]: purged_snaps scrub ok 2024-04-24T08:17:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:52 smithi153 ceph-mon[32746]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-24T08:17:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:52 smithi153 ceph-mon[32746]: osdmap e82: 12 total, 11 up, 12 in 2024-04-24T08:17:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:52 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:52 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:53.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.001+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5320063730 con 0x7f5320103d80 2024-04-24T08:17:53.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.002+0000 7f531d600700 1 -- 172.21.15.146:0/4238040210 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7f530c09dbc0 con 0x7f5320103d80 2024-04-24T08:17:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:17:53.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53080a3cc0 msgr2=0x7f53080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53080a3cc0 0x7f53080a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5314005950 tx=0x7f5314009260 comp rx=0 tx=0).stop 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 msgr2=0x7f5320074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 0x7f5320074ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f530c00b950 tx=0x7f530c00f690 comp rx=0 tx=0).stop 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 shutdown_connections 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5320104720 0x7f5320075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53080a3cc0 0x7f53080a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5320102b80 0x7f5320076890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 --2- 172.21.15.146:0/4238040210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5320103d80 0x7f5320074ee0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 >> 172.21.15.146:0/4238040210 conn(0x7f53200fe140 msgr2=0x7f5320107080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 shutdown_connections 2024-04-24T08:17:53.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:53.006+0000 7f5327800700 1 -- 172.21.15.146:0/4238040210 wait complete. 2024-04-24T08:17:53.687 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] boot 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4238040210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:53.688 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:53 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.735 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1713946672,"num_in_osds":12,"osd_in_since":1713946652,"num_remapped_pgs":0} 2024-04-24T08:17:53.735 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T08:17:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] boot 2024-04-24T08:17:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T08:17:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4238040210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:53 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:53.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: osd.11 [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] boot 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4238040210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:54.027 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:54.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:53 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:17:54.150 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:17:54.624 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:54 smithi146 ceph-mon[27805]: Detected new or changed devices on smithi177 2024-04-24T08:17:54.624 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:54 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-24T08:17:54.624 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:54 smithi146 ceph-mon[27805]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T08:17:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:54 smithi177 ceph-mon[33046]: Detected new or changed devices on smithi177 2024-04-24T08:17:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:54 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-24T08:17:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:54 smithi177 ceph-mon[33046]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T08:17:54.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:54 smithi153 ceph-mon[32746]: Detected new or changed devices on smithi177 2024-04-24T08:17:54.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:54 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-24T08:17:54.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:54 smithi153 ceph-mon[32746]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T08:17:55.728 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:55 smithi146 ceph-mon[27805]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:55 smithi177 ceph-mon[33046]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:55.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:55 smithi153 ceph-mon[32746]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.213+0000 7f21b1000700 1 -- 172.21.15.146:0/3445410386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac105810 msgr2=0x7f21ac109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.213+0000 7f21b1000700 1 --2- 172.21.15.146:0/3445410386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac105810 0x7f21ac109860 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f21980099c0 tx=0x7f2198009cd0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.214+0000 7f21b1000700 1 -- 172.21.15.146:0/3445410386 shutdown_connections 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.214+0000 7f21b1000700 1 --2- 172.21.15.146:0/3445410386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac109f40 0x7f21ac111b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.214+0000 7f21b1000700 1 --2- 172.21.15.146:0/3445410386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21ac104e60 0x7f21ac105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.214+0000 7f21b1000700 1 --2- 172.21.15.146:0/3445410386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac105810 0x7f21ac109860 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.214+0000 7f21b1000700 1 -- 172.21.15.146:0/3445410386 >> 172.21.15.146:0/3445410386 conn(0x7f21ac100730 msgr2=0x7f21ac102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.214+0000 7f21b1000700 1 -- 172.21.15.146:0/3445410386 shutdown_connections 2024-04-24T08:17:56.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.214+0000 7f21b1000700 1 -- 172.21.15.146:0/3445410386 wait complete. 2024-04-24T08:17:56.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.215+0000 7f21b1000700 1 Processor -- start 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.215+0000 7f21b1000700 1 -- start start 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21b1000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21ac104e60 0x7f21ac19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21b1000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac105810 0x7f21ac19b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21b1000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac109f40 0x7f21ac19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21b1000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac114f70 con 0x7f21ac109f40 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21b1000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac114df0 con 0x7f21ac104e60 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21b1000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac1150f0 con 0x7f21ac105810 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21abe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac109f40 0x7f21ac19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21aaa00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac105810 0x7f21ac19b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21aaa00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac105810 0x7f21ac19b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40686/0 (socket says 172.21.15.146:40686) 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21aaa00700 1 -- 172.21.15.146:0/540286658 learned_addr learned my addr 172.21.15.146:0/540286658 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:17:56.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.216+0000 7f21ab400700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21ac104e60 0x7f21ac19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21aaa00700 1 -- 172.21.15.146:0/540286658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21ac104e60 msgr2=0x7f21ac19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21aaa00700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21ac104e60 0x7f21ac19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21aaa00700 1 -- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac109f40 msgr2=0x7f21ac19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21aaa00700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac109f40 0x7f21ac19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21aaa00700 1 -- 172.21.15.146:0/540286658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac1a0390 con 0x7f21ac105810 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21abe00700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac109f40 0x7f21ac19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21ab400700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21ac104e60 0x7f21ac19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21aaa00700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac105810 0x7f21ac19b960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2198003e00 tx=0x7f2198003e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21a3e00700 1 -- 172.21.15.146:0/540286658 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198009e60 con 0x7f21ac105810 2024-04-24T08:17:56.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21a3e00700 1 -- 172.21.15.146:0/540286658 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2198003680 con 0x7f21ac105810 2024-04-24T08:17:56.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21a3e00700 1 -- 172.21.15.146:0/540286658 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219800fd40 con 0x7f21ac105810 2024-04-24T08:17:56.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.217+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac1a0620 con 0x7f21ac105810 2024-04-24T08:17:56.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.218+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ac1a7f00 con 0x7f21ac105810 2024-04-24T08:17:56.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.219+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ac0697e0 con 0x7f21ac105810 2024-04-24T08:17:56.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.221+0000 7f21a3e00700 1 -- 172.21.15.146:0/540286658 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f219800f460 con 0x7f21ac105810 2024-04-24T08:17:56.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.222+0000 7f21a3e00700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f21900a3cc0 0x7f21900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.222+0000 7f21a3e00700 1 -- 172.21.15.146:0/540286658 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f21980d0cf0 con 0x7f21ac105810 2024-04-24T08:17:56.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.222+0000 7f21ab400700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f21900a3cc0 0x7f21900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.223+0000 7f21ab400700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f21900a3cc0 0x7f21900a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f219c007fd0 tx=0x7f219c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.225+0000 7f21a3e00700 1 -- 172.21.15.146:0/540286658 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2198013070 con 0x7f21ac105810 2024-04-24T08:17:56.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.460+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21ac19c6e0 con 0x7f21ac105810 2024-04-24T08:17:56.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.462+0000 7f21a3e00700 1 -- 172.21.15.146:0/540286658 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18499 (secure 0 0 0) 0x7f219809db60 con 0x7f21ac105810 2024-04-24T08:17:56.465 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:17:56.465 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":84,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:17:54.027821+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:17:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f21900a3cc0 msgr2=0x7f21900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f21900a3cc0 0x7f21900a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f219c007fd0 tx=0x7f219c0073d0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac105810 msgr2=0x7f21ac19b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac105810 0x7f21ac19b960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2198003e00 tx=0x7f2198003e30 comp rx=0 tx=0).stop 2024-04-24T08:17:56.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 shutdown_connections 2024-04-24T08:17:56.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f21900a3cc0 0x7f21900a6180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21ac109f40 0x7f21ac19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21ac104e60 0x7f21ac19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 --2- 172.21.15.146:0/540286658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21ac105810 0x7f21ac19b960 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 >> 172.21.15.146:0/540286658 conn(0x7f21ac100730 msgr2=0x7f21ac102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:56.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.465+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 shutdown_connections 2024-04-24T08:17:56.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:56.466+0000 7f21b1000700 1 -- 172.21.15.146:0/540286658 wait complete. 2024-04-24T08:17:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/540286658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:17:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/540286658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:17:56.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/540286658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:17:57.191 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:17:57.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-24T08:17:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:17:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:57 smithi177 ceph-mon[33046]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:57 smithi153 ceph-mon[32746]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:57.914 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:57 smithi146 ceph-mon[27805]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:59.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 -- 172.21.15.146:0/1114258464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c10a050 msgr2=0x7ff99c111c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 --2- 172.21.15.146:0/1114258464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c10a050 0x7ff99c111c20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff99400b0a0 tx=0x7ff99400b3b0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 -- 172.21.15.146:0/1114258464 shutdown_connections 2024-04-24T08:17:59.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 --2- 172.21.15.146:0/1114258464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c10a050 0x7ff99c111c20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 --2- 172.21.15.146:0/1114258464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff99c104f70 0x7ff99c105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 --2- 172.21.15.146:0/1114258464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c105920 0x7ff99c109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 -- 172.21.15.146:0/1114258464 >> 172.21.15.146:0/1114258464 conn(0x7ff99c100800 msgr2=0x7ff99c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:59.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 -- 172.21.15.146:0/1114258464 shutdown_connections 2024-04-24T08:17:59.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.699+0000 7ff9a1200700 1 -- 172.21.15.146:0/1114258464 wait complete. 2024-04-24T08:17:59.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.700+0000 7ff9a1200700 1 Processor -- start 2024-04-24T08:17:59.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff9a1200700 1 -- start start 2024-04-24T08:17:59.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff9a1200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c104f70 0x7ff99c19b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff9a1200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c105920 0x7ff99c19ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff9a1200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff99c10a050 0x7ff99c19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff9a1200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c1150b0 con 0x7ff99c10a050 2024-04-24T08:17:59.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff9a1200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c114f30 con 0x7ff99c105920 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff9a1200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c115230 con 0x7ff99c104f70 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff999e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c104f70 0x7ff99c19b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff999e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c104f70 0x7ff99c19b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40706/0 (socket says 172.21.15.146:40706) 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff999e00700 1 -- 172.21.15.146:0/880523234 learned_addr learned my addr 172.21.15.146:0/880523234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff99a800700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff99c10a050 0x7ff99c19fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.701+0000 7ff999400700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c105920 0x7ff99c19ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff999e00700 1 -- 172.21.15.146:0/880523234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c105920 msgr2=0x7ff99c19ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff999e00700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c105920 0x7ff99c19ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff999e00700 1 -- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff99c10a050 msgr2=0x7ff99c19fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff999e00700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff99c10a050 0x7ff99c19fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff999e00700 1 -- 172.21.15.146:0/880523234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c1a0480 con 0x7ff99c104f70 2024-04-24T08:17:59.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff99a800700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff99c10a050 0x7ff99c19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:17:59.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff999e00700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c104f70 0x7ff99c19b4c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00e9a0 tx=0x7ff98c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:59.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff992a00700 1 -- 172.21.15.146:0/880523234 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00cbd0 con 0x7ff99c104f70 2024-04-24T08:17:59.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99c1a0770 con 0x7ff99c104f70 2024-04-24T08:17:59.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff992a00700 1 -- 172.21.15.146:0/880523234 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff98c00cd70 con 0x7ff99c104f70 2024-04-24T08:17:59.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.702+0000 7ff992a00700 1 -- 172.21.15.146:0/880523234 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c018750 con 0x7ff99c104f70 2024-04-24T08:17:59.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.703+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99c1a7ff0 con 0x7ff99c104f70 2024-04-24T08:17:59.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.704+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff99c0697e0 con 0x7ff99c104f70 2024-04-24T08:17:59.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.706+0000 7ff992a00700 1 -- 172.21.15.146:0/880523234 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff98c010430 con 0x7ff99c104f70 2024-04-24T08:17:59.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.707+0000 7ff992a00700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff9800a3c40 0x7ff9800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.707+0000 7ff992a00700 1 -- 172.21.15.146:0/880523234 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7ff98c0d02a0 con 0x7ff99c104f70 2024-04-24T08:17:59.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.707+0000 7ff999400700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff9800a3c40 0x7ff9800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.708+0000 7ff999400700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff9800a3c40 0x7ff9800a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff99c10b560 tx=0x7ff988009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:59.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.711+0000 7ff992a00700 1 -- 172.21.15.146:0/880523234 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff98c09d080 con 0x7ff99c104f70 2024-04-24T08:17:59.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:17:59 smithi177 ceph-mon[33046]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:59.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.937+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff99c02dfe0 con 0x7ff99c104f70 2024-04-24T08:17:59.939 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:17:59 smithi146 ceph-mon[27805]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:17:59.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.938+0000 7ff992a00700 1 -- 172.21.15.146:0/880523234 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7ff98c09c980 con 0x7ff99c104f70 2024-04-24T08:17:59.940 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff9800a3c40 msgr2=0x7ff9800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff9800a3c40 0x7ff9800a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff99c10b560 tx=0x7ff988009260 comp rx=0 tx=0).stop 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c104f70 msgr2=0x7ff99c19b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c104f70 0x7ff99c19b4c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00e9a0 tx=0x7ff98c00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 shutdown_connections 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff9800a3c40 0x7ff9800a6100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff99c10a050 0x7ff99c19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff99c104f70 0x7ff99c19b4c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 --2- 172.21.15.146:0/880523234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff99c105920 0x7ff99c19ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 >> 172.21.15.146:0/880523234 conn(0x7ff99c100800 msgr2=0x7ff99c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:59.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 shutdown_connections 2024-04-24T08:17:59.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:17:59.941+0000 7ff9a1200700 1 -- 172.21.15.146:0/880523234 wait complete. 2024-04-24T08:17:59.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:17:59 smithi153 ceph-mon[32746]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:00.690 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:00 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/880523234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.a on smithi146 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.d on smithi146 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.g on smithi146 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.j on smithi146 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.b on smithi153 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.e on smithi153 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.h on smithi153 2024-04-24T08:18:00.760 INFO:tasks.cephadm:Adding mds.k on smithi153 2024-04-24T08:18:00.761 INFO:tasks.cephadm:Adding mds.c on smithi177 2024-04-24T08:18:00.761 INFO:tasks.cephadm:Adding mds.f on smithi177 2024-04-24T08:18:00.761 INFO:tasks.cephadm:Adding mds.i on smithi177 2024-04-24T08:18:00.761 INFO:tasks.cephadm:Adding mds.l on smithi177 2024-04-24T08:18:00.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l' 2024-04-24T08:18:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:00 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/880523234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:01.011 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:18:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:00 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/880523234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:01.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:01 smithi177 ceph-mon[33046]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:01.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:01 smithi153 ceph-mon[32746]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:01.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:01 smithi146 ceph-mon[27805]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:02.670 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 -- 172.21.15.177:0/1270689969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 msgr2=0x7f9ce4105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/1270689969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4105a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009990 tx=0x7f9ce0009ca0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 -- 172.21.15.177:0/1270689969 shutdown_connections 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/1270689969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce41088b0 0x7f9ce410aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/1270689969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4105a40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/1270689969 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ce4105f80 0x7f9ce4108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 -- 172.21.15.177:0/1270689969 >> 172.21.15.177:0/1270689969 conn(0x7f9ce40fcfe0 msgr2=0x7f9ce40ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 -- 172.21.15.177:0/1270689969 shutdown_connections 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 -- 172.21.15.177:0/1270689969 wait complete. 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 Processor -- start 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.668+0000 7f9cec2a7700 1 -- start start 2024-04-24T08:18:02.671 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ce4105f80 0x7f9ce41993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce41088b0 0x7f9ce41a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce410e810 con 0x7f9ce4105f80 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce410e690 con 0x7f9ce41088b0 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce410e990 con 0x7f9ce4103650 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:52394/0 (socket says 172.21.15.177:52394) 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 -- 172.21.15.177:0/219391128 learned_addr learned my addr 172.21.15.177:0/219391128 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 -- 172.21.15.177:0/219391128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce41088b0 msgr2=0x7f9ce41a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce41088b0 0x7f9ce41a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 -- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ce4105f80 msgr2=0x7f9ce41993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ce4105f80 0x7f9ce41993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 -- 172.21.15.177:0/219391128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce41a0ad0 con 0x7f9ce4103650 2024-04-24T08:18:02.672 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cea043700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4198e70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00047b0 tx=0x7f9ce0003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.673 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cdb7fe700 1 -- 172.21.15.177:0/219391128 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0027070 con 0x7f9ce4103650 2024-04-24T08:18:02.673 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cdb7fe700 1 -- 172.21.15.177:0/219391128 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce0022470 con 0x7f9ce4103650 2024-04-24T08:18:02.673 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cdb7fe700 1 -- 172.21.15.177:0/219391128 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000f6c0 con 0x7f9ce4103650 2024-04-24T08:18:02.673 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce41a0d60 con 0x7f9ce4103650 2024-04-24T08:18:02.673 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.669+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce41a1240 con 0x7f9ce4103650 2024-04-24T08:18:02.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.670+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce40697e0 con 0x7f9ce4103650 2024-04-24T08:18:02.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.672+0000 7f9cdb7fe700 1 -- 172.21.15.177:0/219391128 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9ce002a070 con 0x7f9ce4103650 2024-04-24T08:18:02.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.672+0000 7f9cdb7fe700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9cd00a3cc0 0x7f9cd00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.672+0000 7f9cdb7fe700 1 -- 172.21.15.177:0/219391128 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f9ce00d1400 con 0x7f9ce4103650 2024-04-24T08:18:02.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.673+0000 7f9ce9842700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9cd00a3cc0 0x7f9cd00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.674 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.674+0000 7f9ce9842700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9cd00a3cc0 0x7f9cd00a6180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9cd40097b0 tx=0x7f9cd4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.676 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.676+0000 7f9cdb7fe700 1 -- 172.21.15.177:0/219391128 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ce0031070 con 0x7f9ce4103650 2024-04-24T08:18:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:02 smithi177 ceph-mon[33046]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:02.886 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.885+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}) -- 0x7f9ce40631f0 con 0x7f9cd00a3cc0 2024-04-24T08:18:02.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.902+0000 7f9cdb7fe700 1 -- 172.21.15.177:0/219391128 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f9ce40631f0 con 0x7f9cd00a3cc0 2024-04-24T08:18:02.903 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mds.all update... 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9cd00a3cc0 msgr2=0x7f9cd00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9cd00a3cc0 0x7f9cd00a6180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9cd40097b0 tx=0x7f9cd4006d20 comp rx=0 tx=0).stop 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 msgr2=0x7f9ce4198e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4198e70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00047b0 tx=0x7f9ce0003680 comp rx=0 tx=0).stop 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 shutdown_connections 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ce41088b0 0x7f9ce41a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ce4103650 0x7f9ce4198e70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9cd00a3cc0 0x7f9cd00a6180 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 --2- 172.21.15.177:0/219391128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ce4105f80 0x7f9ce41993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.907 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 >> 172.21.15.177:0/219391128 conn(0x7f9ce40fcfe0 msgr2=0x7f9ce4103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:02.907 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 shutdown_connections 2024-04-24T08:18:02.907 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-24T08:18:02.905+0000 7f9cec2a7700 1 -- 172.21.15.177:0/219391128 wait complete. 2024-04-24T08:18:02.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:02 smithi153 ceph-mon[32746]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:02.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:02 smithi146 ceph-mon[27805]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:03.597 DEBUG:teuthology.orchestra.run.smithi146:mds.a> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.a.service 2024-04-24T08:18:03.604 DEBUG:teuthology.orchestra.run.smithi146:mds.d> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.d.service 2024-04-24T08:18:03.650 DEBUG:teuthology.orchestra.run.smithi146:mds.g> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.g.service 2024-04-24T08:18:03.697 DEBUG:teuthology.orchestra.run.smithi146:mds.j> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.j.service 2024-04-24T08:18:03.744 DEBUG:teuthology.orchestra.run.smithi153:mds.b> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.b.service 2024-04-24T08:18:03.747 DEBUG:teuthology.orchestra.run.smithi153:mds.e> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.e.service 2024-04-24T08:18:03.789 DEBUG:teuthology.orchestra.run.smithi153:mds.h> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.h.service 2024-04-24T08:18:03.833 DEBUG:teuthology.orchestra.run.smithi153:mds.k> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.k.service 2024-04-24T08:18:03.876 DEBUG:teuthology.orchestra.run.smithi177:mds.c> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.c.service 2024-04-24T08:18:03.879 DEBUG:teuthology.orchestra.run.smithi177:mds.f> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.f.service 2024-04-24T08:18:03.922 DEBUG:teuthology.orchestra.run.smithi177:mds.i> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.i.service 2024-04-24T08:18:03.966 DEBUG:teuthology.orchestra.run.smithi177:mds.l> sudo journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.l.service 2024-04-24T08:18:04.011 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-24T08:18:04.011 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:04.079 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:03 smithi146 ceph-mon[27805]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:04.079 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:03 smithi146 ceph-mon[27805]: Saving service mds.all spec with placement smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-24T08:18:04.079 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:04.079 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:03 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:04.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:03 smithi153 ceph-mon[32746]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:04.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:03 smithi153 ceph-mon[32746]: Saving service mds.all spec with placement smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-24T08:18:04.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:04.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:03 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 -- 172.21.15.146:0/1196893899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a810a030 msgr2=0x7f46a81108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 --2- 172.21.15.146:0/1196893899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a810a030 0x7f46a81108c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009ca0 tx=0x7f46a002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 -- 172.21.15.146:0/1196893899 shutdown_connections 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 --2- 172.21.15.146:0/1196893899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a810a030 0x7f46a81108c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 --2- 172.21.15.146:0/1196893899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a8109670 0x7f46a8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 --2- 172.21.15.146:0/1196893899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a8108480 0x7f46a81088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 -- 172.21.15.146:0/1196893899 >> 172.21.15.146:0/1196893899 conn(0x7f46a8077440 msgr2=0x7f46a8079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 -- 172.21.15.146:0/1196893899 shutdown_connections 2024-04-24T08:18:04.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.243+0000 7f46ada00640 1 -- 172.21.15.146:0/1196893899 wait complete. 2024-04-24T08:18:04.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 Processor -- start 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 -- start start 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a8108480 0x7f46a819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a8109670 0x7f46a819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a810a030 0x7f46a81a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8114130 con 0x7f46a810a030 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8113fb0 con 0x7f46a8109670 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.244+0000 7f46ada00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46a81142b0 con 0x7f46a8108480 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a8108480 0x7f46a819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a8108480 0x7f46a819e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45930/0 (socket says 172.21.15.146:45930) 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46ad000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a810a030 0x7f46a81a5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 -- 172.21.15.146:0/3074250050 learned_addr learned my addr 172.21.15.146:0/3074250050 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7400640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a8109670 0x7f46a819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 -- 172.21.15.146:0/3074250050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a8109670 msgr2=0x7f46a819ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a8109670 0x7f46a819ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 -- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a810a030 msgr2=0x7f46a81a5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a810a030 0x7f46a81a5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 -- 172.21.15.146:0/3074250050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a81a6360 con 0x7f46a8108480 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46ad000640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a810a030 0x7f46a81a5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.245+0000 7f46a7e00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a8108480 0x7f46a819e740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f469800e9c0 tx=0x7f469800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.246+0000 7f46a4c00640 1 -- 172.21.15.146:0/3074250050 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469800cd80 con 0x7f46a8108480 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.246+0000 7f46a4c00640 1 -- 172.21.15.146:0/3074250050 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46980035f0 con 0x7f46a8108480 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.246+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a81a6650 con 0x7f46a8108480 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.246+0000 7f46a4c00640 1 -- 172.21.15.146:0/3074250050 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698010690 con 0x7f46a8108480 2024-04-24T08:18:04.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.246+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a81a6bf0 con 0x7f46a8108480 2024-04-24T08:18:04.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.247+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a810f740 con 0x7f46a8108480 2024-04-24T08:18:04.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.249+0000 7f46a4c00640 1 -- 172.21.15.146:0/3074250050 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f46980037a0 con 0x7f46a8108480 2024-04-24T08:18:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.250+0000 7f46a4c00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.250+0000 7f46a4c00640 1 -- 172.21.15.146:0/3074250050 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f46980d0360 con 0x7f46a8108480 2024-04-24T08:18:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.250+0000 7f46a7400640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.251+0000 7f46a7400640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f46a819fc50 tx=0x7f469400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.254+0000 7f46a4c00640 1 -- 172.21.15.146:0/3074250050 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f469809d140 con 0x7f46a8108480 2024-04-24T08:18:04.273 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:03 smithi177 ceph-mon[33046]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:04.273 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:03 smithi177 ceph-mon[33046]: Saving service mds.all spec with placement smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-24T08:18:04.273 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:04.273 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:03 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:04.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.456+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46a8062c30 con 0x7f46a8108480 2024-04-24T08:18:04.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.458+0000 7f46a4c00640 1 -- 172.21.15.146:0/3074250050 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18499 (secure 0 0 0) 0x7f469809ca40 con 0x7f46a8108480 2024-04-24T08:18:04.460 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:04.460 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":84,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:17:54.027821+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:04.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.460+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f468c0a3c30 msgr2=0x7f468c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.460+0000 7f46ada00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f468c0a3c30 0x7f468c0a60f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f46a819fc50 tx=0x7f469400b410 comp rx=0 tx=0).stop 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.460+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a8108480 msgr2=0x7f46a819e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.460+0000 7f46ada00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a8108480 0x7f46a819e740 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f469800e9c0 tx=0x7f469800ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 shutdown_connections 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f468c0a3c30 0x7f468c0a60f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46a810a030 0x7f46a81a5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46a8109670 0x7f46a819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 --2- 172.21.15.146:0/3074250050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46a8108480 0x7f46a819e740 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 >> 172.21.15.146:0/3074250050 conn(0x7f46a8077440 msgr2=0x7f46a810c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 shutdown_connections 2024-04-24T08:18:04.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.461+0000 7f46ada00640 1 -- 172.21.15.146:0/3074250050 wait complete. 2024-04-24T08:18:04.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:04.482 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:04.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 -- 172.21.15.146:0/3958498695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5010a0f0 msgr2=0x7f3b50110980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 --2- 172.21.15.146:0/3958498695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5010a0f0 0x7f3b50110980 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009ca0 tx=0x7f3b4c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 -- 172.21.15.146:0/3958498695 shutdown_connections 2024-04-24T08:18:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 --2- 172.21.15.146:0/3958498695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b50109730 0x7f3b50109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 --2- 172.21.15.146:0/3958498695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5010a0f0 0x7f3b50110980 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 --2- 172.21.15.146:0/3958498695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 0x7f3b50108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 -- 172.21.15.146:0/3958498695 >> 172.21.15.146:0/3958498695 conn(0x7f3b500774d0 msgr2=0x7f3b50079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 -- 172.21.15.146:0/3958498695 shutdown_connections 2024-04-24T08:18:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.690+0000 7f3b57200640 1 -- 172.21.15.146:0/3958498695 wait complete. 2024-04-24T08:18:04.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.691+0000 7f3b57200640 1 Processor -- start 2024-04-24T08:18:04.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.691+0000 7f3b57200640 1 -- start start 2024-04-24T08:18:04.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b57200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 0x7f3b5019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b57200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b50109730 0x7f3b5019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b57200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b5010a0f0 0x7f3b501a5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b57200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501141d0 con 0x7f3b50109730 2024-04-24T08:18:04.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b57200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3b50114050 con 0x7f3b50108540 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b57200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b50114350 con 0x7f3b5010a0f0 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 0x7f3b5019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 0x7f3b5019e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43128/0 (socket says 172.21.15.146:43128) 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 -- 172.21.15.146:0/3387177861 learned_addr learned my addr 172.21.15.146:0/3387177861 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55400640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b50109730 0x7f3b5019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b56800640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b5010a0f0 0x7f3b501a5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 -- 172.21.15.146:0/3387177861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b5010a0f0 msgr2=0x7f3b501a5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b5010a0f0 0x7f3b501a5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 -- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b50109730 msgr2=0x7f3b5019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b50109730 0x7f3b5019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b55e00640 1 -- 172.21.15.146:0/3387177861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b501a63b0 con 0x7f3b50108540 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.692+0000 7f3b56800640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b5010a0f0 0x7f3b501a5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.693+0000 7f3b55e00640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 0x7f3b5019e790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400e9c0 tx=0x7f3b4400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.693+0000 7f3b4aa00640 1 -- 172.21.15.146:0/3387177861 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400cd80 con 0x7f3b50108540 2024-04-24T08:18:04.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.693+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b501a66a0 con 0x7f3b50108540 2024-04-24T08:18:04.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.693+0000 7f3b4aa00640 1 -- 172.21.15.146:0/3387177861 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b440035f0 con 0x7f3b50108540 2024-04-24T08:18:04.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.693+0000 7f3b4aa00640 1 -- 172.21.15.146:0/3387177861 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44010690 con 0x7f3b50108540 2024-04-24T08:18:04.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.693+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b501a6c40 con 0x7f3b50108540 2024-04-24T08:18:04.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.694+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b5010f800 con 0x7f3b50108540 2024-04-24T08:18:04.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.696+0000 7f3b4aa00640 1 -- 172.21.15.146:0/3387177861 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f3b440037a0 con 0x7f3b50108540 2024-04-24T08:18:04.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.698+0000 7f3b4aa00640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b380a3bb0 0x7f3b380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.698+0000 7f3b4aa00640 1 -- 172.21.15.146:0/3387177861 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f3b440d0490 con 0x7f3b50108540 2024-04-24T08:18:04.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.698+0000 7f3b55400640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b380a3bb0 0x7f3b380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.699+0000 7f3b55400640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b380a3bb0 0x7f3b380a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b5019fca0 tx=0x7f3b40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.702+0000 7f3b4aa00640 1 -- 172.21.15.146:0/3387177861 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b4409d270 con 0x7f3b50108540 2024-04-24T08:18:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.906+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b50062c00 con 0x7f3b50108540 2024-04-24T08:18:04.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.909+0000 7f3b4aa00640 1 -- 172.21.15.146:0/3387177861 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f3b4409cb70 con 0x7f3b50108540 2024-04-24T08:18:04.911 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:18:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.911+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b380a3bb0 msgr2=0x7f3b380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.911+0000 7f3b57200640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b380a3bb0 0x7f3b380a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b5019fca0 tx=0x7f3b40005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 msgr2=0x7f3b5019e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 0x7f3b5019e790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400e9c0 tx=0x7f3b4400ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 shutdown_connections 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b5010a0f0 0x7f3b501a5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b380a3bb0 0x7f3b380a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b50109730 0x7f3b5019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 --2- 172.21.15.146:0/3387177861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b50108540 0x7f3b5019e790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 >> 172.21.15.146:0/3387177861 conn(0x7f3b500774d0 msgr2=0x7f3b5010c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 shutdown_connections 2024-04-24T08:18:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:04.912+0000 7f3b57200640 1 -- 172.21.15.146:0/3387177861 wait complete. 2024-04-24T08:18:04.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:05.047 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:05.047 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:05.047 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:05.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:05.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:05.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:05.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: Deploying daemon mds.l on smithi177 2024-04-24T08:18:05.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:05.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:04 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3074250050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:05.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 -- 172.21.15.146:0/3120024346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 msgr2=0x7f95c410ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 --2- 172.21.15.146:0/3120024346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 0x7f95c410ce70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009900 tx=0x7f95b802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 -- 172.21.15.146:0/3120024346 shutdown_connections 2024-04-24T08:18:05.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 --2- 172.21.15.146:0/3120024346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 0x7f95c4114e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 --2- 172.21.15.146:0/3120024346 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c410dc40 0x7f95c410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 --2- 172.21.15.146:0/3120024346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 0x7f95c410ce70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 -- 172.21.15.146:0/3120024346 >> 172.21.15.146:0/3120024346 conn(0x7f95c4073200 msgr2=0x7f95c4073610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:05.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 -- 172.21.15.146:0/3120024346 shutdown_connections 2024-04-24T08:18:05.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.137+0000 7f95cc200640 1 -- 172.21.15.146:0/3120024346 wait complete. 2024-04-24T08:18:05.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 Processor -- start 2024-04-24T08:18:05.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 -- start start 2024-04-24T08:18:05.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 0x7f95c41a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c410dc40 0x7f95c41a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 0x7f95c41aa310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f95c41186f0 con 0x7f95c410e600 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95c4118570 con 0x7f95c410dc40 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95cc200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f95c4118870 con 0x7f95c410ca50 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95ca200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 0x7f95c41aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95ca200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 0x7f95c41aa310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43086/0 (socket says 172.21.15.146:43086) 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.139+0000 7f95c9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 0x7f95c41a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95ca200640 1 -- 172.21.15.146:0/1941366645 learned_addr learned my addr 172.21.15.146:0/1941366645 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95c8e00640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c410dc40 0x7f95c41a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95c9800640 1 -- 172.21.15.146:0/1941366645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c410dc40 msgr2=0x7f95c41a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95c9800640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c410dc40 0x7f95c41a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95c9800640 1 -- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 msgr2=0x7f95c41aa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95c9800640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 0x7f95c41aa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95c9800640 1 -- 172.21.15.146:0/1941366645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c41aa980 con 0x7f95c410ca50 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95ca200640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 0x7f95c41aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.140+0000 7f95c9800640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 0x7f95c41a2d60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009230 tx=0x7f95b8032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.141+0000 7f95c2000640 1 -- 172.21.15.146:0/1941366645 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b803f030 con 0x7f95c410ca50 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.141+0000 7f95c2000640 1 -- 172.21.15.146:0/1941366645 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b8003ce0 con 0x7f95c410ca50 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.141+0000 7f95c2000640 1 -- 172.21.15.146:0/1941366645 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b8035070 con 0x7f95c410ca50 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.141+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c41aac10 con 0x7f95c410ca50 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.141+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c41ab080 con 0x7f95c410ca50 2024-04-24T08:18:05.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.142+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c4113d70 con 0x7f95c410ca50 2024-04-24T08:18:05.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.144+0000 7f95c2000640 1 -- 172.21.15.146:0/1941366645 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f95b8042030 con 0x7f95c410ca50 2024-04-24T08:18:05.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.145+0000 7f95c2000640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f95ac0a3c30 0x7f95ac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.145+0000 7f95c2000640 1 -- 172.21.15.146:0/1941366645 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f95b80f04b0 con 0x7f95c410ca50 2024-04-24T08:18:05.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.145+0000 7f95c8e00640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f95ac0a3c30 0x7f95ac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.146+0000 7f95c8e00640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f95ac0a3c30 0x7f95ac0a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95c41a4270 tx=0x7f95b000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:05.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.149+0000 7f95c2000640 1 -- 172.21.15.146:0/1941366645 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95b80bcc40 con 0x7f95c410ca50 2024-04-24T08:18:05.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: Deploying daemon mds.l on smithi177 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:05.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:04 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3074250050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:05.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:05.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:05.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:05.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:05.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:05.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:05.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: Deploying daemon mds.l on smithi177 2024-04-24T08:18:05.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:05.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:04 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3074250050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:05.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.394+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95c4062c00 con 0x7f95c410ca50 2024-04-24T08:18:05.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.395+0000 7f95c2000640 1 -- 172.21.15.146:0/1941366645 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f95b80c0f40 con 0x7f95c410ca50 2024-04-24T08:18:05.397 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:05.397 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-24T08:18:05.397 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 1 2024-04-24T08:18:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f95ac0a3c30 msgr2=0x7f95ac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f95ac0a3c30 0x7f95ac0a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95c41a4270 tx=0x7f95b000b390 comp rx=0 tx=0).stop 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 msgr2=0x7f95c41a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 0x7f95c41a2d60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009230 tx=0x7f95b8032d50 comp rx=0 tx=0).stop 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 shutdown_connections 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f95ac0a3c30 0x7f95ac0a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95c410e600 0x7f95c41aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95c410dc40 0x7f95c41a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 --2- 172.21.15.146:0/1941366645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95c410ca50 0x7f95c41a2d60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.397+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 >> 172.21.15.146:0/1941366645 conn(0x7f95c4073200 msgr2=0x7f95c4110f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.398+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 shutdown_connections 2024-04-24T08:18:05.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:05.398+0000 7f95cc200640 1 -- 172.21.15.146:0/1941366645 wait complete. 2024-04-24T08:18:05.965 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:05 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3387177861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:05.965 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:05 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1941366645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:06.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:05 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3387177861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:06.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:05 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1941366645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:06.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:05 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3387177861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:06.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:05 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1941366645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:06.315 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 systemd[1]: Starting Ceph mds.l for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:06.639 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 podman[63508]: 2024-04-24 08:18:06.272306339 +0000 UTC m=+0.018196814 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:06.639 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 podman[63508]: 2024-04-24 08:18:06.394280145 +0000 UTC m=+0.140170665 container create 8f4f881ab3472989e283d9df472de3d9130aa320666aec59e0a52d196c8d1107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-24T08:18:06.639 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 podman[63508]: 2024-04-24 08:18:06.638565785 +0000 UTC m=+0.384456259 container init 8f4f881ab3472989e283d9df472de3d9130aa320666aec59e0a52d196c8d1107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:18:06.914 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 podman[63508]: 2024-04-24 08:18:06.641299064 +0000 UTC m=+0.387189541 container start 8f4f881ab3472989e283d9df472de3d9130aa320666aec59e0a52d196c8d1107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=) 2024-04-24T08:18:06.914 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mds[63527]: -- 172.21.15.177:0/2796673132 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d565741a40 con 0x55d565714800 2024-04-24T08:18:06.914 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l[63523]: starting mds.l at 2024-04-24T08:18:06.915 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 bash[63508]: 8f4f881ab3472989e283d9df472de3d9130aa320666aec59e0a52d196c8d1107 2024-04-24T08:18:06.915 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:18:06 smithi177 systemd[1]: Started Ceph mds.l for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:07.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.177:6832/3628851028,v1:172.21.15.177:6833/3628851028] up:boot 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: fsmap 1 up:standby 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:06 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.177:6832/3628851028,v1:172.21.15.177:6833/3628851028] up:boot 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: fsmap 1 up:standby 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:07.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:06 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.177:6832/3628851028,v1:172.21.15.177:6833/3628851028] up:boot 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: fsmap 1 up:standby 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:07.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:06 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:07.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:07.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.632+0000 7fdc4e800640 1 -- 172.21.15.146:0/40644430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48104280 msgr2=0x7fdc481046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.632+0000 7fdc4e800640 1 --2- 172.21.15.146:0/40644430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48104280 0x7fdc481046a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009900 tx=0x7fdc3802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.633+0000 7fdc4e800640 1 -- 172.21.15.146:0/40644430 shutdown_connections 2024-04-24T08:18:07.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.633+0000 7fdc4e800640 1 --2- 172.21.15.146:0/40644430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc48105470 0x7fdc481058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.633+0000 7fdc4e800640 1 --2- 172.21.15.146:0/40644430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48105e30 0x7fdc4810c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.633+0000 7fdc4e800640 1 --2- 172.21.15.146:0/40644430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48104280 0x7fdc481046a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.633+0000 7fdc4e800640 1 -- 172.21.15.146:0/40644430 >> 172.21.15.146:0/40644430 conn(0x7fdc480ff7f0 msgr2=0x7fdc48101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:07.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.633+0000 7fdc4e800640 1 -- 172.21.15.146:0/40644430 shutdown_connections 2024-04-24T08:18:07.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.633+0000 7fdc4e800640 1 -- 172.21.15.146:0/40644430 wait complete. 2024-04-24T08:18:07.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 Processor -- start 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 -- start start 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48104280 0x7fdc4819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48105470 0x7fdc4819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc48105e30 0x7fdc481a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810ff30 con 0x7fdc48105470 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810fdb0 con 0x7fdc48104280 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc4e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481100b0 con 0x7fdc48105e30 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc47400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48104280 0x7fdc4819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.634+0000 7fdc47400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48104280 0x7fdc4819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43168/0 (socket says 172.21.15.146:43168) 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc3fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48105470 0x7fdc4819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47400640 1 -- 172.21.15.146:0/2197855265 learned_addr learned my addr 172.21.15.146:0/2197855265 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc48105e30 0x7fdc481a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47400640 1 -- 172.21.15.146:0/2197855265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc48105e30 msgr2=0x7fdc481a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47400640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc48105e30 0x7fdc481a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47400640 1 -- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48105470 msgr2=0x7fdc4819ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47400640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48105470 0x7fdc4819ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47400640 1 -- 172.21.15.146:0/2197855265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc481a2210 con 0x7fdc48104280 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc47400640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48104280 0x7fdc4819a5f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc38031440 tx=0x7fdc38032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc44c00640 1 -- 172.21.15.146:0/2197855265 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3803f030 con 0x7fdc48104280 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.635+0000 7fdc44c00640 1 -- 172.21.15.146:0/2197855265 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc38003ce0 con 0x7fdc48104280 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.636+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc481a24a0 con 0x7fdc48104280 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.636+0000 7fdc44c00640 1 -- 172.21.15.146:0/2197855265 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38035070 con 0x7fdc48104280 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.636+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc481a29e0 con 0x7fdc48104280 2024-04-24T08:18:07.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.637+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4810b5a0 con 0x7fdc48104280 2024-04-24T08:18:07.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.638+0000 7fdc44c00640 1 -- 172.21.15.146:0/2197855265 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fdc38042030 con 0x7fdc48104280 2024-04-24T08:18:07.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.640+0000 7fdc44c00640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc2c0a3c30 0x7fdc2c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.640+0000 7fdc44c00640 1 -- 172.21.15.146:0/2197855265 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fdc380f0400 con 0x7fdc48104280 2024-04-24T08:18:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.640+0000 7fdc3fe00640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc2c0a3c30 0x7fdc2c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.641+0000 7fdc3fe00640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc2c0a3c30 0x7fdc2c0a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdc300095f0 tx=0x7fdc30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.644+0000 7fdc44c00640 1 -- 172.21.15.146:0/2197855265 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc380bd1e0 con 0x7fdc48104280 2024-04-24T08:18:07.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.888+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc4802d620 con 0x7fdc48104280 2024-04-24T08:18:07.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.890+0000 7fdc44c00640 1 -- 172.21.15.146:0/2197855265 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fdc380bcae0 con 0x7fdc48104280 2024-04-24T08:18:07.891 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:07.891 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-24T08:18:07.892 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 2 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc2c0a3c30 msgr2=0x7fdc2c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc2c0a3c30 0x7fdc2c0a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdc300095f0 tx=0x7fdc30005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48104280 msgr2=0x7fdc4819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48104280 0x7fdc4819a5f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc38031440 tx=0x7fdc38032ce0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 shutdown_connections 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc48105e30 0x7fdc481a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc2c0a3c30 0x7fdc2c0a60f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc48105470 0x7fdc4819ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 --2- 172.21.15.146:0/2197855265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc48104280 0x7fdc4819a5f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 >> 172.21.15.146:0/2197855265 conn(0x7fdc480ff7f0 msgr2=0x7fdc48108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:07.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 shutdown_connections 2024-04-24T08:18:07.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:07.892+0000 7fdc4e800640 1 -- 172.21.15.146:0/2197855265 wait complete. 2024-04-24T08:18:08.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:07 smithi153 ceph-mon[32746]: Deploying daemon mds.k on smithi153 2024-04-24T08:18:08.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:07 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2197855265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:08.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:07 smithi146 ceph-mon[27805]: Deploying daemon mds.k on smithi153 2024-04-24T08:18:08.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:07 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2197855265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:08.314 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:07 smithi177 ceph-mon[33046]: Deploying daemon mds.k on smithi153 2024-04-24T08:18:08.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:07 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2197855265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:09.190 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:08 smithi153 ceph-mon[32746]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:09.191 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:08 smithi153 systemd[1]: Starting Ceph mds.k for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:09.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:08 smithi146 ceph-mon[27805]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:08 smithi177 ceph-mon[33046]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:09.460 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 podman[61830]: 2024-04-24 08:18:09.093054377 +0000 UTC m=+0.018665425 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:09.460 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 podman[61830]: 2024-04-24 08:18:09.264773896 +0000 UTC m=+0.190384943 container create 5015291c49b3be6f6d3c7992293538d22b7881e223f38786c3952f83b9dc4b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-24T08:18:09.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:09.961 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 podman[61830]: 2024-04-24 08:18:09.502482098 +0000 UTC m=+0.428093145 container init 5015291c49b3be6f6d3c7992293538d22b7881e223f38786c3952f83b9dc4b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:18:09.961 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 podman[61830]: 2024-04-24 08:18:09.505190625 +0000 UTC m=+0.430801672 container start 5015291c49b3be6f6d3c7992293538d22b7881e223f38786c3952f83b9dc4b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-24T08:18:09.961 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 ceph-mds[61849]: -- 172.21.15.153:0/4164536235 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563344c59a40 con 0x563344c2d000 2024-04-24T08:18:09.961 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k[61845]: starting mds.k at 2024-04-24T08:18:09.961 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 bash[61830]: 5015291c49b3be6f6d3c7992293538d22b7881e223f38786c3952f83b9dc4b53 2024-04-24T08:18:09.961 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:18:09 smithi153 systemd[1]: Started Ceph mds.k for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:10.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.107+0000 7f9e1f200640 1 -- 172.21.15.146:0/107842344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20102900 msgr2=0x7f9e20104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.107+0000 7f9e1f200640 1 --2- 172.21.15.146:0/107842344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20102900 0x7f9e20104cf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009860 tx=0x7f9e1002a340 comp rx=0 tx=0).stop 2024-04-24T08:18:10.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.108+0000 7f9e1f200640 1 -- 172.21.15.146:0/107842344 shutdown_connections 2024-04-24T08:18:10.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.108+0000 7f9e1f200640 1 --2- 172.21.15.146:0/107842344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e20107b60 0x7f9e20109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.108+0000 7f9e1f200640 1 --2- 172.21.15.146:0/107842344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20105230 0x7f9e20107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.108+0000 7f9e1f200640 1 --2- 172.21.15.146:0/107842344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20102900 0x7f9e20104cf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.108+0000 7f9e1f200640 1 -- 172.21.15.146:0/107842344 >> 172.21.15.146:0/107842344 conn(0x7f9e200fc2d0 msgr2=0x7f9e200fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:10.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.108+0000 7f9e1f200640 1 -- 172.21.15.146:0/107842344 shutdown_connections 2024-04-24T08:18:10.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.108+0000 7f9e1f200640 1 -- 172.21.15.146:0/107842344 wait complete. 2024-04-24T08:18:10.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 Processor -- start 2024-04-24T08:18:10.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 -- start start 2024-04-24T08:18:10.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20102900 0x7f9e2019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e20105230 0x7f9e2019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20107b60 0x7f9e201a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010db20 con 0x7f9e20102900 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010d9a0 con 0x7f9e20105230 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010dca0 con 0x7f9e20107b60 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20102900 0x7f9e2019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20102900 0x7f9e2019a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43112/0 (socket says 172.21.15.146:43112) 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1de00640 1 -- 172.21.15.146:0/461074255 learned_addr learned my addr 172.21.15.146:0/461074255 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e1e800640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20107b60 0x7f9e201a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.109+0000 7f9e17e00640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e20105230 0x7f9e2019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e1de00640 1 -- 172.21.15.146:0/461074255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20107b60 msgr2=0x7f9e201a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e1de00640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20107b60 0x7f9e201a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e1de00640 1 -- 172.21.15.146:0/461074255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e20105230 msgr2=0x7f9e2019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e1de00640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e20105230 0x7f9e2019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e1de00640 1 -- 172.21.15.146:0/461074255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e201a1fd0 con 0x7f9e20102900 2024-04-24T08:18:10.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e1e800640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20107b60 0x7f9e201a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:10.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e1de00640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20102900 0x7f9e2019a3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9e10005940 tx=0x7f9e10033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:10.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.110+0000 7f9e16a00640 1 -- 172.21.15.146:0/461074255 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1003f020 con 0x7f9e20102900 2024-04-24T08:18:10.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.111+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e201a2260 con 0x7f9e20102900 2024-04-24T08:18:10.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.111+0000 7f9e16a00640 1 -- 172.21.15.146:0/461074255 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e10030850 con 0x7f9e20102900 2024-04-24T08:18:10.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.111+0000 7f9e16a00640 1 -- 172.21.15.146:0/461074255 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10037040 con 0x7f9e20102900 2024-04-24T08:18:10.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.111+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e201a2750 con 0x7f9e20102900 2024-04-24T08:18:10.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.112+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e201096b0 con 0x7f9e20102900 2024-04-24T08:18:10.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.115+0000 7f9e16a00640 1 -- 172.21.15.146:0/461074255 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9e10042020 con 0x7f9e20102900 2024-04-24T08:18:10.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.117+0000 7f9e16a00640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e040a3bb0 0x7f9e040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.117+0000 7f9e16a00640 1 -- 172.21.15.146:0/461074255 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f9e100f0330 con 0x7f9e20102900 2024-04-24T08:18:10.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.117+0000 7f9e17e00640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e040a3bb0 0x7f9e040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.118+0000 7f9e17e00640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e040a3bb0 0x7f9e040a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9e2019b8c0 tx=0x7f9e08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:10.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.122+0000 7f9e16a00640 1 -- 172.21.15.146:0/461074255 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e10049020 con 0x7f9e20102900 2024-04-24T08:18:10.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.369+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e2002d840 con 0x7f9e20102900 2024-04-24T08:18:10.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.369+0000 7f9e16a00640 1 -- 172.21.15.146:0/461074255 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f9e100bd110 con 0x7f9e20102900 2024-04-24T08:18:10.372 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:10.372 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T08:18:10.372 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 3 2024-04-24T08:18:10.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e040a3bb0 msgr2=0x7f9e040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e040a3bb0 0x7f9e040a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9e2019b8c0 tx=0x7f9e08009260 comp rx=0 tx=0).stop 2024-04-24T08:18:10.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20102900 msgr2=0x7f9e2019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20102900 0x7f9e2019a3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9e10005940 tx=0x7f9e10033d90 comp rx=0 tx=0).stop 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 shutdown_connections 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e20107b60 0x7f9e201a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e040a3bb0 0x7f9e040a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e20105230 0x7f9e2019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 --2- 172.21.15.146:0/461074255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e20102900 0x7f9e2019a3b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.372+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 >> 172.21.15.146:0/461074255 conn(0x7f9e200fc2d0 msgr2=0x7f9e201020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.373+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 shutdown_connections 2024-04-24T08:18:10.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:10.373+0000 7f9e1f200640 1 -- 172.21.15.146:0/461074255 wait complete. 2024-04-24T08:18:10.895 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:10.895 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:10.895 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:10.895 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: Deploying daemon mds.j on smithi146 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.153:6832/3377566039,v1:172.21.15.153:6833/3377566039] up:boot 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: fsmap 2 up:standby 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/461074255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:10.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:10 smithi146 ceph-mon[27805]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:10.904 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:10.904 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:10.904 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:10.904 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:10.904 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: Deploying daemon mds.j on smithi146 2024-04-24T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.153:6832/3377566039,v1:172.21.15.153:6833/3377566039] up:boot 2024-04-24T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: fsmap 2 up:standby 2024-04-24T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/461074255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:10.905 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:10 smithi153 ceph-mon[32746]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:11.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:11.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:11.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:11.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:11.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:11.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:11.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: Deploying daemon mds.j on smithi146 2024-04-24T08:18:11.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.153:6832/3377566039,v1:172.21.15.153:6833/3377566039] up:boot 2024-04-24T08:18:11.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: fsmap 2 up:standby 2024-04-24T08:18:11.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:18:11.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/461074255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:11.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:10 smithi177 ceph-mon[33046]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:12.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:12.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.793+0000 7fe376a00640 1 -- 172.21.15.146:0/232473921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 msgr2=0x7fe370104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.793+0000 7fe376a00640 1 --2- 172.21.15.146:0/232473921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 0x7fe370104680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe360009900 tx=0x7fe36002a410 comp rx=0 tx=0).stop 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.793+0000 7fe376a00640 1 -- 172.21.15.146:0/232473921 shutdown_connections 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.793+0000 7fe376a00640 1 --2- 172.21.15.146:0/232473921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe370105e10 0x7fe37010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.793+0000 7fe376a00640 1 --2- 172.21.15.146:0/232473921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe370105450 0x7fe3701058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.793+0000 7fe376a00640 1 --2- 172.21.15.146:0/232473921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 0x7fe370104680 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.793+0000 7fe376a00640 1 -- 172.21.15.146:0/232473921 >> 172.21.15.146:0/232473921 conn(0x7fe3700ff7f0 msgr2=0x7fe370101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.794+0000 7fe376a00640 1 -- 172.21.15.146:0/232473921 shutdown_connections 2024-04-24T08:18:12.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.794+0000 7fe376a00640 1 -- 172.21.15.146:0/232473921 wait complete. 2024-04-24T08:18:12.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 Processor -- start 2024-04-24T08:18:12.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 -- start start 2024-04-24T08:18:12.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 0x7fe37019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:12.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe370105450 0x7fe37019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe370105e10 0x7fe3701a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe37010ff40 con 0x7fe370104260 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe37010fdc0 con 0x7fe370105e10 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe376a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701100c0 con 0x7fe370105450 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe36fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 0x7fe37019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe36f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe370105450 0x7fe37019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe36f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe370105450 0x7fe37019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:35632/0 (socket says 172.21.15.146:35632) 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe36f400640 1 -- 172.21.15.146:0/4230371464 learned_addr learned my addr 172.21.15.146:0/4230371464 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.795+0000 7fe374a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe370105e10 0x7fe3701a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe374a00640 1 -- 172.21.15.146:0/4230371464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe370105450 msgr2=0x7fe37019aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe374a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe370105450 0x7fe37019aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe374a00640 1 -- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 msgr2=0x7fe37019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe374a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 0x7fe37019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe374a00640 1 -- 172.21.15.146:0/4230371464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3701a2160 con 0x7fe370105e10 2024-04-24T08:18:12.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe374a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe370105e10 0x7fe3701a1af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe368009fd0 tx=0x7fe36800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:12.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe36cc00640 1 -- 172.21.15.146:0/4230371464 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36800ce20 con 0x7fe370105e10 2024-04-24T08:18:12.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe36cc00640 1 -- 172.21.15.146:0/4230371464 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe368005020 con 0x7fe370105e10 2024-04-24T08:18:12.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3701a2450 con 0x7fe370105e10 2024-04-24T08:18:12.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe36cc00640 1 -- 172.21.15.146:0/4230371464 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe368010720 con 0x7fe370105e10 2024-04-24T08:18:12.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.796+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3701a29a0 con 0x7fe370105e10 2024-04-24T08:18:12.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.797+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37010b5a0 con 0x7fe370105e10 2024-04-24T08:18:12.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.799+0000 7fe36cc00640 1 -- 172.21.15.146:0/4230371464 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe368005540 con 0x7fe370105e10 2024-04-24T08:18:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.801+0000 7fe36cc00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe3540a3bb0 0x7fe3540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:12.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.801+0000 7fe36cc00640 1 -- 172.21.15.146:0/4230371464 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fe3680d05d0 con 0x7fe370105e10 2024-04-24T08:18:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.801+0000 7fe36fe00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe3540a3bb0 0x7fe3540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.802+0000 7fe36fe00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe3540a3bb0 0x7fe3540a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe36002aa00 tx=0x7fe3600315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:12.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:12.805+0000 7fe36cc00640 1 -- 172.21.15.146:0/4230371464 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe36809d3b0 con 0x7fe370105e10 2024-04-24T08:18:12.992 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:12 smithi146 systemd[1]: Starting Ceph mds.j for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:13.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.063+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe370062c00 con 0x7fe370105e10 2024-04-24T08:18:13.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.064+0000 7fe36cc00640 1 -- 172.21.15.146:0/4230371464 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fe36809ccb0 con 0x7fe370105e10 2024-04-24T08:18:13.065 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:13.066 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T08:18:13.066 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 3 2024-04-24T08:18:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe3540a3bb0 msgr2=0x7fe3540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe3540a3bb0 0x7fe3540a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe36002aa00 tx=0x7fe3600315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe370105e10 msgr2=0x7fe3701a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe370105e10 0x7fe3701a1af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe368009fd0 tx=0x7fe36800ef10 comp rx=0 tx=0).stop 2024-04-24T08:18:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 shutdown_connections 2024-04-24T08:18:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe370105450 0x7fe37019aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe3540a3bb0 0x7fe3540a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe370105e10 0x7fe3701a1af0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 --2- 172.21.15.146:0/4230371464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe370104260 0x7fe37019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 >> 172.21.15.146:0/4230371464 conn(0x7fe3700ff7f0 msgr2=0x7fe370108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:13.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 shutdown_connections 2024-04-24T08:18:13.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:13.067+0000 7fe376a00640 1 -- 172.21.15.146:0/4230371464 wait complete. 2024-04-24T08:18:13.336 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 podman[68094]: 2024-04-24 08:18:12.958455536 +0000 UTC m=+0.031084744 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:13.336 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 podman[68094]: 2024-04-24 08:18:13.07446535 +0000 UTC m=+0.147094560 container create 2e91614dea602b93596bc2ef1d4cdb67367a59368b1c75266765f703883c1563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T08:18:13.336 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 podman[68094]: 2024-04-24 08:18:13.334256524 +0000 UTC m=+0.406885774 container init 2e91614dea602b93596bc2ef1d4cdb67367a59368b1c75266765f703883c1563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-24T08:18:13.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:13 smithi146 ceph-mon[27805]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:13.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:13 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4230371464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:13.742 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 podman[68094]: 2024-04-24 08:18:13.346212737 +0000 UTC m=+0.418841948 container start 2e91614dea602b93596bc2ef1d4cdb67367a59368b1c75266765f703883c1563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:18:13.743 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 ceph-mds[68116]: -- 172.21.15.146:0/4113031996 <== mon.0 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d131b0da40 con 0x55d131ae0800 2024-04-24T08:18:13.743 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j[68112]: starting mds.j at 2024-04-24T08:18:13.743 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 bash[68094]: 2e91614dea602b93596bc2ef1d4cdb67367a59368b1c75266765f703883c1563 2024-04-24T08:18:13.743 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:18:13 smithi146 systemd[1]: Started Ceph mds.j for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:13.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:13 smithi177 ceph-mon[33046]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:13.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:13 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4230371464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:13 smithi153 ceph-mon[32746]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:13 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4230371464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6834/2947430860,v1:172.21.15.146:6835/2947430860] up:boot 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: fsmap 3 up:standby 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:14.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6834/2947430860,v1:172.21.15.146:6835/2947430860] up:boot 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: fsmap 3 up:standby 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:14.831 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:14.832 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6834/2947430860,v1:172.21.15.146:6835/2947430860] up:boot 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: fsmap 3 up:standby 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:15.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 -- 172.21.15.146:0/2816309191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24109670 msgr2=0x7fee24109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 --2- 172.21.15.146:0/2816309191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24109670 0x7fee24109af0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009900 tx=0x7fee0c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 -- 172.21.15.146:0/2816309191 shutdown_connections 2024-04-24T08:18:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 --2- 172.21.15.146:0/2816309191 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee2410a030 0x7fee241108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 --2- 172.21.15.146:0/2816309191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24109670 0x7fee24109af0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 --2- 172.21.15.146:0/2816309191 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee24108480 0x7fee241088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 -- 172.21.15.146:0/2816309191 >> 172.21.15.146:0/2816309191 conn(0x7fee24077440 msgr2=0x7fee24079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 -- 172.21.15.146:0/2816309191 shutdown_connections 2024-04-24T08:18:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.282+0000 7fee29200640 1 -- 172.21.15.146:0/2816309191 wait complete. 2024-04-24T08:18:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 Processor -- start 2024-04-24T08:18:15.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 -- start start 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24108480 0x7fee2419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee24109670 0x7fee2419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2410a030 0x7fee241a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fee24114110 con 0x7fee24108480 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee24113f90 con 0x7fee24109670 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee29200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee24114290 con 0x7fee2410a030 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee23400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24108480 0x7fee2419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.284+0000 7fee23400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24108480 0x7fee2419e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49374/0 (socket says 172.21.15.146:49374) 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23400640 1 -- 172.21.15.146:0/2384426584 learned_addr learned my addr 172.21.15.146:0/2384426584 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23e00640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2410a030 0x7fee241a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee1aa00640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee24109670 0x7fee2419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23400640 1 -- 172.21.15.146:0/2384426584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2410a030 msgr2=0x7fee241a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23400640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2410a030 0x7fee241a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23400640 1 -- 172.21.15.146:0/2384426584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee24109670 msgr2=0x7fee2419ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23400640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee24109670 0x7fee2419ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23400640 1 -- 172.21.15.146:0/2384426584 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee241a63e0 con 0x7fee24108480 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee23400640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24108480 0x7fee2419e7c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fee1400eb70 tx=0x7fee1400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.285+0000 7fee20c00640 1 -- 172.21.15.146:0/2384426584 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14003b70 con 0x7fee24108480 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.286+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee241a66d0 con 0x7fee24108480 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.286+0000 7fee20c00640 1 -- 172.21.15.146:0/2384426584 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1400ee70 con 0x7fee24108480 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.286+0000 7fee20c00640 1 -- 172.21.15.146:0/2384426584 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee140107d0 con 0x7fee24108480 2024-04-24T08:18:15.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.286+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee241a6ba0 con 0x7fee24108480 2024-04-24T08:18:15.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.287+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2410f820 con 0x7fee24108480 2024-04-24T08:18:15.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.289+0000 7fee20c00640 1 -- 172.21.15.146:0/2384426584 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fee14004000 con 0x7fee24108480 2024-04-24T08:18:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.290+0000 7fee20c00640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fee080a3bb0 0x7fee080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.291+0000 7fee20c00640 1 -- 172.21.15.146:0/2384426584 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fee140d07c0 con 0x7fee24108480 2024-04-24T08:18:15.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.291+0000 7fee1aa00640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fee080a3bb0 0x7fee080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.291+0000 7fee1aa00640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fee080a3bb0 0x7fee080a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0314e0 tx=0x7fee0c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.295+0000 7fee20c00640 1 -- 172.21.15.146:0/2384426584 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee1409d5a0 con 0x7fee24108480 2024-04-24T08:18:15.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.535+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee2402d620 con 0x7fee24108480 2024-04-24T08:18:15.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.536+0000 7fee20c00640 1 -- 172.21.15.146:0/2384426584 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7fee1409cea0 con 0x7fee24108480 2024-04-24T08:18:15.538 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:15.538 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-04-24T08:18:15.538 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 4 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fee080a3bb0 msgr2=0x7fee080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fee080a3bb0 0x7fee080a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0314e0 tx=0x7fee0c000bc0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24108480 msgr2=0x7fee2419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24108480 0x7fee2419e7c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fee1400eb70 tx=0x7fee1400c6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 shutdown_connections 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee2410a030 0x7fee241a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fee080a3bb0 0x7fee080a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee24109670 0x7fee2419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 --2- 172.21.15.146:0/2384426584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee24108480 0x7fee2419e7c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 >> 172.21.15.146:0/2384426584 conn(0x7fee24077440 msgr2=0x7fee2410c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 shutdown_connections 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:15.540+0000 7fee29200640 1 -- 172.21.15.146:0/2384426584 wait complete. 2024-04-24T08:18:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:15 smithi177 ceph-mon[33046]: Deploying daemon mds.d on smithi146 2024-04-24T08:18:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:15 smithi177 ceph-mon[33046]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:15.827 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:15 smithi146 ceph-mon[27805]: Deploying daemon mds.d on smithi146 2024-04-24T08:18:15.827 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:15 smithi146 ceph-mon[27805]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:15.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:15 smithi153 ceph-mon[32746]: Deploying daemon mds.d on smithi146 2024-04-24T08:18:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:15 smithi153 ceph-mon[32746]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:16.655 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:16 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2384426584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:16.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:16 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2384426584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:16.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:16 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2384426584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:16.993 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:16 smithi146 systemd[1]: Starting Ceph mds.d for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:17.307 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 podman[68451]: 2024-04-24 08:18:16.916758243 +0000 UTC m=+0.039516282 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:17.307 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 podman[68451]: 2024-04-24 08:18:17.011741083 +0000 UTC m=+0.134499140 container create 6d99d3073189ad03a5d3d02daed44eb48f2bf547a7eaa3a8abaa54d4d48e5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:18:17.307 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 podman[68451]: 2024-04-24 08:18:17.249115789 +0000 UTC m=+0.371873886 container init 6d99d3073189ad03a5d3d02daed44eb48f2bf547a7eaa3a8abaa54d4d48e5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T08:18:17.308 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 podman[68451]: 2024-04-24 08:18:17.259330091 +0000 UTC m=+0.382088154 container start 6d99d3073189ad03a5d3d02daed44eb48f2bf547a7eaa3a8abaa54d4d48e5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-24T08:18:17.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:17.634 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mds[68470]: -- 172.21.15.146:0/498606946 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e86c3ada40 con 0x55e86c380800 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d[68466]: starting mds.d at 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 bash[68451]: 6d99d3073189ad03a5d3d02daed44eb48f2bf547a7eaa3a8abaa54d4d48e5388 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:18:17 smithi146 systemd[1]: Started Ceph mds.d for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mon[27805]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:17.635 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:17 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 -- 172.21.15.146:0/429454869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40769c0 msgr2=0x7fafb4076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 --2- 172.21.15.146:0/429454869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40769c0 0x7fafb4076de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009900 tx=0x7fafa802a410 comp rx=0 tx=0).stop 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 -- 172.21.15.146:0/429454869 shutdown_connections 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 --2- 172.21.15.146:0/429454869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40785a0 0x7fafb4078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 --2- 172.21.15.146:0/429454869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafb4077c00 0x7fafb4078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 --2- 172.21.15.146:0/429454869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40769c0 0x7fafb4076de0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 -- 172.21.15.146:0/429454869 >> 172.21.15.146:0/429454869 conn(0x7fafb40ff610 msgr2=0x7fafb4101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 -- 172.21.15.146:0/429454869 shutdown_connections 2024-04-24T08:18:17.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.770+0000 7fafba400640 1 -- 172.21.15.146:0/429454869 wait complete. 2024-04-24T08:18:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.771+0000 7fafba400640 1 Processor -- start 2024-04-24T08:18:17.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafba400640 1 -- start start 2024-04-24T08:18:17.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafba400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40769c0 0x7fafb419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafba400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafb4077c00 0x7fafb419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafba400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40785a0 0x7fafb41a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafba400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fafb4114110 con 0x7fafb4077c00 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafba400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafb4113f90 con 0x7fafb40785a0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafba400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafb4114290 con 0x7fafb40769c0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafb9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40769c0 0x7fafb419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafb3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafb4077c00 0x7fafb419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafb9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40769c0 0x7fafb419e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:35700/0 (socket says 172.21.15.146:35700) 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafb9000640 1 -- 172.21.15.146:0/3942998172 learned_addr learned my addr 172.21.15.146:0/3942998172 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.772+0000 7fafb9a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40785a0 0x7fafb41a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb9000640 1 -- 172.21.15.146:0/3942998172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40785a0 msgr2=0x7fafb41a5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb9000640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40785a0 0x7fafb41a5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb9000640 1 -- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafb4077c00 msgr2=0x7fafb419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb9000640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafb4077c00 0x7fafb419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb9000640 1 -- 172.21.15.146:0/3942998172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb41a6350 con 0x7fafb40769c0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb9a00640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40785a0 0x7fafb41a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb9000640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40769c0 0x7fafb419e730 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009230 tx=0x7fafa80042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb1600640 1 -- 172.21.15.146:0/3942998172 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa803f030 con 0x7fafb40769c0 2024-04-24T08:18:17.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb41a65e0 con 0x7fafb40769c0 2024-04-24T08:18:17.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb1600640 1 -- 172.21.15.146:0/3942998172 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafa8003ce0 con 0x7fafb40769c0 2024-04-24T08:18:17.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.773+0000 7fafb1600640 1 -- 172.21.15.146:0/3942998172 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8035070 con 0x7fafb40769c0 2024-04-24T08:18:17.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.774+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb41a6b20 con 0x7fafb40769c0 2024-04-24T08:18:17.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.774+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb4110b80 con 0x7fafb40769c0 2024-04-24T08:18:17.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.777+0000 7fafb1600640 1 -- 172.21.15.146:0/3942998172 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fafa8042030 con 0x7fafb40769c0 2024-04-24T08:18:17.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.778+0000 7fafb1600640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faf9c0a3bb0 0x7faf9c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.778+0000 7fafb1600640 1 -- 172.21.15.146:0/3942998172 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fafa80f00f0 con 0x7fafb40769c0 2024-04-24T08:18:17.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.778+0000 7fafb3e00640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faf9c0a3bb0 0x7faf9c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.779+0000 7fafb3e00640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faf9c0a3bb0 0x7faf9c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fafb419fc40 tx=0x7fafa0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:17.783+0000 7fafb1600640 1 -- 172.21.15.146:0/3942998172 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafa80bc880 con 0x7fafb40769c0 2024-04-24T08:18:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:17 smithi177 ceph-mon[33046]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:17 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:17 smithi153 ceph-mon[32746]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:17 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:18.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.027+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafb402d620 con 0x7fafb40769c0 2024-04-24T08:18:18.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.028+0000 7fafb1600640 1 -- 172.21.15.146:0/3942998172 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7fafa80c0b80 con 0x7fafb40769c0 2024-04-24T08:18:18.029 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:18.029 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-24T08:18:18.029 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 5 2024-04-24T08:18:18.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faf9c0a3bb0 msgr2=0x7faf9c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:18.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faf9c0a3bb0 0x7faf9c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fafb419fc40 tx=0x7fafa0005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:18.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40769c0 msgr2=0x7fafb419e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:18.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40769c0 0x7fafb419e730 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009230 tx=0x7fafa80042f0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 shutdown_connections 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7faf9c0a3bb0 0x7faf9c0a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafb40785a0 0x7fafb41a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafb4077c00 0x7fafb419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 --2- 172.21.15.146:0/3942998172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafb40769c0 0x7fafb419e730 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 >> 172.21.15.146:0/3942998172 conn(0x7fafb40ff610 msgr2=0x7fafb4100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 shutdown_connections 2024-04-24T08:18:18.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:18.031+0000 7fafba400640 1 -- 172.21.15.146:0/3942998172 wait complete. 2024-04-24T08:18:18.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:18 smithi153 ceph-mon[32746]: Deploying daemon mds.e on smithi153 2024-04-24T08:18:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:18 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6836/1079190046,v1:172.21.15.146:6837/1079190046] up:boot 2024-04-24T08:18:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:18 smithi153 ceph-mon[32746]: fsmap 4 up:standby 2024-04-24T08:18:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:18 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:18:18.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:18 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3942998172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:18 smithi177 ceph-mon[33046]: Deploying daemon mds.e on smithi153 2024-04-24T08:18:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:18 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6836/1079190046,v1:172.21.15.146:6837/1079190046] up:boot 2024-04-24T08:18:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:18 smithi177 ceph-mon[33046]: fsmap 4 up:standby 2024-04-24T08:18:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:18 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:18:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:18 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3942998172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:18.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:18 smithi146 ceph-mon[27805]: Deploying daemon mds.e on smithi153 2024-04-24T08:18:18.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:18 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6836/1079190046,v1:172.21.15.146:6837/1079190046] up:boot 2024-04-24T08:18:18.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:18 smithi146 ceph-mon[27805]: fsmap 4 up:standby 2024-04-24T08:18:18.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:18 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:18:18.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:18 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3942998172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:19.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:19 smithi177 ceph-mon[33046]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:19.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:19 smithi153 ceph-mon[32746]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:19.914 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:19 smithi153 systemd[1]: Starting Ceph mds.e for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:19.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:19 smithi146 ceph-mon[27805]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:20.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:20.179 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:19 smithi153 podman[62149]: 2024-04-24 08:18:19.816328437 +0000 UTC m=+0.018401579 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:20.179 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:19 smithi153 podman[62149]: 2024-04-24 08:18:19.926087497 +0000 UTC m=+0.128160631 container create aee38ccaeee3f5b18b8aba7d7630ba2026487ca9c13204030902ee4f3c5deaec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424) 2024-04-24T08:18:20.179 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:20 smithi153 podman[62149]: 2024-04-24 08:18:20.15326651 +0000 UTC m=+0.355339659 container init aee38ccaeee3f5b18b8aba7d7630ba2026487ca9c13204030902ee4f3c5deaec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-04-24T08:18:20.179 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:20 smithi153 podman[62149]: 2024-04-24 08:18:20.155796195 +0000 UTC m=+0.357869335 container start aee38ccaeee3f5b18b8aba7d7630ba2026487ca9c13204030902ee4f3c5deaec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 -- 172.21.15.146:0/1411381874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c109670 msgr2=0x7f8c0c109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 --2- 172.21.15.146:0/1411381874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c109670 0x7f8c0c109af0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009900 tx=0x7f8bf802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 -- 172.21.15.146:0/1411381874 shutdown_connections 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 --2- 172.21.15.146:0/1411381874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c0c10a030 0x7f8c0c1108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 --2- 172.21.15.146:0/1411381874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c109670 0x7f8c0c109af0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 --2- 172.21.15.146:0/1411381874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c0c108480 0x7f8c0c1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 -- 172.21.15.146:0/1411381874 >> 172.21.15.146:0/1411381874 conn(0x7f8c0c077440 msgr2=0x7f8c0c079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 -- 172.21.15.146:0/1411381874 shutdown_connections 2024-04-24T08:18:20.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.266+0000 7f8c0be00640 1 -- 172.21.15.146:0/1411381874 wait complete. 2024-04-24T08:18:20.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.267+0000 7f8c0be00640 1 Processor -- start 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.267+0000 7f8c0be00640 1 -- start start 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c108480 0x7f8c0c19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c0c109670 0x7f8c0c19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c0c10a030 0x7f8c0c1a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c114160 con 0x7f8c0c10a030 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0be00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c113fe0 con 0x7f8c0c108480 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c1142e0 con 0x7f8c0c109670 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c108480 0x7f8c0c19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c108480 0x7f8c0c19e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:40236/0 (socket says 172.21.15.146:40236) 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 -- 172.21.15.146:0/2909330390 learned_addr learned my addr 172.21.15.146:0/2909330390 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0a000640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c0c109670 0x7f8c0c19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0b400640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c0c10a030 0x7f8c0c1a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 -- 172.21.15.146:0/2909330390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c0c109670 msgr2=0x7f8c0c19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c0c109670 0x7f8c0c19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 -- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c0c10a030 msgr2=0x7f8c0c1a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c0c10a030 0x7f8c0c1a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0aa00640 1 -- 172.21.15.146:0/2909330390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0c1a6380 con 0x7f8c0c108480 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.268+0000 7f8c0b400640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c0c10a030 0x7f8c0c1a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.269+0000 7f8c0aa00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c108480 0x7f8c0c19e850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00e9c0 tx=0x7f8bfc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.269+0000 7f8c10a00640 1 -- 172.21.15.146:0/2909330390 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc00cd80 con 0x7f8c0c108480 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.269+0000 7f8c10a00640 1 -- 172.21.15.146:0/2909330390 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bfc0035f0 con 0x7f8c0c108480 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.269+0000 7f8c10a00640 1 -- 172.21.15.146:0/2909330390 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc010690 con 0x7f8c0c108480 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.269+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0c1a6670 con 0x7f8c0c108480 2024-04-24T08:18:20.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.269+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c0c1a6b40 con 0x7f8c0c108480 2024-04-24T08:18:20.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.270+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c0c10f740 con 0x7f8c0c108480 2024-04-24T08:18:20.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.272+0000 7f8c10a00640 1 -- 172.21.15.146:0/2909330390 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f8bfc0037a0 con 0x7f8c0c108480 2024-04-24T08:18:20.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.273+0000 7f8c10a00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8bf00a3c30 0x7f8bf00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.274+0000 7f8c10a00640 1 -- 172.21.15.146:0/2909330390 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f8bfc0d0310 con 0x7f8c0c108480 2024-04-24T08:18:20.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.274+0000 7f8c0a000640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8bf00a3c30 0x7f8bf00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.275+0000 7f8c0a000640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8bf00a3c30 0x7f8bf00a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8031440 tx=0x7f8bf80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.278+0000 7f8c10a00640 1 -- 172.21.15.146:0/2909330390 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8bfc09d170 con 0x7f8c0c108480 2024-04-24T08:18:20.460 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:20 smithi153 ceph-mds[62168]: -- 172.21.15.153:0/1975426676 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c053179a40 con 0x55c05314c800 2024-04-24T08:18:20.461 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:20 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e[62164]: starting mds.e at 2024-04-24T08:18:20.461 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:20 smithi153 bash[62149]: aee38ccaeee3f5b18b8aba7d7630ba2026487ca9c13204030902ee4f3c5deaec 2024-04-24T08:18:20.461 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:18:20 smithi153 systemd[1]: Started Ceph mds.e for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.533+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c0c02d8d0 con 0x7f8c0c108480 2024-04-24T08:18:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.534+0000 7f8c10a00640 1 -- 172.21.15.146:0/2909330390 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7f8bfc09ca70 con 0x7f8c0c108480 2024-04-24T08:18:20.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:20.535 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-24T08:18:20.535 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 5 2024-04-24T08:18:20.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.536+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8bf00a3c30 msgr2=0x7f8bf00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.536+0000 7f8c0be00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8bf00a3c30 0x7f8bf00a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8031440 tx=0x7f8bf80315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c108480 msgr2=0x7f8c0c19e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c108480 0x7f8c0c19e850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00e9c0 tx=0x7f8bfc00ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 shutdown_connections 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c0c109670 0x7f8c0c19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8bf00a3c30 0x7f8bf00a60f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c0c10a030 0x7f8c0c1a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 --2- 172.21.15.146:0/2909330390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c0c108480 0x7f8c0c19e850 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 >> 172.21.15.146:0/2909330390 conn(0x7f8c0c077440 msgr2=0x7f8c0c10c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 shutdown_connections 2024-04-24T08:18:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:20.537+0000 7f8c0be00640 1 -- 172.21.15.146:0/2909330390 wait complete. 2024-04-24T08:18:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:21.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:21.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:21.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: Deploying daemon mds.f on smithi177 2024-04-24T08:18:21.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:21.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2909330390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:21.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.153:6834/3655889612,v1:172.21.15.153:6835/3655889612] up:boot 2024-04-24T08:18:21.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: fsmap 5 up:standby 2024-04-24T08:18:21.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:21 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: Deploying daemon mds.f on smithi177 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2909330390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.153:6834/3655889612,v1:172.21.15.153:6835/3655889612] up:boot 2024-04-24T08:18:21.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: fsmap 5 up:standby 2024-04-24T08:18:21.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:21 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:18:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: Deploying daemon mds.f on smithi177 2024-04-24T08:18:21.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:21.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2909330390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:21.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.153:6834/3655889612,v1:172.21.15.153:6835/3655889612] up:boot 2024-04-24T08:18:21.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: fsmap 5 up:standby 2024-04-24T08:18:21.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:21 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:18:22.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:22.565 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 systemd[1]: Starting Ceph mds.f for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 -- 172.21.15.146:0/858222115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c109d00 msgr2=0x7f7e5c10c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 --2- 172.21.15.146:0/858222115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c109d00 0x7f7e5c10c1a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7e54009ca0 tx=0x7f7e5402a950 comp rx=0 tx=0).stop 2024-04-24T08:18:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 -- 172.21.15.146:0/858222115 shutdown_connections 2024-04-24T08:18:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 --2- 172.21.15.146:0/858222115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 0x7f7e5c074e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 --2- 172.21.15.146:0/858222115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c109d00 0x7f7e5c10c1a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 --2- 172.21.15.146:0/858222115 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5c075fe0 0x7f7e5c074430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 -- 172.21.15.146:0/858222115 >> 172.21.15.146:0/858222115 conn(0x7f7e5c0fd3e0 msgr2=0x7f7e5c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 -- 172.21.15.146:0/858222115 shutdown_connections 2024-04-24T08:18:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.775+0000 7f7e62800640 1 -- 172.21.15.146:0/858222115 wait complete. 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 Processor -- start 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 -- start start 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 0x7f7e5c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c075fe0 0x7f7e5c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5c109d00 0x7f7e5c1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c10fd00 con 0x7f7e5c075fe0 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c10fb80 con 0x7f7e5c109d00 2024-04-24T08:18:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.776+0000 7f7e62800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c10fe80 con 0x7f7e5c074a00 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c075fe0 0x7f7e5c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e61400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 0x7f7e5c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c075fe0 0x7f7e5c19a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36442/0 (socket says 172.21.15.146:36442) 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e61400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 0x7f7e5c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39024/0 (socket says 172.21.15.146:39024) 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 -- 172.21.15.146:0/2594734087 learned_addr learned my addr 172.21.15.146:0/2594734087 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e61e00640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5c109d00 0x7f7e5c1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 -- 172.21.15.146:0/2594734087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 msgr2=0x7f7e5c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 0x7f7e5c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 -- 172.21.15.146:0/2594734087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5c109d00 msgr2=0x7f7e5c1a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5c109d00 0x7f7e5c1a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 -- 172.21.15.146:0/2594734087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5c1a1f10 con 0x7f7e5c075fe0 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e61400640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 0x7f7e5c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.777+0000 7f7e60a00640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c075fe0 0x7f7e5c19a830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800e9f0 tx=0x7f7e4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.778+0000 7f7e5a000640 1 -- 172.21.15.146:0/2594734087 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4800cd80 con 0x7f7e5c075fe0 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.778+0000 7f7e5a000640 1 -- 172.21.15.146:0/2594734087 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e48010910 con 0x7f7e5c075fe0 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.778+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5c1a2200 con 0x7f7e5c075fe0 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.778+0000 7f7e5a000640 1 -- 172.21.15.146:0/2594734087 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48018990 con 0x7f7e5c075fe0 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.778+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5c1a2750 con 0x7f7e5c075fe0 2024-04-24T08:18:22.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.779+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e5c108ac0 con 0x7f7e5c075fe0 2024-04-24T08:18:22.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.782+0000 7f7e5a000640 1 -- 172.21.15.146:0/2594734087 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7e48010ac0 con 0x7f7e5c075fe0 2024-04-24T08:18:22.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.783+0000 7f7e5a000640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e440a3bb0 0x7f7e440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.783+0000 7f7e5a000640 1 -- 172.21.15.146:0/2594734087 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f7e480d0320 con 0x7f7e5c075fe0 2024-04-24T08:18:22.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.784+0000 7f7e61400640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e440a3bb0 0x7f7e440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.784+0000 7f7e61400640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e440a3bb0 0x7f7e440a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7e50009fc0 tx=0x7f7e50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:22.788+0000 7f7e5a000640 1 -- 172.21.15.146:0/2594734087 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e4809cab0 con 0x7f7e5c075fe0 2024-04-24T08:18:22.949 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 podman[63814]: 2024-04-24 08:18:22.589636815 +0000 UTC m=+0.018112457 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:22.949 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 podman[63814]: 2024-04-24 08:18:22.712541516 +0000 UTC m=+0.141017143 container create 5c73c36cb987e2f6f56a477f577c139c3eff894c39ca727d0fd0232d89a77fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-24T08:18:22.949 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 podman[63814]: 2024-04-24 08:18:22.923220876 +0000 UTC m=+0.351696504 container init 5c73c36cb987e2f6f56a477f577c139c3eff894c39ca727d0fd0232d89a77fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-24T08:18:22.950 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 podman[63814]: 2024-04-24 08:18:22.925796809 +0000 UTC m=+0.354272440 container start 5c73c36cb987e2f6f56a477f577c139c3eff894c39ca727d0fd0232d89a77fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-24T08:18:23.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.036+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e5c062c00 con 0x7f7e5c075fe0 2024-04-24T08:18:23.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.037+0000 7f7e5a000640 1 -- 172.21.15.146:0/2594734087 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4708 (secure 0 0 0) 0x7f7e480a0db0 con 0x7f7e5c075fe0 2024-04-24T08:18:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:23.038 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-24T08:18:23.038 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 6 2024-04-24T08:18:23.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e440a3bb0 msgr2=0x7f7e440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e440a3bb0 0x7f7e440a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7e50009fc0 tx=0x7f7e50005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c075fe0 msgr2=0x7f7e5c19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c075fe0 0x7f7e5c19a830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800e9f0 tx=0x7f7e4800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 shutdown_connections 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7e440a3bb0 0x7f7e440a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e5c109d00 0x7f7e5c1a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5c075fe0 0x7f7e5c19a830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 --2- 172.21.15.146:0/2594734087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e5c074a00 0x7f7e5c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 >> 172.21.15.146:0/2594734087 conn(0x7f7e5c0fd3e0 msgr2=0x7f7e5c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 shutdown_connections 2024-04-24T08:18:23.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:23.040+0000 7f7e62800640 1 -- 172.21.15.146:0/2594734087 wait complete. 2024-04-24T08:18:23.315 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 ceph-mds[63833]: -- 172.21.15.177:0/3172935192 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d36279a40 con 0x558d3624cc00 2024-04-24T08:18:23.315 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f[63829]: starting mds.f at 2024-04-24T08:18:23.315 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:22 smithi177 bash[63814]: 5c73c36cb987e2f6f56a477f577c139c3eff894c39ca727d0fd0232d89a77fc8 2024-04-24T08:18:23.315 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:18:23 smithi177 systemd[1]: Started Ceph mds.f for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2594734087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:23.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:23.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:23.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:23 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2594734087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:23 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:23.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:23.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2594734087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:23.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:23.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:23.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:23.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:23 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:24.788 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:24 smithi146 ceph-mon[27805]: Deploying daemon mds.c on smithi177 2024-04-24T08:18:24.788 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:24 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.177:6834/2429639426,v1:172.21.15.177:6835/2429639426] up:boot 2024-04-24T08:18:24.788 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:24 smithi146 ceph-mon[27805]: fsmap 6 up:standby 2024-04-24T08:18:24.788 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:24 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:18:24.789 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:24 smithi177 ceph-mon[33046]: Deploying daemon mds.c on smithi177 2024-04-24T08:18:24.789 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:24 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.177:6834/2429639426,v1:172.21.15.177:6835/2429639426] up:boot 2024-04-24T08:18:24.790 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:24 smithi177 ceph-mon[33046]: fsmap 6 up:standby 2024-04-24T08:18:24.790 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:24 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:18:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:24 smithi153 ceph-mon[32746]: Deploying daemon mds.c on smithi177 2024-04-24T08:18:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:24 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.177:6834/2429639426,v1:172.21.15.177:6835/2429639426] up:boot 2024-04-24T08:18:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:24 smithi153 ceph-mon[32746]: fsmap 6 up:standby 2024-04-24T08:18:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:24 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:18:25.061 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 -- 172.21.15.146:0/3894527403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca8077c00 msgr2=0x7fcca8078060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 --2- 172.21.15.146:0/3894527403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca8077c00 0x7fcca8078060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc94009900 tx=0x7fcc9402a410 comp rx=0 tx=0).stop 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 -- 172.21.15.146:0/3894527403 shutdown_connections 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 --2- 172.21.15.146:0/3894527403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80785a0 0x7fcca8078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 --2- 172.21.15.146:0/3894527403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca8077c00 0x7fcca8078060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 --2- 172.21.15.146:0/3894527403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcca80769c0 0x7fcca8076de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 -- 172.21.15.146:0/3894527403 >> 172.21.15.146:0/3894527403 conn(0x7fcca80ff610 msgr2=0x7fcca8101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 -- 172.21.15.146:0/3894527403 shutdown_connections 2024-04-24T08:18:25.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.272+0000 7fcca7e00640 1 -- 172.21.15.146:0/3894527403 wait complete. 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.273+0000 7fcca7e00640 1 Processor -- start 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.273+0000 7fcca7e00640 1 -- start start 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80769c0 0x7fcca819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcca8077c00 0x7fcca819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80785a0 0x7fcca81a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcca8114150 con 0x7fcca80769c0 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcca8113fd0 con 0x7fcca80785a0 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcca81142d0 con 0x7fcca8077c00 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80769c0 0x7fcca819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80769c0 0x7fcca819e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36456/0 (socket says 172.21.15.146:36456) 2024-04-24T08:18:25.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 -- 172.21.15.146:0/2306262474 learned_addr learned my addr 172.21.15.146:0/2306262474 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6000640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcca8077c00 0x7fcca819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7400640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80785a0 0x7fcca81a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 -- 172.21.15.146:0/2306262474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcca8077c00 msgr2=0x7fcca819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcca8077c00 0x7fcca819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 -- 172.21.15.146:0/2306262474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80785a0 msgr2=0x7fcca81a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80785a0 0x7fcca81a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6a00640 1 -- 172.21.15.146:0/2306262474 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca81a63c0 con 0x7fcca80769c0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca6000640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcca8077c00 0x7fcca819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.274+0000 7fcca7400640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80785a0 0x7fcca81a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.275+0000 7fcca6a00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80769c0 0x7fcca819e7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800e9f0 tx=0x7fcc9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.275+0000 7fcc9f400640 1 -- 172.21.15.146:0/2306262474 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9800cd80 con 0x7fcca80769c0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.275+0000 7fcc9f400640 1 -- 172.21.15.146:0/2306262474 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc98005500 con 0x7fcca80769c0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.275+0000 7fcc9f400640 1 -- 172.21.15.146:0/2306262474 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc98010690 con 0x7fcca80769c0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.275+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca81a66b0 con 0x7fcca80769c0 2024-04-24T08:18:25.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.275+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca81a6bd0 con 0x7fcca80769c0 2024-04-24T08:18:25.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.276+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca8110b20 con 0x7fcca80769c0 2024-04-24T08:18:25.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.279+0000 7fcc9f400640 1 -- 172.21.15.146:0/2306262474 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fcc980046c0 con 0x7fcca80769c0 2024-04-24T08:18:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.280+0000 7fcc9f400640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcc8c0a3c30 0x7fcc8c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.281+0000 7fcc9f400640 1 -- 172.21.15.146:0/2306262474 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fcc980d0580 con 0x7fcca80769c0 2024-04-24T08:18:25.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.281+0000 7fcca6000640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcc8c0a3c30 0x7fcc8c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:25.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.282+0000 7fcca6000640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcc8c0a3c30 0x7fcc8c0a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc94031440 tx=0x7fcc940315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:25.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.285+0000 7fcc9f400640 1 -- 172.21.15.146:0/2306262474 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc98053ff0 con 0x7fcca80769c0 2024-04-24T08:18:25.483 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 systemd[1]: Starting Ceph mds.c for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:25.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.531+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcca802d620 con 0x7fcca80769c0 2024-04-24T08:18:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.533+0000 7fcc9f400640 1 -- 172.21.15.146:0/2306262474 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5536 (secure 0 0 0) 0x7fcc980bbe20 con 0x7fcca80769c0 2024-04-24T08:18:25.534 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:25.534 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-24T08:18:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 7 2024-04-24T08:18:25.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcc8c0a3c30 msgr2=0x7fcc8c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:25.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcc8c0a3c30 0x7fcc8c0a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc94031440 tx=0x7fcc940315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80769c0 msgr2=0x7fcca819e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:25.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80769c0 0x7fcca819e7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcc9800e9f0 tx=0x7fcc9800edf0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 shutdown_connections 2024-04-24T08:18:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcca8077c00 0x7fcca819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcc8c0a3c30 0x7fcc8c0a60f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcca80785a0 0x7fcca81a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.535+0000 7fcca7e00640 1 --2- 172.21.15.146:0/2306262474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcca80769c0 0x7fcca819e7a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.536+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 >> 172.21.15.146:0/2306262474 conn(0x7fcca80ff610 msgr2=0x7fcca8100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.536+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 shutdown_connections 2024-04-24T08:18:25.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:25.536+0000 7fcca7e00640 1 -- 172.21.15.146:0/2306262474 wait complete. 2024-04-24T08:18:25.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:25 smithi177 ceph-mon[33046]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:25.745 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 podman[64119]: 2024-04-24 08:18:25.385127959 +0000 UTC m=+0.017906674 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:25.745 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 podman[64119]: 2024-04-24 08:18:25.482733895 +0000 UTC m=+0.115512595 container create 6f268dd544ed84372ded2e8d685f1ba2795d507af43af90785ab09bfb388a051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:18:25.745 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 podman[64119]: 2024-04-24 08:18:25.718845657 +0000 UTC m=+0.351624350 container init 6f268dd544ed84372ded2e8d685f1ba2795d507af43af90785ab09bfb388a051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-24T08:18:25.745 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 podman[64119]: 2024-04-24 08:18:25.721461443 +0000 UTC m=+0.354240147 container start 6f268dd544ed84372ded2e8d685f1ba2795d507af43af90785ab09bfb388a051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:18:25.745 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 ceph-mds[64138]: -- 172.21.15.177:0/613465087 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed341f1a40 con 0x55ed341c4c00 2024-04-24T08:18:25.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:25 smithi153 ceph-mon[32746]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:25.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:25 smithi146 ceph-mon[27805]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:26.065 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c[64134]: starting mds.c at 2024-04-24T08:18:26.065 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 bash[64119]: 6f268dd544ed84372ded2e8d685f1ba2795d507af43af90785ab09bfb388a051 2024-04-24T08:18:26.065 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:18:25 smithi177 systemd[1]: Started Ceph mds.c for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:26 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2306262474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:26 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:26.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:26 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2306262474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:26.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:26.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:26.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:26 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:26 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2306262474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:26 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:27.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:27.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 -- 172.21.15.146:0/721732559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c108660 msgr2=0x7f179c108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:27.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 --2- 172.21.15.146:0/721732559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c108660 0x7f179c108a80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f178c009900 tx=0x7f178c02a400 comp rx=0 tx=0).stop 2024-04-24T08:18:27.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 -- 172.21.15.146:0/721732559 shutdown_connections 2024-04-24T08:18:27.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 --2- 172.21.15.146:0/721732559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c109850 0x7f179c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:27.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 --2- 172.21.15.146:0/721732559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c10a210 0x7f179c110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:27.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 --2- 172.21.15.146:0/721732559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c108660 0x7f179c108a80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:27.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 -- 172.21.15.146:0/721732559 >> 172.21.15.146:0/721732559 conn(0x7f179c077620 msgr2=0x7f179c079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:27.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 -- 172.21.15.146:0/721732559 shutdown_connections 2024-04-24T08:18:27.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.772+0000 7f17a1c00640 1 -- 172.21.15.146:0/721732559 wait complete. 2024-04-24T08:18:27.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.773+0000 7f17a1c00640 1 Processor -- start 2024-04-24T08:18:27.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f17a1c00640 1 -- start start 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f17a1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c108660 0x7f179c19e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f17a1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c109850 0x7f179c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f17a1c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c10a210 0x7f179c1a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f17a1c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f179c1142e0 con 0x7f179c108660 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f17a1c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f179c114160 con 0x7f179c10a210 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f17a1c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f179c114460 con 0x7f179c109850 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f179aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c108660 0x7f179c19e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f179aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c108660 0x7f179c19e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36480/0 (socket says 172.21.15.146:36480) 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f179aa00640 1 -- 172.21.15.146:0/2059375925 learned_addr learned my addr 172.21.15.146:0/2059375925 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f179a000640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c109850 0x7f179c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.774+0000 7f179b400640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c10a210 0x7f179c1a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:27.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f179aa00640 1 -- 172.21.15.146:0/2059375925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c109850 msgr2=0x7f179c19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f179aa00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c109850 0x7f179c19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f179aa00640 1 -- 172.21.15.146:0/2059375925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c10a210 msgr2=0x7f179c1a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f179aa00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c10a210 0x7f179c1a5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f179aa00640 1 -- 172.21.15.146:0/2059375925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c1a6600 con 0x7f179c108660 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f179a000640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c109850 0x7f179c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f179aa00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c108660 0x7f179c19e9e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f178c031610 tx=0x7f178c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f1793400640 1 -- 172.21.15.146:0/2059375925 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c03f030 con 0x7f179c108660 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f1793400640 1 -- 172.21.15.146:0/2059375925 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f178c003ad0 con 0x7f179c108660 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179c1a6890 con 0x7f179c108660 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f1793400640 1 -- 172.21.15.146:0/2059375925 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c035070 con 0x7f179c108660 2024-04-24T08:18:27.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.775+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179c1a6d80 con 0x7f179c108660 2024-04-24T08:18:27.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.777+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f179c10f9d0 con 0x7f179c108660 2024-04-24T08:18:27.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.778+0000 7f1793400640 1 -- 172.21.15.146:0/2059375925 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f178c042030 con 0x7f179c108660 2024-04-24T08:18:27.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.780+0000 7f1793400640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f17800a3bb0 0x7f17800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:27.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.781+0000 7f1793400640 1 -- 172.21.15.146:0/2059375925 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f178c0f0570 con 0x7f179c108660 2024-04-24T08:18:27.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.781+0000 7f179a000640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f17800a3bb0 0x7f17800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:27.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.781+0000 7f179a000640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f17800a3bb0 0x7f17800a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f179c19fef0 tx=0x7f1788005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:27.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:27.785+0000 7f1793400640 1 -- 172.21.15.146:0/2059375925 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f178c0bcd00 con 0x7f179c108660 2024-04-24T08:18:27.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:27 smithi153 ceph-mon[32746]: Deploying daemon mds.h on smithi153 2024-04-24T08:18:27.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:27 smithi153 ceph-mon[32746]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:27.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:27 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.177:6836/2495345664,v1:172.21.15.177:6837/2495345664] up:boot 2024-04-24T08:18:27.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:27 smithi153 ceph-mon[32746]: fsmap 7 up:standby 2024-04-24T08:18:27.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:27 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:18:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:27 smithi177 ceph-mon[33046]: Deploying daemon mds.h on smithi153 2024-04-24T08:18:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:27 smithi177 ceph-mon[33046]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:27 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.177:6836/2495345664,v1:172.21.15.177:6837/2495345664] up:boot 2024-04-24T08:18:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:27 smithi177 ceph-mon[33046]: fsmap 7 up:standby 2024-04-24T08:18:27.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:27 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:18:27.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:27 smithi146 ceph-mon[27805]: Deploying daemon mds.h on smithi153 2024-04-24T08:18:27.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:27 smithi146 ceph-mon[27805]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:27.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:27 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.177:6836/2495345664,v1:172.21.15.177:6837/2495345664] up:boot 2024-04-24T08:18:27.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:27 smithi146 ceph-mon[27805]: fsmap 7 up:standby 2024-04-24T08:18:27.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:27 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:18:28.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.031+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f179c02d620 con 0x7f179c108660 2024-04-24T08:18:28.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.033+0000 7f1793400640 1 -- 172.21.15.146:0/2059375925 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6364 (secure 0 0 0) 0x7f178c0c1000 con 0x7f179c108660 2024-04-24T08:18:28.034 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:28.034 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-24T08:18:28.034 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 8 2024-04-24T08:18:28.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f17800a3bb0 msgr2=0x7f17800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:28.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f17800a3bb0 0x7f17800a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f179c19fef0 tx=0x7f1788005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:28.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c108660 msgr2=0x7f179c19e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:28.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c108660 0x7f179c19e9e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f178c031610 tx=0x7f178c004270 comp rx=0 tx=0).stop 2024-04-24T08:18:28.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 shutdown_connections 2024-04-24T08:18:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f179c109850 0x7f179c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f17800a3bb0 0x7f17800a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f179c10a210 0x7f179c1a5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 --2- 172.21.15.146:0/2059375925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179c108660 0x7f179c19e9e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 >> 172.21.15.146:0/2059375925 conn(0x7f179c077620 msgr2=0x7f179c10cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 shutdown_connections 2024-04-24T08:18:28.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:28.036+0000 7f17a1c00640 1 -- 172.21.15.146:0/2059375925 wait complete. 2024-04-24T08:18:28.445 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 podman[62458]: 2024-04-24 08:18:28.320647838 +0000 UTC m=+0.018732031 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:28.445 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 podman[62458]: 2024-04-24 08:18:28.444418877 +0000 UTC m=+0.142503059 container create 8bacd1b36dfece32ec1194df24dee7e03ded91a0885aedc62fe35b99e6e32c84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T08:18:28.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:28 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2059375925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:28.707 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 podman[62458]: 2024-04-24 08:18:28.680462826 +0000 UTC m=+0.378547011 container init 8bacd1b36dfece32ec1194df24dee7e03ded91a0885aedc62fe35b99e6e32c84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-24T08:18:28.707 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 podman[62458]: 2024-04-24 08:18:28.683259874 +0000 UTC m=+0.381344056 container start 8bacd1b36dfece32ec1194df24dee7e03ded91a0885aedc62fe35b99e6e32c84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-24T08:18:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:28 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2059375925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:28.960 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 ceph-mds[62485]: -- 172.21.15.153:0/1957892507 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571477c3a40 con 0x557147796c00 2024-04-24T08:18:28.961 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h[62476]: starting mds.h at 2024-04-24T08:18:28.961 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 bash[62458]: 8bacd1b36dfece32ec1194df24dee7e03ded91a0885aedc62fe35b99e6e32c84 2024-04-24T08:18:28.961 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:18:28 smithi153 systemd[1]: Started Ceph mds.h for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:28.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:28 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2059375925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:29.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:29 smithi153 ceph-mon[32746]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:29.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:29 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:29 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:29 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:29 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:29.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:29 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:29.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:29 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:29 smithi177 ceph-mon[33046]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:29 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:29 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:29 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:29 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:29.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:29 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:29.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:29 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:29.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:29 smithi146 ceph-mon[27805]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:29.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:29 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:29 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:29 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:29.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:29 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:29.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:29 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:29.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:29 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:30.057 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:30.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.265+0000 7f04de000640 1 -- 172.21.15.146:0/485333178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d81019f0 msgr2=0x7f04d8101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.265+0000 7f04de000640 1 --2- 172.21.15.146:0/485333178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d81019f0 0x7f04d8101e70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009ca0 tx=0x7f04d002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.266+0000 7f04de000640 1 -- 172.21.15.146:0/485333178 shutdown_connections 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.266+0000 7f04de000640 1 --2- 172.21.15.146:0/485333178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d8101050 0x7f04d81014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.266+0000 7f04de000640 1 --2- 172.21.15.146:0/485333178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d81019f0 0x7f04d8101e70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.266+0000 7f04de000640 1 --2- 172.21.15.146:0/485333178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04d81006f0 0x7f04d8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.266+0000 7f04de000640 1 -- 172.21.15.146:0/485333178 >> 172.21.15.146:0/485333178 conn(0x7f04d80fc310 msgr2=0x7f04d80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.266+0000 7f04de000640 1 -- 172.21.15.146:0/485333178 shutdown_connections 2024-04-24T08:18:30.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.266+0000 7f04de000640 1 -- 172.21.15.146:0/485333178 wait complete. 2024-04-24T08:18:30.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 Processor -- start 2024-04-24T08:18:30.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 -- start start 2024-04-24T08:18:30.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04d81006f0 0x7f04d819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:30.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d8101050 0x7f04d819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d81019f0 0x7f04d81a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f04d810fd70 con 0x7f04d8101050 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f04d810fbf0 con 0x7f04d81006f0 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04de000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04d810fef0 con 0x7f04d81019f0 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04d7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d8101050 0x7f04d819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04dcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04d81006f0 0x7f04d819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04d7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d8101050 0x7f04d819a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36498/0 (socket says 172.21.15.146:36498) 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04d7e00640 1 -- 172.21.15.146:0/1671979646 learned_addr learned my addr 172.21.15.146:0/1671979646 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.267+0000 7f04dd600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d81019f0 0x7f04d81a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d7e00640 1 -- 172.21.15.146:0/1671979646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d81019f0 msgr2=0x7f04d81a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:30.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d7e00640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d81019f0 0x7f04d81a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d7e00640 1 -- 172.21.15.146:0/1671979646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04d81006f0 msgr2=0x7f04d819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d7e00640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04d81006f0 0x7f04d819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d7e00640 1 -- 172.21.15.146:0/1671979646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d81a1f60 con 0x7f04d8101050 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d7e00640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d8101050 0x7f04d819a880 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f04c400e9f0 tx=0x7f04c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d5600640 1 -- 172.21.15.146:0/1671979646 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c400cd80 con 0x7f04d8101050 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d5600640 1 -- 172.21.15.146:0/1671979646 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04c4005500 con 0x7f04d8101050 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d81a2250 con 0x7f04d8101050 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04d5600640 1 -- 172.21.15.146:0/1671979646 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4010690 con 0x7f04d8101050 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.268+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d81a27a0 con 0x7f04d8101050 2024-04-24T08:18:30.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.270+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d810b580 con 0x7f04d8101050 2024-04-24T08:18:30.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.272+0000 7f04d5600640 1 -- 172.21.15.146:0/1671979646 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f04c40046c0 con 0x7f04d8101050 2024-04-24T08:18:30.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.274+0000 7f04d5600640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f04c00a3c30 0x7f04c00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:30.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.274+0000 7f04d5600640 1 -- 172.21.15.146:0/1671979646 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f04c40d0310 con 0x7f04d8101050 2024-04-24T08:18:30.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.274+0000 7f04dcc00640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f04c00a3c30 0x7f04c00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:30.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.275+0000 7f04dcc00640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f04c00a3c30 0x7f04c00a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f04cc005d00 tx=0x7f04cc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:30.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.278+0000 7f04d5600640 1 -- 172.21.15.146:0/1671979646 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04c409d0f0 con 0x7f04d8101050 2024-04-24T08:18:30.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.520+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04d802d620 con 0x7f04d8101050 2024-04-24T08:18:30.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.521+0000 7f04d5600640 1 -- 172.21.15.146:0/1671979646 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7189 (secure 0 0 0) 0x7f04c409c9f0 con 0x7f04d8101050 2024-04-24T08:18:30.522 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:30.522 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-24T08:18:30.522 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 9 2024-04-24T08:18:30.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.523+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f04c00a3c30 msgr2=0x7f04c00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:30.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.523+0000 7f04de000640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f04c00a3c30 0x7f04c00a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f04cc005d00 tx=0x7f04cc00b410 comp rx=0 tx=0).stop 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.523+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d8101050 msgr2=0x7f04d819a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.523+0000 7f04de000640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d8101050 0x7f04d819a880 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f04c400e9f0 tx=0x7f04c400edf0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 shutdown_connections 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d81019f0 0x7f04d81a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f04c00a3c30 0x7f04c00a60f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04d8101050 0x7f04d819a880 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 --2- 172.21.15.146:0/1671979646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04d81006f0 0x7f04d819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 >> 172.21.15.146:0/1671979646 conn(0x7f04d80fc310 msgr2=0x7f04d80fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 shutdown_connections 2024-04-24T08:18:30.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:30.524+0000 7f04de000640 1 -- 172.21.15.146:0/1671979646 wait complete. 2024-04-24T08:18:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:30 smithi177 ceph-mon[33046]: Deploying daemon mds.i on smithi177 2024-04-24T08:18:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:30 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.153:6836/565470680,v1:172.21.15.153:6837/565470680] up:boot 2024-04-24T08:18:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:30 smithi177 ceph-mon[33046]: fsmap 8 up:standby 2024-04-24T08:18:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:30 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:18:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:30 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:18:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:30 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1671979646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:30.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:30 smithi153 ceph-mon[32746]: Deploying daemon mds.i on smithi177 2024-04-24T08:18:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:30 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.153:6836/565470680,v1:172.21.15.153:6837/565470680] up:boot 2024-04-24T08:18:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:30 smithi153 ceph-mon[32746]: fsmap 8 up:standby 2024-04-24T08:18:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:30 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:18:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:30 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:18:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:30 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1671979646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:30 smithi146 ceph-mon[27805]: Deploying daemon mds.i on smithi177 2024-04-24T08:18:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:30 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.153:6836/565470680,v1:172.21.15.153:6837/565470680] up:boot 2024-04-24T08:18:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:30 smithi146 ceph-mon[27805]: fsmap 8 up:standby 2024-04-24T08:18:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:30 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:18:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:30 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:18:30.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:30 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1671979646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:31.220 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 systemd[1]: Starting Ceph mds.i for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:31.497 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 podman[64427]: 2024-04-24 08:18:31.219804088 +0000 UTC m=+0.111741529 container create bdc88fcb6b872863c250b67ed4b02ef387ee0b230a8b565146f987c1ca45f3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, ceph=True) 2024-04-24T08:18:31.497 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 podman[64427]: 2024-04-24 08:18:31.126221833 +0000 UTC m=+0.018159305 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:31.498 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 podman[64427]: 2024-04-24 08:18:31.430465557 +0000 UTC m=+0.322402998 container init bdc88fcb6b872863c250b67ed4b02ef387ee0b230a8b565146f987c1ca45f3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-24T08:18:31.498 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 podman[64427]: 2024-04-24 08:18:31.433107377 +0000 UTC m=+0.325044820 container start bdc88fcb6b872863c250b67ed4b02ef387ee0b230a8b565146f987c1ca45f3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:18:31.498 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 ceph-mds[64446]: -- 172.21.15.177:0/132603402 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562be547fa40 con 0x562be5452c00 2024-04-24T08:18:31.498 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i[64442]: starting mds.i at 2024-04-24T08:18:31.498 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 bash[64427]: bdc88fcb6b872863c250b67ed4b02ef387ee0b230a8b565146f987c1ca45f3d5 2024-04-24T08:18:31.815 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:18:31 smithi177 systemd[1]: Started Ceph mds.i for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:31 smithi177 ceph-mon[33046]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:31 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:31.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:31 smithi153 ceph-mon[32746]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:31.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:31 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:31 smithi146 ceph-mon[27805]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:31 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:32.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:32.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.741+0000 7feb4d800640 1 -- 172.21.15.146:0/1413206220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 msgr2=0x7feb481058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.741+0000 7feb4d800640 1 --2- 172.21.15.146:0/1413206220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 0x7feb481058d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb30009900 tx=0x7feb3002a410 comp rx=0 tx=0).stop 2024-04-24T08:18:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.741+0000 7feb4d800640 1 -- 172.21.15.146:0/1413206220 shutdown_connections 2024-04-24T08:18:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.741+0000 7feb4d800640 1 --2- 172.21.15.146:0/1413206220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48105e10 0x7feb4810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.742+0000 7feb4d800640 1 --2- 172.21.15.146:0/1413206220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 0x7feb481058d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.742+0000 7feb4d800640 1 --2- 172.21.15.146:0/1413206220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb48104260 0x7feb48104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.742+0000 7feb4d800640 1 -- 172.21.15.146:0/1413206220 >> 172.21.15.146:0/1413206220 conn(0x7feb480ff7f0 msgr2=0x7feb48101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.742+0000 7feb4d800640 1 -- 172.21.15.146:0/1413206220 shutdown_connections 2024-04-24T08:18:32.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.742+0000 7feb4d800640 1 -- 172.21.15.146:0/1413206220 wait complete. 2024-04-24T08:18:32.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 Processor -- start 2024-04-24T08:18:32.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 -- start start 2024-04-24T08:18:32.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48104260 0x7feb4819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 0x7feb4819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb48105e10 0x7feb481a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feb4810ff40 con 0x7feb48104260 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb4810fdc0 con 0x7feb48105450 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb4d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb481100c0 con 0x7feb48105e10 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.743+0000 7feb46a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48104260 0x7feb4819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48104260 0x7feb4819a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46544/0 (socket says 172.21.15.146:46544) 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 -- 172.21.15.146:0/1840490505 learned_addr learned my addr 172.21.15.146:0/1840490505 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb3fe00640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 0x7feb4819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb47400640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb48105e10 0x7feb481a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 -- 172.21.15.146:0/1840490505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb48105e10 msgr2=0x7feb481a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb48105e10 0x7feb481a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 -- 172.21.15.146:0/1840490505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 msgr2=0x7feb4819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 0x7feb4819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 -- 172.21.15.146:0/1840490505 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb481a2190 con 0x7feb48104260 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.744+0000 7feb46a00640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48104260 0x7feb4819a570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb3800e9c0 tx=0x7feb3800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.745+0000 7feb3f400640 1 -- 172.21.15.146:0/1840490505 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3800cd80 con 0x7feb48104260 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.745+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb481a2480 con 0x7feb48104260 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.745+0000 7feb3f400640 1 -- 172.21.15.146:0/1840490505 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb380035f0 con 0x7feb48104260 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.745+0000 7feb3f400640 1 -- 172.21.15.146:0/1840490505 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38010690 con 0x7feb48104260 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.745+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb481a29f0 con 0x7feb48104260 2024-04-24T08:18:32.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.746+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb4810b580 con 0x7feb48104260 2024-04-24T08:18:32.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.748+0000 7feb3f400640 1 -- 172.21.15.146:0/1840490505 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7feb380037a0 con 0x7feb48104260 2024-04-24T08:18:32.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.749+0000 7feb3f400640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7feb2c0a3bb0 0x7feb2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.750+0000 7feb3f400640 1 -- 172.21.15.146:0/1840490505 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7feb380d03e0 con 0x7feb48104260 2024-04-24T08:18:32.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.750+0000 7feb3fe00640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7feb2c0a3bb0 0x7feb2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.750+0000 7feb3fe00640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7feb2c0a3bb0 0x7feb2c0a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feb30031440 tx=0x7feb300315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:32.754+0000 7feb3f400640 1 -- 172.21.15.146:0/1840490505 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb3809d1c0 con 0x7feb48104260 2024-04-24T08:18:32.882 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:32.882 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.177:6838/1533073551,v1:172.21.15.177:6839/1533073551] up:boot 2024-04-24T08:18:32.882 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: fsmap 9 up:standby 2024-04-24T08:18:32.882 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:18:32.882 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:32.883 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:32.883 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:32.883 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:32.883 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: Deploying daemon mds.g on smithi146 2024-04-24T08:18:32.883 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:32 smithi146 ceph-mon[27805]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:32.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.177:6838/1533073551,v1:172.21.15.177:6839/1533073551] up:boot 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: fsmap 9 up:standby 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: Deploying daemon mds.g on smithi146 2024-04-24T08:18:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:32 smithi153 ceph-mon[32746]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:33.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.015+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb48062c00 con 0x7feb48104260 2024-04-24T08:18:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.017+0000 7feb3f400640 1 -- 172.21.15.146:0/1840490505 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8019 (secure 0 0 0) 0x7feb3809cac0 con 0x7feb48104260 2024-04-24T08:18:33.018 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:33.018 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T08:18:33.018 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 10 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7feb2c0a3bb0 msgr2=0x7feb2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7feb2c0a3bb0 0x7feb2c0a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feb30031440 tx=0x7feb300315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48104260 msgr2=0x7feb4819a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48104260 0x7feb4819a570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feb3800e9c0 tx=0x7feb3800ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 shutdown_connections 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb48105e10 0x7feb481a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7feb2c0a3bb0 0x7feb2c0a6070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb48105450 0x7feb4819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 --2- 172.21.15.146:0/1840490505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb48104260 0x7feb4819a570 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 >> 172.21.15.146:0/1840490505 conn(0x7feb480ff7f0 msgr2=0x7feb48108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 shutdown_connections 2024-04-24T08:18:33.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:33.020+0000 7feb4d800640 1 -- 172.21.15.146:0/1840490505 wait complete. 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.177:6838/1533073551,v1:172.21.15.177:6839/1533073551] up:boot 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: fsmap 9 up:standby 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: Deploying daemon mds.g on smithi146 2024-04-24T08:18:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:32 smithi177 ceph-mon[33046]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:33.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:33 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1840490505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:33.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:33 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1840490505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:34.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:33 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1840490505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:34.863 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:34 smithi146 ceph-mon[27805]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:34.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:34 smithi153 ceph-mon[32746]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:35.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:35.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:34 smithi177 ceph-mon[33046]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:35.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.243+0000 7efe60c00640 1 -- 172.21.15.146:0/1965443035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 msgr2=0x7efe5c100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.243+0000 7efe60c00640 1 --2- 172.21.15.146:0/1965443035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c100b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efe4c009900 tx=0x7efe4c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.244+0000 7efe60c00640 1 -- 172.21.15.146:0/1965443035 shutdown_connections 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.244+0000 7efe60c00640 1 --2- 172.21.15.146:0/1965443035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5c101a50 0x7efe5c101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.244+0000 7efe60c00640 1 --2- 172.21.15.146:0/1965443035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe5c1010b0 0x7efe5c101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.244+0000 7efe60c00640 1 --2- 172.21.15.146:0/1965443035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c100b70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.244+0000 7efe60c00640 1 -- 172.21.15.146:0/1965443035 >> 172.21.15.146:0/1965443035 conn(0x7efe5c0fc2d0 msgr2=0x7efe5c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.244+0000 7efe60c00640 1 -- 172.21.15.146:0/1965443035 shutdown_connections 2024-04-24T08:18:35.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.244+0000 7efe60c00640 1 -- 172.21.15.146:0/1965443035 wait complete. 2024-04-24T08:18:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.245+0000 7efe60c00640 1 Processor -- start 2024-04-24T08:18:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe60c00640 1 -- start start 2024-04-24T08:18:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe60c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe60c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5c1010b0 0x7efe5c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe60c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe5c101a50 0x7efe5c1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe60c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c10fd70 con 0x7efe5c101a50 2024-04-24T08:18:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe60c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c10fbf0 con 0x7efe5c1010b0 2024-04-24T08:18:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe60c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efe5c10fef0 con 0x7efe5c100750 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe5b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe5b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:58282/0 (socket says 172.21.15.146:58282) 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe5b400640 1 -- 172.21.15.146:0/3365838274 learned_addr learned my addr 172.21.15.146:0/3365838274 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.246+0000 7efe5aa00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5c1010b0 0x7efe5c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe5be00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe5c101a50 0x7efe5c1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe5b400640 1 -- 172.21.15.146:0/3365838274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5c1010b0 msgr2=0x7efe5c19a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe5b400640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5c1010b0 0x7efe5c19a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe5b400640 1 -- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe5c101a50 msgr2=0x7efe5c1a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe5b400640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe5c101a50 0x7efe5c1a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe5b400640 1 -- 172.21.15.146:0/3365838274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe5c1a1f70 con 0x7efe5c100750 2024-04-24T08:18:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe5b400640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c19a350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efe4c031440 tx=0x7efe4c032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe53e00640 1 -- 172.21.15.146:0/3365838274 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c03f030 con 0x7efe5c100750 2024-04-24T08:18:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe5c1a2200 con 0x7efe5c100750 2024-04-24T08:18:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe53e00640 1 -- 172.21.15.146:0/3365838274 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe4c003ce0 con 0x7efe5c100750 2024-04-24T08:18:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.247+0000 7efe53e00640 1 -- 172.21.15.146:0/3365838274 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4c035070 con 0x7efe5c100750 2024-04-24T08:18:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.248+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe5c1a2740 con 0x7efe5c100750 2024-04-24T08:18:35.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.249+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe5c10b5f0 con 0x7efe5c100750 2024-04-24T08:18:35.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.251+0000 7efe53e00640 1 -- 172.21.15.146:0/3365838274 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7efe4c042030 con 0x7efe5c100750 2024-04-24T08:18:35.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.252+0000 7efe53e00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7efe400a3bb0 0x7efe400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.252+0000 7efe53e00640 1 -- 172.21.15.146:0/3365838274 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7efe4c0f01e0 con 0x7efe5c100750 2024-04-24T08:18:35.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.252+0000 7efe5aa00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7efe400a3bb0 0x7efe400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.253+0000 7efe5aa00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7efe400a3bb0 0x7efe400a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efe5c19b860 tx=0x7efe48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:35.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.256+0000 7efe53e00640 1 -- 172.21.15.146:0/3365838274 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe4c0bcfc0 con 0x7efe5c100750 2024-04-24T08:18:35.492 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:18:35 smithi146 systemd[1]: Started Ceph mds.g for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:35.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.511+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe5c062c00 con 0x7efe5c100750 2024-04-24T08:18:35.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.512+0000 7efe53e00640 1 -- 172.21.15.146:0/3365838274 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8019 (secure 0 0 0) 0x7efe4c0bc8c0 con 0x7efe5c100750 2024-04-24T08:18:35.513 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:35.513 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T08:18:35.514 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 10 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7efe400a3bb0 msgr2=0x7efe400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7efe400a3bb0 0x7efe400a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efe5c19b860 tx=0x7efe48005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 msgr2=0x7efe5c19a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c19a350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efe4c031440 tx=0x7efe4c032ce0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 shutdown_connections 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7efe400a3bb0 0x7efe400a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe5c101a50 0x7efe5c1a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe5c1010b0 0x7efe5c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 --2- 172.21.15.146:0/3365838274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe5c100750 0x7efe5c19a350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 >> 172.21.15.146:0/3365838274 conn(0x7efe5c0fc2d0 msgr2=0x7efe5c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 shutdown_connections 2024-04-24T08:18:35.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:35.515+0000 7efe60c00640 1 -- 172.21.15.146:0/3365838274 wait complete. 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: Deploying daemon mds.a on smithi146 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3365838274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] up:boot 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: fsmap 10 up:standby 2024-04-24T08:18:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:36 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:18:36.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: Deploying daemon mds.a on smithi146 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3365838274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] up:boot 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: fsmap 10 up:standby 2024-04-24T08:18:36.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:36 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: Deploying daemon mds.a on smithi146 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3365838274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] up:boot 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: fsmap 10 up:standby 2024-04-24T08:18:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:36 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:18:37.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:37.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:37 smithi153 ceph-mon[32746]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:37.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:37 smithi146 ceph-mon[27805]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:37.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 -- 172.21.15.146:0/3502473634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98105270 msgr2=0x7f1e981056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 --2- 172.21.15.146:0/3502473634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98105270 0x7f1e981056f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009900 tx=0x7f1e8402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 -- 172.21.15.146:0/3502473634 shutdown_connections 2024-04-24T08:18:37.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 --2- 172.21.15.146:0/3502473634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98105270 0x7f1e981056f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 --2- 172.21.15.146:0/3502473634 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e98105c30 0x7f1e9810c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 --2- 172.21.15.146:0/3502473634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e98104080 0x7f1e981044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 -- 172.21.15.146:0/3502473634 >> 172.21.15.146:0/3502473634 conn(0x7f1e980ff610 msgr2=0x7f1e98101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:37.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 -- 172.21.15.146:0/3502473634 shutdown_connections 2024-04-24T08:18:37.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.751+0000 7f1e97e00640 1 -- 172.21.15.146:0/3502473634 wait complete. 2024-04-24T08:18:37.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 Processor -- start 2024-04-24T08:18:37.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 -- start start 2024-04-24T08:18:37.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98104080 0x7f1e9819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e98105270 0x7f1e9819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e98105c30 0x7f1e981a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9810fd60 con 0x7f1e98105270 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9810fbe0 con 0x7f1e98105c30 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.752+0000 7f1e97e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9810fee0 con 0x7f1e98104080 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98104080 0x7f1e9819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e98105270 0x7f1e9819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98104080 0x7f1e9819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:58300/0 (socket says 172.21.15.146:58300) 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96a00640 1 -- 172.21.15.146:0/353361892 learned_addr learned my addr 172.21.15.146:0/353361892 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e97400640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e98105c30 0x7f1e981a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96000640 1 -- 172.21.15.146:0/353361892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98104080 msgr2=0x7f1e9819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96000640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98104080 0x7f1e9819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96000640 1 -- 172.21.15.146:0/353361892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e98105c30 msgr2=0x7f1e981a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96000640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e98105c30 0x7f1e981a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96000640 1 -- 172.21.15.146:0/353361892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e981a1fb0 con 0x7f1e98105270 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96a00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98104080 0x7f1e9819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.753+0000 7f1e96000640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e98105270 0x7f1e9819a8d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1e8402a910 tx=0x7f1e84032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.754+0000 7f1e8f400640 1 -- 172.21.15.146:0/353361892 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8403f030 con 0x7f1e98105270 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.754+0000 7f1e8f400640 1 -- 172.21.15.146:0/353361892 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e84003ce0 con 0x7f1e98105270 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.754+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e981a2240 con 0x7f1e98105270 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.754+0000 7f1e8f400640 1 -- 172.21.15.146:0/353361892 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e84035070 con 0x7f1e98105270 2024-04-24T08:18:37.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.754+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e981a2700 con 0x7f1e98105270 2024-04-24T08:18:37.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.755+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9810b3a0 con 0x7f1e98105270 2024-04-24T08:18:37.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.757+0000 7f1e8f400640 1 -- 172.21.15.146:0/353361892 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f1e84042030 con 0x7f1e98105270 2024-04-24T08:18:37.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.758+0000 7f1e8f400640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1e7c0a3bb0 0x7f1e7c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.759+0000 7f1e8f400640 1 -- 172.21.15.146:0/353361892 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f1e840f0460 con 0x7f1e98105270 2024-04-24T08:18:37.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.759+0000 7f1e96a00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1e7c0a3bb0 0x7f1e7c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.759+0000 7f1e96a00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1e7c0a3bb0 0x7f1e7c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1e880095f0 tx=0x7f1e88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:37.763+0000 7f1e8f400640 1 -- 172.21.15.146:0/353361892 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e840bcbf0 con 0x7f1e98105270 2024-04-24T08:18:37.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:37 smithi177 ceph-mon[33046]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:38.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.032+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e98062c00 con 0x7f1e98105270 2024-04-24T08:18:38.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.033+0000 7f1e8f400640 1 -- 172.21.15.146:0/353361892 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8848 (secure 0 0 0) 0x7f1e840c0ef0 con 0x7f1e98105270 2024-04-24T08:18:38.034 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:38.034 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-24T08:18:38.034 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 11 2024-04-24T08:18:38.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.035+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1e7c0a3bb0 msgr2=0x7f1e7c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.035+0000 7f1e97e00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1e7c0a3bb0 0x7f1e7c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1e880095f0 tx=0x7f1e88005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e98105270 msgr2=0x7f1e9819a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e98105270 0x7f1e9819a8d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1e8402a910 tx=0x7f1e84032ce0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 shutdown_connections 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1e7c0a3bb0 0x7f1e7c0a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1e98105c30 0x7f1e981a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e98105270 0x7f1e9819a8d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 --2- 172.21.15.146:0/353361892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e98104080 0x7f1e9819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 >> 172.21.15.146:0/353361892 conn(0x7f1e980ff610 msgr2=0x7f1e98108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 shutdown_connections 2024-04-24T08:18:38.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:38.036+0000 7f1e97e00640 1 -- 172.21.15.146:0/353361892 wait complete. 2024-04-24T08:18:38.657 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:38 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/353361892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:38.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:38 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/353361892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:38 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/353361892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:38.993 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:38 smithi146 systemd[1]: Starting Ceph mds.a for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:39.371 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 podman[69440]: 2024-04-24 08:18:38.926421223 +0000 UTC m=+0.038950177 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:39.371 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 podman[69440]: 2024-04-24 08:18:39.049145614 +0000 UTC m=+0.161674600 container create 16a5089a1089ab04ba94be053848bae524222cd0fc47201a65e0bb9fc27e5cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-24T08:18:39.371 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 podman[69440]: 2024-04-24 08:18:39.36969523 +0000 UTC m=+0.482224222 container init 16a5089a1089ab04ba94be053848bae524222cd0fc47201a65e0bb9fc27e5cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T08:18:39.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:39 smithi153 ceph-mon[32746]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:39.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:39 smithi146 ceph-mon[27805]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:39.743 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 podman[69440]: 2024-04-24 08:18:39.379834122 +0000 UTC m=+0.492363138 container start 16a5089a1089ab04ba94be053848bae524222cd0fc47201a65e0bb9fc27e5cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:18:39.743 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 ceph-mds[69459]: -- 172.21.15.146:0/1542894944 <== mon.0 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5613a7e05a40 con 0x5613a7dd8c00 2024-04-24T08:18:39.743 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a[69454]: starting mds.a at 2024-04-24T08:18:39.743 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 bash[69440]: 16a5089a1089ab04ba94be053848bae524222cd0fc47201a65e0bb9fc27e5cbf 2024-04-24T08:18:39.743 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:18:39 smithi146 systemd[1]: Started Ceph mds.a for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:39 smithi177 ceph-mon[33046]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:40.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 -- 172.21.15.146:0/1133668199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 msgr2=0x7f0f58101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 --2- 172.21.15.146:0/1133668199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f58101510 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009860 tx=0x7f0f4402a340 comp rx=0 tx=0).stop 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 -- 172.21.15.146:0/1133668199 shutdown_connections 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 --2- 172.21.15.146:0/1133668199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f58101a50 0x7f0f58101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 --2- 172.21.15.146:0/1133668199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f58101510 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 --2- 172.21.15.146:0/1133668199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f58100750 0x7f0f58100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 -- 172.21.15.146:0/1133668199 >> 172.21.15.146:0/1133668199 conn(0x7f0f580fc2d0 msgr2=0x7f0f580fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 -- 172.21.15.146:0/1133668199 shutdown_connections 2024-04-24T08:18:40.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.282+0000 7f0f5d200640 1 -- 172.21.15.146:0/1133668199 wait complete. 2024-04-24T08:18:40.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.283+0000 7f0f5d200640 1 Processor -- start 2024-04-24T08:18:40.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.283+0000 7f0f5d200640 1 -- start start 2024-04-24T08:18:40.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f5d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f58100750 0x7f0f5819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f5d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f5819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f5d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f58101a50 0x7f0f581a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f5d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5810fd20 con 0x7f0f58101a50 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f5d200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5810fba0 con 0x7f0f581010b0 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f5d200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5810fea0 con 0x7f0f58100750 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f5819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f5819a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:40216/0 (socket says 172.21.15.146:40216) 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 -- 172.21.15.146:0/3289409680 learned_addr learned my addr 172.21.15.146:0/3289409680 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f57e00640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f58101a50 0x7f0f581a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f57400640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f58100750 0x7f0f5819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 -- 172.21.15.146:0/3289409680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f58100750 msgr2=0x7f0f5819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f58100750 0x7f0f5819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 -- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f58101a50 msgr2=0x7f0f581a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f58101a50 0x7f0f581a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f56a00640 1 -- 172.21.15.146:0/3289409680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f581a1f80 con 0x7f0f581010b0 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.284+0000 7f0f57e00640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f58101a50 0x7f0f581a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.285+0000 7f0f56a00640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f5819a8a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f44035040 tx=0x7f0f44033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.285+0000 7f0f4fe00640 1 -- 172.21.15.146:0/3289409680 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44037040 con 0x7f0f581010b0 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.285+0000 7f0f4fe00640 1 -- 172.21.15.146:0/3289409680 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f44030850 con 0x7f0f581010b0 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.285+0000 7f0f4fe00640 1 -- 172.21.15.146:0/3289409680 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44004920 con 0x7f0f581010b0 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.285+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f581a2210 con 0x7f0f581010b0 2024-04-24T08:18:40.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.285+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f581a2750 con 0x7f0f581010b0 2024-04-24T08:18:40.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.286+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f5810b4a0 con 0x7f0f581010b0 2024-04-24T08:18:40.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.289+0000 7f0f4fe00640 1 -- 172.21.15.146:0/3289409680 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f0f44004050 con 0x7f0f581010b0 2024-04-24T08:18:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.291+0000 7f0f4fe00640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0f3c0a3c30 0x7f0f3c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.291+0000 7f0f4fe00640 1 -- 172.21.15.146:0/3289409680 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f0f440effe0 con 0x7f0f581010b0 2024-04-24T08:18:40.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.291+0000 7f0f57400640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0f3c0a3c30 0x7f0f3c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.292+0000 7f0f57400640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0f3c0a3c30 0x7f0f3c0a60f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0f48005fd0 tx=0x7f0f48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:40.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.295+0000 7f0f4fe00640 1 -- 172.21.15.146:0/3289409680 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f44042300 con 0x7f0f581010b0 2024-04-24T08:18:40.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.543+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f58062c00 con 0x7f0f581010b0 2024-04-24T08:18:40.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.543+0000 7f0f4fe00640 1 -- 172.21.15.146:0/3289409680 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9677 (secure 0 0 0) 0x7f0f440bce20 con 0x7f0f581010b0 2024-04-24T08:18:40.544 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:40.544 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-04-24T08:18:40.545 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 12 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0f3c0a3c30 msgr2=0x7f0f3c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0f3c0a3c30 0x7f0f3c0a60f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0f48005fd0 tx=0x7f0f48009260 comp rx=0 tx=0).stop 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 msgr2=0x7f0f5819a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f5819a8a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f44035040 tx=0x7f0f44033d90 comp rx=0 tx=0).stop 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 shutdown_connections 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0f3c0a3c30 0x7f0f3c0a60f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f58101a50 0x7f0f581a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f581010b0 0x7f0f5819a8a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 --2- 172.21.15.146:0/3289409680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f58100750 0x7f0f5819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 >> 172.21.15.146:0/3289409680 conn(0x7f0f580fc2d0 msgr2=0x7f0f580fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:40.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 shutdown_connections 2024-04-24T08:18:40.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:40.546+0000 7f0f5d200640 1 -- 172.21.15.146:0/3289409680 wait complete. 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] up:boot 2024-04-24T08:18:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: fsmap 11 up:standby 2024-04-24T08:18:40.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:18:40.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:40 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3289409680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] up:boot 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: fsmap 11 up:standby 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:18:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:40 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3289409680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:18:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:18:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] up:boot 2024-04-24T08:18:41.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: fsmap 11 up:standby 2024-04-24T08:18:41.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:18:41.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:40 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3289409680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:41.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:41 smithi146 ceph-mon[27805]: Deploying daemon mds.b on smithi153 2024-04-24T08:18:41.896 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:41 smithi146 ceph-mon[27805]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:41.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:41 smithi153 ceph-mon[32746]: Deploying daemon mds.b on smithi153 2024-04-24T08:18:41.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:41 smithi153 ceph-mon[32746]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:41.952 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:41 smithi153 systemd[1]: Starting Ceph mds.b for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:18:41.966 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:41 smithi177 ceph-mon[33046]: Deploying daemon mds.b on smithi153 2024-04-24T08:18:41.966 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:41 smithi177 ceph-mon[33046]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:42.211 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:41 smithi153 podman[62777]: 2024-04-24 08:18:41.854318486 +0000 UTC m=+0.018360251 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:18:42.211 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:41 smithi153 podman[62777]: 2024-04-24 08:18:41.954330166 +0000 UTC m=+0.118371919 container create e1abbd03208fb96e7f27c06ff5e6aa87391fc59e298e4004373028fd8274cfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:18:42.211 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:42 smithi153 podman[62777]: 2024-04-24 08:18:42.173241514 +0000 UTC m=+0.337283267 container init e1abbd03208fb96e7f27c06ff5e6aa87391fc59e298e4004373028fd8274cfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-24T08:18:42.211 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:42 smithi153 podman[62777]: 2024-04-24 08:18:42.175787901 +0000 UTC m=+0.339829663 container start e1abbd03208fb96e7f27c06ff5e6aa87391fc59e298e4004373028fd8274cfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:18:42.211 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:42 smithi153 ceph-mds[62796]: -- 172.21.15.153:0/1036678146 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e5e8f65a40 con 0x55e5e8f39000 2024-04-24T08:18:42.211 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: starting mds.b at 2024-04-24T08:18:42.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:42.710 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:42 smithi153 bash[62777]: e1abbd03208fb96e7f27c06ff5e6aa87391fc59e298e4004373028fd8274cfe6 2024-04-24T08:18:42.710 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:42 smithi153 systemd[1]: Started Ceph mds.b for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:18:42.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 -- 172.21.15.146:0/2920244695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d40769c0 msgr2=0x7f67d4076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:42.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 --2- 172.21.15.146:0/2920244695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d40769c0 0x7f67d4076de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009900 tx=0x7f67c402a410 comp rx=0 tx=0).stop 2024-04-24T08:18:42.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 -- 172.21.15.146:0/2920244695 shutdown_connections 2024-04-24T08:18:42.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 --2- 172.21.15.146:0/2920244695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d40785a0 0x7f67d4078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:42.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 --2- 172.21.15.146:0/2920244695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d4077c00 0x7f67d4078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:42.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 --2- 172.21.15.146:0/2920244695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d40769c0 0x7f67d4076de0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:42.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 -- 172.21.15.146:0/2920244695 >> 172.21.15.146:0/2920244695 conn(0x7f67d40ff7f0 msgr2=0x7f67d4101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:42.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.770+0000 7f67d9400640 1 -- 172.21.15.146:0/2920244695 shutdown_connections 2024-04-24T08:18:42.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.771+0000 7f67d9400640 1 -- 172.21.15.146:0/2920244695 wait complete. 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.771+0000 7f67d9400640 1 Processor -- start 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d9400640 1 -- start start 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40769c0 0x7f67d419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d9400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d4077c00 0x7f67d419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d40785a0 0x7f67d41a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f67d41142f0 con 0x7f67d40769c0 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d9400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67d4114170 con 0x7f67d4077c00 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d9400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67d4114470 con 0x7f67d40785a0 2024-04-24T08:18:42.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40769c0 0x7f67d419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40769c0 0x7f67d419e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47668/0 (socket says 172.21.15.146:47668) 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d2800640 1 -- 172.21.15.146:0/2882251611 learned_addr learned my addr 172.21.15.146:0/2882251611 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d40785a0 0x7f67d41a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.772+0000 7f67d1e00640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d4077c00 0x7f67d419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d2800640 1 -- 172.21.15.146:0/2882251611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d40785a0 msgr2=0x7f67d41a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d2800640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d40785a0 0x7f67d41a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d2800640 1 -- 172.21.15.146:0/2882251611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d4077c00 msgr2=0x7f67d419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d2800640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d4077c00 0x7f67d419eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d2800640 1 -- 172.21.15.146:0/2882251611 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d41a6580 con 0x7f67d40769c0 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d3200640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d40785a0 0x7f67d41a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d1e00640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d4077c00 0x7f67d419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d2800640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40769c0 0x7f67d419e960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f67c4031440 tx=0x7f67c4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67cb400640 1 -- 172.21.15.146:0/2882251611 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c403f030 con 0x7f67d40769c0 2024-04-24T08:18:42.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67cb400640 1 -- 172.21.15.146:0/2882251611 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c4003ce0 con 0x7f67d40769c0 2024-04-24T08:18:42.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d41a6810 con 0x7f67d40769c0 2024-04-24T08:18:42.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.773+0000 7f67cb400640 1 -- 172.21.15.146:0/2882251611 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4035070 con 0x7f67d40769c0 2024-04-24T08:18:42.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.774+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d41a6d50 con 0x7f67d40769c0 2024-04-24T08:18:42.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.775+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d4110d60 con 0x7f67d40769c0 2024-04-24T08:18:42.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.777+0000 7f67cb400640 1 -- 172.21.15.146:0/2882251611 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f67c4042030 con 0x7f67d40769c0 2024-04-24T08:18:42.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.778+0000 7f67cb400640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f67b80a3c30 0x7f67b80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:42.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.778+0000 7f67cb400640 1 -- 172.21.15.146:0/2882251611 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f67c40f0410 con 0x7f67d40769c0 2024-04-24T08:18:42.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.779+0000 7f67d1e00640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f67b80a3c30 0x7f67b80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:42.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.779+0000 7f67d1e00640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f67b80a3c30 0x7f67b80a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67d419fe70 tx=0x7f67c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:42.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:42.783+0000 7f67cb400640 1 -- 172.21.15.146:0/2882251611 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67c40bcba0 con 0x7f67d40769c0 2024-04-24T08:18:43.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.029+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67d402d620 con 0x7f67d40769c0 2024-04-24T08:18:43.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.031+0000 7f67cb400640 1 -- 172.21.15.146:0/2882251611 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10506 (secure 0 0 0) 0x7f67c40c0ea0 con 0x7f67d40769c0 2024-04-24T08:18:43.032 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:43.032 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-24T08:18:43.033 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 13 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f67b80a3c30 msgr2=0x7f67b80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f67b80a3c30 0x7f67b80a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67d419fe70 tx=0x7f67c0005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40769c0 msgr2=0x7f67d419e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40769c0 0x7f67d419e960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f67c4031440 tx=0x7f67c4032ce0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 shutdown_connections 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67d40785a0 0x7f67d41a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f67b80a3c30 0x7f67b80a60f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67d4077c00 0x7f67d419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 --2- 172.21.15.146:0/2882251611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67d40769c0 0x7f67d419e960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.034+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 >> 172.21.15.146:0/2882251611 conn(0x7f67d40ff7f0 msgr2=0x7f67d4100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:43.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.035+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 shutdown_connections 2024-04-24T08:18:43.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.035+0000 7f67d9400640 1 -- 172.21.15.146:0/2882251611 wait complete. 2024-04-24T08:18:43.054 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:43.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 -- 172.21.15.146:0/2524737788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101c40 msgr2=0x7f29a41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2524737788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101c40 0x7f29a41020c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f299c009ca0 tx=0x7f299c02a950 comp rx=0 tx=0).stop 2024-04-24T08:18:43.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 -- 172.21.15.146:0/2524737788 shutdown_connections 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2524737788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101c40 0x7f29a41020c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2524737788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a41012a0 0x7f29a4101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2524737788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a4100940 0x7f29a4100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 -- 172.21.15.146:0/2524737788 >> 172.21.15.146:0/2524737788 conn(0x7f29a40fc4e0 msgr2=0x7f29a40fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 -- 172.21.15.146:0/2524737788 shutdown_connections 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.251+0000 7f29a9e00640 1 -- 172.21.15.146:0/2524737788 wait complete. 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 Processor -- start 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 -- start start 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a4100940 0x7f29a419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a41012a0 0x7f29a419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a4101c40 0x7f29a41a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f29a410ff30 con 0x7f29a4101c40 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29a410fdb0 con 0x7f29a4100940 2024-04-24T08:18:43.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.252+0000 7f29a9e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29a41100b0 con 0x7f29a41012a0 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a41012a0 0x7f29a419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a41012a0 0x7f29a419aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:49348/0 (socket says 172.21.15.146:49348) 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2000640 1 -- 172.21.15.146:0/2632994408 learned_addr learned my addr 172.21.15.146:0/2632994408 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2a00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a4100940 0x7f29a419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a3400640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a4101c40 0x7f29a41a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2a00640 1 -- 172.21.15.146:0/2632994408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a41012a0 msgr2=0x7f29a419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2a00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a41012a0 0x7f29a419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2a00640 1 -- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a4101c40 msgr2=0x7f29a41a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2a00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a4101c40 0x7f29a41a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2a00640 1 -- 172.21.15.146:0/2632994408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a41a2140 con 0x7f29a4100940 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2000640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a41012a0 0x7f29a419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a3400640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a4101c40 0x7f29a41a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.253+0000 7f29a2a00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a4100940 0x7f29a419a520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f299400e9f0 tx=0x7f299400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:43.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.254+0000 7f29a8a00640 1 -- 172.21.15.146:0/2632994408 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299400ccc0 con 0x7f29a4100940 2024-04-24T08:18:43.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.254+0000 7f29a8a00640 1 -- 172.21.15.146:0/2632994408 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2994010910 con 0x7f29a4100940 2024-04-24T08:18:43.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.254+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a41a2430 con 0x7f29a4100940 2024-04-24T08:18:43.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.254+0000 7f29a8a00640 1 -- 172.21.15.146:0/2632994408 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994018960 con 0x7f29a4100940 2024-04-24T08:18:43.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.254+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a41a2980 con 0x7f29a4100940 2024-04-24T08:18:43.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.255+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a410b7d0 con 0x7f29a4100940 2024-04-24T08:18:43.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.257+0000 7f29a8a00640 1 -- 172.21.15.146:0/2632994408 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f2994010ac0 con 0x7f29a4100940 2024-04-24T08:18:43.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.258+0000 7f29a8a00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f29880a3bb0 0x7f29880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.258+0000 7f29a8a00640 1 -- 172.21.15.146:0/2632994408 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f29940d03a0 con 0x7f29a4100940 2024-04-24T08:18:43.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.258+0000 7f29a2000640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f29880a3bb0 0x7f29880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.259+0000 7f29a2000640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f29880a3bb0 0x7f29880a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f29a419ba30 tx=0x7f2990005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:43.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.260+0000 7f29a8a00640 1 -- 172.21.15.146:0/2632994408 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f299409cb30 con 0x7f29a4100940 2024-04-24T08:18:43.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.478+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29a402d620 con 0x7f29a4100940 2024-04-24T08:18:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.480+0000 7f29a8a00640 1 -- 172.21.15.146:0/2632994408 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18499 (secure 0 0 0) 0x7f29940a0e30 con 0x7f29a4100940 2024-04-24T08:18:43.481 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:43.481 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":84,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:17:54.027821+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:43.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f29880a3bb0 msgr2=0x7f29880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f29880a3bb0 0x7f29880a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f29a419ba30 tx=0x7f2990005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:43.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a4100940 msgr2=0x7f29a419a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a4100940 0x7f29a419a520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f299400e9f0 tx=0x7f299400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 shutdown_connections 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a41012a0 0x7f29a419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f29880a3bb0 0x7f29880a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a4101c40 0x7f29a41a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 --2- 172.21.15.146:0/2632994408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a4100940 0x7f29a419a520 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 >> 172.21.15.146:0/2632994408 conn(0x7f29a40fc4e0 msgr2=0x7f29a40fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 shutdown_connections 2024-04-24T08:18:43.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.483+0000 7f29a9e00640 1 -- 172.21.15.146:0/2632994408 wait complete. 2024-04-24T08:18:43.506 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:43.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] up:boot 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: fsmap 12 up:standby 2024-04-24T08:18:43.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:18:43.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:43.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:43 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2882251611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] up:boot 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: fsmap 12 up:standby 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:43.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:43 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2882251611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:43.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 -- 172.21.15.146:0/374101776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a0101230 msgr2=0x7f34a0101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 --2- 172.21.15.146:0/374101776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a0101230 0x7f34a0101650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3490009900 tx=0x7f349002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 -- 172.21.15.146:0/374101776 shutdown_connections 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 --2- 172.21.15.146:0/374101776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a01006c0 0x7f34a01022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 --2- 172.21.15.146:0/374101776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 0x7f34a0100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 --2- 172.21.15.146:0/374101776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a0101230 0x7f34a0101650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 -- 172.21.15.146:0/374101776 >> 172.21.15.146:0/374101776 conn(0x7f34a00fb800 msgr2=0x7f34a00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 -- 172.21.15.146:0/374101776 shutdown_connections 2024-04-24T08:18:43.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.687+0000 7f34a4a00640 1 -- 172.21.15.146:0/374101776 wait complete. 2024-04-24T08:18:43.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 Processor -- start 2024-04-24T08:18:43.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 -- start start 2024-04-24T08:18:43.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 0x7f34a019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a01006c0 0x7f34a019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a0101230 0x7f34a01a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f34a010ff30 con 0x7f34a00ffd00 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f34a010fdb0 con 0x7f34a01006c0 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.688+0000 7f34a4a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34a01100b0 con 0x7f34a0101230 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 0x7f34a019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a01006c0 0x7f34a019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 0x7f34a019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47710/0 (socket says 172.21.15.146:47710) 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 -- 172.21.15.146:0/3075542184 learned_addr learned my addr 172.21.15.146:0/3075542184 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349e800640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a0101230 0x7f34a01a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 -- 172.21.15.146:0/3075542184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a0101230 msgr2=0x7f34a01a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a0101230 0x7f34a01a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 -- 172.21.15.146:0/3075542184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a01006c0 msgr2=0x7f34a019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a01006c0 0x7f34a019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 -- 172.21.15.146:0/3075542184 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a01a21e0 con 0x7f34a00ffd00 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349d400640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a01006c0 0x7f34a019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.689+0000 7f349de00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 0x7f34a019a5c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3490031440 tx=0x7f3490032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.690+0000 7f3496a00640 1 -- 172.21.15.146:0/3075542184 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349003f030 con 0x7f34a00ffd00 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.690+0000 7f3496a00640 1 -- 172.21.15.146:0/3075542184 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3490003ce0 con 0x7f34a00ffd00 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.690+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a01a2470 con 0x7f34a00ffd00 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.690+0000 7f3496a00640 1 -- 172.21.15.146:0/3075542184 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3490035070 con 0x7f34a00ffd00 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.690+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a01a2960 con 0x7f34a00ffd00 2024-04-24T08:18:43.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.691+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a010c500 con 0x7f34a00ffd00 2024-04-24T08:18:43.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.693+0000 7f3496a00640 1 -- 172.21.15.146:0/3075542184 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f3490042030 con 0x7f34a00ffd00 2024-04-24T08:18:43.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.694+0000 7f3496a00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f34840a3bb0 0x7f34840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:43.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.694+0000 7f3496a00640 1 -- 172.21.15.146:0/3075542184 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f34900f0170 con 0x7f34a00ffd00 2024-04-24T08:18:43.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.694+0000 7f349d400640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f34840a3bb0 0x7f34840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:43.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.695+0000 7f349d400640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f34840a3bb0 0x7f34840a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f34a019bad0 tx=0x7f348c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:43.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.698+0000 7f3496a00640 1 -- 172.21.15.146:0/3075542184 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34900bc900 con 0x7f34a00ffd00 2024-04-24T08:18:43.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] up:boot 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: fsmap 12 up:standby 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:43.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:43 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2882251611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:43.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.901+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f34a0062c00 con 0x7f34a00ffd00 2024-04-24T08:18:43.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.902+0000 7f3496a00640 1 -- 172.21.15.146:0/3075542184 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f34900c0c00 con 0x7f34a00ffd00 2024-04-24T08:18:43.903 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:18:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.904+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f34840a3bb0 msgr2=0x7f34840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.904+0000 7f34a4a00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f34840a3bb0 0x7f34840a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f34a019bad0 tx=0x7f348c005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.904+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 msgr2=0x7f34a019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.904+0000 7f34a4a00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 0x7f34a019a5c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3490031440 tx=0x7f3490032ce0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 shutdown_connections 2024-04-24T08:18:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34a0101230 0x7f34a01a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f34840a3bb0 0x7f34840a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34a01006c0 0x7f34a019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 --2- 172.21.15.146:0/3075542184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34a00ffd00 0x7f34a019a5c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:43.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 >> 172.21.15.146:0/3075542184 conn(0x7f34a00fb800 msgr2=0x7f34a00fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:43.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 shutdown_connections 2024-04-24T08:18:43.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:43.905+0000 7f34a4a00640 1 -- 172.21.15.146:0/3075542184 wait complete. 2024-04-24T08:18:43.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:18:44.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 -- 172.21.15.146:0/3875124799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a40749e0 msgr2=0x7fa6a4074e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 --2- 172.21.15.146:0/3875124799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a40749e0 0x7fa6a4074e40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa690009900 tx=0x7fa69002a480 comp rx=0 tx=0).stop 2024-04-24T08:18:44.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 -- 172.21.15.146:0/3875124799 shutdown_connections 2024-04-24T08:18:44.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 --2- 172.21.15.146:0/3875124799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a40749e0 0x7fa6a4074e40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 --2- 172.21.15.146:0/3875124799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4109f00 0x7fa6a410c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 --2- 172.21.15.146:0/3875124799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6a4075fc0 0x7fa6a4074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 -- 172.21.15.146:0/3875124799 >> 172.21.15.146:0/3875124799 conn(0x7fa6a40fd5e0 msgr2=0x7fa6a40ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:44.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 -- 172.21.15.146:0/3875124799 shutdown_connections 2024-04-24T08:18:44.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.109+0000 7fa6ab200640 1 -- 172.21.15.146:0/3875124799 wait complete. 2024-04-24T08:18:44.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.110+0000 7fa6ab200640 1 Processor -- start 2024-04-24T08:18:44.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6ab200640 1 -- start start 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6ab200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6a40749e0 0x7fa6a419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6ab200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a4075fc0 0x7fa6a419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6ab200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4109f00 0x7fa6a41a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6ab200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a410ff40 con 0x7fa6a40749e0 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6ab200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a410fdc0 con 0x7fa6a4109f00 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6ab200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a41100c0 con 0x7fa6a4075fc0 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6a40749e0 0x7fa6a419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a4075fc0 0x7fa6a419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a4075fc0 0x7fa6a419aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:49394/0 (socket says 172.21.15.146:49394) 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 -- 172.21.15.146:0/1557104013 learned_addr learned my addr 172.21.15.146:0/1557104013 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a9200640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4109f00 0x7fa6a41a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 -- 172.21.15.146:0/1557104013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4109f00 msgr2=0x7fa6a41a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4109f00 0x7fa6a41a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 -- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6a40749e0 msgr2=0x7fa6a419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6a40749e0 0x7fa6a419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.111+0000 7fa6a3400640 1 -- 172.21.15.146:0/1557104013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a41a2170 con 0x7fa6a4075fc0 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.112+0000 7fa6a3400640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a4075fc0 0x7fa6a419aa90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa690031440 tx=0x7fa690032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.112+0000 7fa6a0c00640 1 -- 172.21.15.146:0/1557104013 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69003f030 con 0x7fa6a4075fc0 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.112+0000 7fa6a0c00640 1 -- 172.21.15.146:0/1557104013 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa690003ce0 con 0x7fa6a4075fc0 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.112+0000 7fa6a0c00640 1 -- 172.21.15.146:0/1557104013 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690035070 con 0x7fa6a4075fc0 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.112+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a41a2400 con 0x7fa6a4075fc0 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.112+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a41a28a0 con 0x7fa6a4075fc0 2024-04-24T08:18:44.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.113+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a4108cc0 con 0x7fa6a4075fc0 2024-04-24T08:18:44.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.115+0000 7fa6a0c00640 1 -- 172.21.15.146:0/1557104013 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa690042030 con 0x7fa6a4075fc0 2024-04-24T08:18:44.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.116+0000 7fa6a0c00640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6880a3bb0 0x7fa6880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.116+0000 7fa6a0c00640 1 -- 172.21.15.146:0/1557104013 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa6900f0870 con 0x7fa6a4075fc0 2024-04-24T08:18:44.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.117+0000 7fa6a3e00640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6880a3bb0 0x7fa6880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.117+0000 7fa6a3e00640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6880a3bb0 0x7fa6880a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6940095f0 tx=0x7fa694005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:44.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.120+0000 7fa6a0c00640 1 -- 172.21.15.146:0/1557104013 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6900bd650 con 0x7fa6a4075fc0 2024-04-24T08:18:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.334+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa6a4062c00 con 0x7fa6a4075fc0 2024-04-24T08:18:44.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.335+0000 7fa6a0c00640 1 -- 172.21.15.146:0/1557104013 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) ==== 84+0+65 (secure 0 0 0) 0x7fa6900bcf50 con 0x7fa6a4075fc0 2024-04-24T08:18:44.336 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:44.336 INFO:teuthology.orchestra.run.smithi146.stdout:[ 2024-04-24T08:18:44.336 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:18:44.336 INFO:teuthology.orchestra.run.smithi146.stdout: "poolnum": 1, 2024-04-24T08:18:44.336 INFO:teuthology.orchestra.run.smithi146.stdout: "poolname": ".mgr" 2024-04-24T08:18:44.336 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:18:44.336 INFO:teuthology.orchestra.run.smithi146.stdout:] 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6880a3bb0 msgr2=0x7fa6880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6880a3bb0 0x7fa6880a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa6940095f0 tx=0x7fa694005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a4075fc0 msgr2=0x7fa6a419aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a4075fc0 0x7fa6a419aa90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa690031440 tx=0x7fa690032ce0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 shutdown_connections 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6a4075fc0 0x7fa6a419aa90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6880a3bb0 0x7fa6880a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6a4109f00 0x7fa6a41a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 --2- 172.21.15.146:0/1557104013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6a40749e0 0x7fa6a419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 >> 172.21.15.146:0/1557104013 conn(0x7fa6a40fd5e0 msgr2=0x7fa6a4105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:44.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 shutdown_connections 2024-04-24T08:18:44.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.339+0000 7fa6ab200640 1 -- 172.21.15.146:0/1557104013 wait complete. 2024-04-24T08:18:44.358 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:18:44.358 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:18:44.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 -- 172.21.15.146:0/3441385560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105450 msgr2=0x7f4c281058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/3441385560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105450 0x7f4c281058d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009900 tx=0x7f4c1002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 -- 172.21.15.146:0/3441385560 shutdown_connections 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/3441385560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28105e10 0x7f4c2810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/3441385560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105450 0x7f4c281058d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/3441385560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c28104260 0x7f4c28104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 -- 172.21.15.146:0/3441385560 >> 172.21.15.146:0/3441385560 conn(0x7f4c280ff7b0 msgr2=0x7f4c28101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 -- 172.21.15.146:0/3441385560 shutdown_connections 2024-04-24T08:18:44.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.554+0000 7f4c2ca00640 1 -- 172.21.15.146:0/3441385560 wait complete. 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 Processor -- start 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 -- start start 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28104260 0x7f4c2819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c28105450 0x7f4c2819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105e10 0x7f4c281a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2810fef0 con 0x7f4c28105e10 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2810fd70 con 0x7f4c28104260 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c2ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c28110070 con 0x7f4c28105450 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.555+0000 7f4c25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28104260 0x7f4c2819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28104260 0x7f4c2819a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33554/0 (socket says 172.21.15.146:33554) 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 -- 172.21.15.146:0/751456515 learned_addr learned my addr 172.21.15.146:0/751456515 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c26800640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105e10 0x7f4c281a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c1fe00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c28105450 0x7f4c2819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 -- 172.21.15.146:0/751456515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c28105450 msgr2=0x7f4c2819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c28105450 0x7f4c2819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 -- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105e10 msgr2=0x7f4c281a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105e10 0x7f4c281a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 -- 172.21.15.146:0/751456515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c281a2170 con 0x7f4c28104260 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c1fe00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c28105450 0x7f4c2819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c25e00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28104260 0x7f4c2819a550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e9c0 tx=0x7f4c1800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c1ea00640 1 -- 172.21.15.146:0/751456515 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800cd80 con 0x7f4c28104260 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c281a2460 con 0x7f4c28104260 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c1ea00640 1 -- 172.21.15.146:0/751456515 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c180035f0 con 0x7f4c28104260 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.556+0000 7f4c1ea00640 1 -- 172.21.15.146:0/751456515 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18010690 con 0x7f4c28104260 2024-04-24T08:18:44.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.557+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c281a2930 con 0x7f4c28104260 2024-04-24T08:18:44.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.558+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c2810b580 con 0x7f4c28104260 2024-04-24T08:18:44.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.559+0000 7f4c1ea00640 1 -- 172.21.15.146:0/751456515 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4c180037a0 con 0x7f4c28104260 2024-04-24T08:18:44.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.561+0000 7f4c1ea00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:44.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.561+0000 7f4c1ea00640 1 -- 172.21.15.146:0/751456515 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f4c180d0410 con 0x7f4c28104260 2024-04-24T08:18:44.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.561+0000 7f4c1fe00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:44.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.562+0000 7f4c1fe00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c10000c00 tx=0x7f4c100095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:44.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.565+0000 7f4c1ea00640 1 -- 172.21.15.146:0/751456515 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c1809d1f0 con 0x7f4c28104260 2024-04-24T08:18:44.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:44 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2632994408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:44 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3075542184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:44 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:44 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:44 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1557104013' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T08:18:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:44 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:44 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:44 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2632994408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:44 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3075542184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:44 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:44 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:44 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1557104013' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T08:18:44.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:44 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:44 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:44.793+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4c2802d620 con 0x7f4c28104260 2024-04-24T08:18:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:44 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2632994408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:44 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3075542184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:18:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:44 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:44 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:44 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1557104013' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T08:18:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:44 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:44 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.424+0000 7f4c1ea00640 1 -- 172.21.15.146:0/751456515 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) ==== 138+0+0 (secure 0 0 0) 0x7f4c1809caf0 con 0x7f4c28104260 2024-04-24T08:18:45.425 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' created 2024-04-24T08:18:45.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c0c0a3bb0 msgr2=0x7f4c0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:45.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4c10000c00 tx=0x7f4c100095f0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28104260 msgr2=0x7f4c2819a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:45.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28104260 0x7f4c2819a550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e9c0 tx=0x7f4c1800ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:45.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 shutdown_connections 2024-04-24T08:18:45.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c28105450 0x7f4c2819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c28105e10 0x7f4c281a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 --2- 172.21.15.146:0/751456515 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28104260 0x7f4c2819a550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.431+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 >> 172.21.15.146:0/751456515 conn(0x7f4c280ff7b0 msgr2=0x7f4c28108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:45.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.432+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 shutdown_connections 2024-04-24T08:18:45.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.432+0000 7f4c2ca00640 1 -- 172.21.15.146:0/751456515 wait complete. 2024-04-24T08:18:45.452 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 -- 172.21.15.146:0/3186474062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc074a60 msgr2=0x7f12cc074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 --2- 172.21.15.146:0/3186474062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc074a60 0x7f12cc074ee0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009900 tx=0x7f12b802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 -- 172.21.15.146:0/3186474062 shutdown_connections 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 --2- 172.21.15.146:0/3186474062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12cc109e50 0x7f12cc10c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 --2- 172.21.15.146:0/3186474062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc074a60 0x7f12cc074ee0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 --2- 172.21.15.146:0/3186474062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12cc076040 0x7f12cc074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 -- 172.21.15.146:0/3186474062 >> 172.21.15.146:0/3186474062 conn(0x7f12cc0fd500 msgr2=0x7f12cc0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 -- 172.21.15.146:0/3186474062 shutdown_connections 2024-04-24T08:18:45.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.652+0000 7f12d0c00640 1 -- 172.21.15.146:0/3186474062 wait complete. 2024-04-24T08:18:45.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 Processor -- start 2024-04-24T08:18:45.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 -- start start 2024-04-24T08:18:45.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12cc074a60 0x7f12cc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:45.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc076040 0x7f12cc19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:45.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12cc109e50 0x7f12cc1a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:45.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc10fdc0 con 0x7f12cc074a60 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc10fc40 con 0x7f12cc076040 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.653+0000 7f12d0c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc10ff40 con 0x7f12cc109e50 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc076040 0x7f12cc19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12cb400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12cc074a60 0x7f12cc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc076040 0x7f12cc19a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33574/0 (socket says 172.21.15.146:33574) 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12cbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12cc109e50 0x7f12cc1a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 -- 172.21.15.146:0/1514301916 learned_addr learned my addr 172.21.15.146:0/1514301916 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 -- 172.21.15.146:0/1514301916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12cc109e50 msgr2=0x7f12cc1a1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12cc109e50 0x7f12cc1a1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 -- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12cc074a60 msgr2=0x7f12cc19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12cc074a60 0x7f12cc19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:45.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 -- 172.21.15.146:0/1514301916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12cc1a20a0 con 0x7f12cc076040 2024-04-24T08:18:45.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.654+0000 7f12caa00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc076040 0x7f12cc19a9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12b8031440 tx=0x7f12b8032e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:45.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.655+0000 7f12c3e00640 1 -- 172.21.15.146:0/1514301916 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b803f030 con 0x7f12cc076040 2024-04-24T08:18:45.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.655+0000 7f12c3e00640 1 -- 172.21.15.146:0/1514301916 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b8003ce0 con 0x7f12cc076040 2024-04-24T08:18:45.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.655+0000 7f12c3e00640 1 -- 172.21.15.146:0/1514301916 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8035070 con 0x7f12cc076040 2024-04-24T08:18:45.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.655+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12cc1a2330 con 0x7f12cc076040 2024-04-24T08:18:45.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.655+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12cc1a27d0 con 0x7f12cc076040 2024-04-24T08:18:45.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.656+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12cc108b80 con 0x7f12cc076040 2024-04-24T08:18:45.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.658+0000 7f12c3e00640 1 -- 172.21.15.146:0/1514301916 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f12b8042030 con 0x7f12cc076040 2024-04-24T08:18:45.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.660+0000 7f12c3e00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f12b00a3bb0 0x7f12b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:45.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.660+0000 7f12c3e00640 1 -- 172.21.15.146:0/1514301916 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7497+0+0 (secure 0 0 0) 0x7f12b80f0720 con 0x7f12cc076040 2024-04-24T08:18:45.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.660+0000 7f12cb400640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f12b00a3bb0 0x7f12b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:45.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.661+0000 7f12cb400640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f12b00a3bb0 0x7f12b00a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00bbe0 tx=0x7f12bc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:45.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.664+0000 7f12c3e00640 1 -- 172.21.15.146:0/1514301916 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12b80bcd80 con 0x7f12cc076040 2024-04-24T08:18:45.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:45.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/751456515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:18:45.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:18:45.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:45.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:45.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:45 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/751456515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:45 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/751456515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:45 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:45.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:45.870+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f12cc062c00 con 0x7f12cc076040 2024-04-24T08:18:46.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.422+0000 7f12c3e00640 1 -- 172.21.15.146:0/1514301916 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) ==== 170+0+0 (secure 0 0 0) 0x7f12b80c1080 con 0x7f12cc076040 2024-04-24T08:18:46.423 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' created 2024-04-24T08:18:46.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.430+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f12b00a3bb0 msgr2=0x7f12b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:46.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.430+0000 7f12d0c00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f12b00a3bb0 0x7f12b00a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00bbe0 tx=0x7f12bc00b390 comp rx=0 tx=0).stop 2024-04-24T08:18:46.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.430+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc076040 msgr2=0x7f12cc19a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:46.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.430+0000 7f12d0c00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc076040 0x7f12cc19a9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12b8031440 tx=0x7f12b8032e20 comp rx=0 tx=0).stop 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.432+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 shutdown_connections 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.432+0000 7f12d0c00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12cc109e50 0x7f12cc1a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.432+0000 7f12d0c00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f12b00a3bb0 0x7f12b00a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.432+0000 7f12d0c00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12cc076040 0x7f12cc19a9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.432+0000 7f12d0c00640 1 --2- 172.21.15.146:0/1514301916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12cc074a60 0x7f12cc19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.432+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 >> 172.21.15.146:0/1514301916 conn(0x7f12cc0fd500 msgr2=0x7f12cc105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.433+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 shutdown_connections 2024-04-24T08:18:46.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.433+0000 7f12d0c00640 1 -- 172.21.15.146:0/1514301916 wait complete. 2024-04-24T08:18:46.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 -- 172.21.15.146:0/3484838003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 msgr2=0x7f754c10c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 --2- 172.21.15.146:0/3484838003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 0x7f754c10c6c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7544009be0 tx=0x7f754402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 -- 172.21.15.146:0/3484838003 shutdown_connections 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 --2- 172.21.15.146:0/3484838003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c105470 0x7f754c1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 --2- 172.21.15.146:0/3484838003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 0x7f754c10c6c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 --2- 172.21.15.146:0/3484838003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f754c104280 0x7f754c1046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 -- 172.21.15.146:0/3484838003 >> 172.21.15.146:0/3484838003 conn(0x7f754c0ff7f0 msgr2=0x7f754c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 -- 172.21.15.146:0/3484838003 shutdown_connections 2024-04-24T08:18:46.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.637+0000 7f7551200640 1 -- 172.21.15.146:0/3484838003 wait complete. 2024-04-24T08:18:46.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 Processor -- start 2024-04-24T08:18:46.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 -- start start 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c104280 0x7f754c19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f754c105470 0x7f754c19aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 0x7f754c1a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f754c10fee0 con 0x7f754c105e30 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f754c10fd60 con 0x7f754c105470 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7551200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f754c110060 con 0x7f754c104280 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.638+0000 7f7549e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c104280 0x7f754c19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c104280 0x7f754c19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:49440/0 (socket says 172.21.15.146:49440) 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 -- 172.21.15.146:0/2167817297 learned_addr learned my addr 172.21.15.146:0/2167817297 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f754a800640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 0x7f754c1a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549400640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f754c105470 0x7f754c19aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 -- 172.21.15.146:0/2167817297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f754c105470 msgr2=0x7f754c19aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:46.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f754c105470 0x7f754c19aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 -- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 msgr2=0x7f754c1a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 0x7f754c1a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 -- 172.21.15.146:0/2167817297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c1a2110 con 0x7f754c104280 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7549e00640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c104280 0x7f754c19a4f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9a0 tx=0x7f753c00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.639+0000 7f7542a00640 1 -- 172.21.15.146:0/2167817297 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c00cd80 con 0x7f754c104280 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.640+0000 7f7542a00640 1 -- 172.21.15.146:0/2167817297 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f753c010910 con 0x7f754c104280 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.640+0000 7f7542a00640 1 -- 172.21.15.146:0/2167817297 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c018a30 con 0x7f754c104280 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.640+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754c1a2400 con 0x7f754c104280 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.640+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754c1a2900 con 0x7f754c104280 2024-04-24T08:18:46.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.641+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f754c10b5a0 con 0x7f754c104280 2024-04-24T08:18:46.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.643+0000 7f7542a00640 1 -- 172.21.15.146:0/2167817297 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f753c010ac0 con 0x7f754c104280 2024-04-24T08:18:46.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.644+0000 7f7542a00640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f75300a3c30 0x7f75300a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:46.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.644+0000 7f7542a00640 1 -- 172.21.15.146:0/2167817297 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7879+0+0 (secure 0 0 0) 0x7f753c0d05f0 con 0x7f754c104280 2024-04-24T08:18:46.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.644+0000 7f7549400640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f75300a3c30 0x7f75300a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:46.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.645+0000 7f7549400640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f75300a3c30 0x7f75300a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f754c19ba00 tx=0x7f7534005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:46.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.648+0000 7f7542a00640 1 -- 172.21.15.146:0/2167817297 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f753c053ff0 con 0x7f754c104280 2024-04-24T08:18:46.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:46 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T08:18:46.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:46 smithi153 ceph-mon[32746]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T08:18:46.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:46 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:46.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:46 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1514301916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:18:46.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:46 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:18:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:46 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T08:18:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:46 smithi146 ceph-mon[27805]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T08:18:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:46 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:46 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1514301916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:18:46.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:46 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:18:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:46 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T08:18:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:46 smithi177 ceph-mon[33046]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T08:18:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:46 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:46 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1514301916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:18:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:46 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:18:46.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:46.892+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f754c02d620 con 0x7f754c104280 2024-04-24T08:18:47.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.454+0000 7f7542a00640 1 -- 172.21.15.146:0/2167817297 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f753c09d0e0 con 0x7f754c104280 2024-04-24T08:18:47.455 INFO:teuthology.orchestra.run.smithi146.stderr:new fs with metadata pool 2 and data pool 3 2024-04-24T08:18:47.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.459+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f75300a3c30 msgr2=0x7f75300a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.459+0000 7f7551200640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f75300a3c30 0x7f75300a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f754c19ba00 tx=0x7f7534005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:47.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.460+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c104280 msgr2=0x7f754c19a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.460+0000 7f7551200640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c104280 0x7f754c19a4f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9a0 tx=0x7f753c00ee60 comp rx=0 tx=0).stop 2024-04-24T08:18:47.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.461+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 shutdown_connections 2024-04-24T08:18:47.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.461+0000 7f7551200640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f75300a3c30 0x7f75300a60f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.461+0000 7f7551200640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754c105e30 0x7f754c1a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.461+0000 7f7551200640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f754c105470 0x7f754c19aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.461+0000 7f7551200640 1 --2- 172.21.15.146:0/2167817297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f754c104280 0x7f754c19a4f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.461+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 >> 172.21.15.146:0/2167817297 conn(0x7f754c0ff7f0 msgr2=0x7f754c108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.462+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 shutdown_connections 2024-04-24T08:18:47.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.462+0000 7f7551200640 1 -- 172.21.15.146:0/2167817297 wait complete. 2024-04-24T08:18:47.481 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:47.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.657+0000 7f3665c00640 1 -- 172.21.15.146:0/703966567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105450 msgr2=0x7f36601058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.657+0000 7f3665c00640 1 --2- 172.21.15.146:0/703966567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105450 0x7f36601058d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3648009860 tx=0x7f364802a340 comp rx=0 tx=0).stop 2024-04-24T08:18:47.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.658+0000 7f3665c00640 1 -- 172.21.15.146:0/703966567 shutdown_connections 2024-04-24T08:18:47.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.658+0000 7f3665c00640 1 --2- 172.21.15.146:0/703966567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105450 0x7f36601058d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.658+0000 7f3665c00640 1 --2- 172.21.15.146:0/703966567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660105e10 0x7f366010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.658+0000 7f3665c00640 1 --2- 172.21.15.146:0/703966567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3660104260 0x7f3660104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.658+0000 7f3665c00640 1 -- 172.21.15.146:0/703966567 >> 172.21.15.146:0/703966567 conn(0x7f36600ff7f0 msgr2=0x7f3660101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.658+0000 7f3665c00640 1 -- 172.21.15.146:0/703966567 shutdown_connections 2024-04-24T08:18:47.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.658+0000 7f3665c00640 1 -- 172.21.15.146:0/703966567 wait complete. 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 Processor -- start 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 -- start start 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660104260 0x7f366019a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3660105450 0x7f366019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105e10 0x7f36601a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f366010fef0 con 0x7f3660104260 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f366010fd70 con 0x7f3660105450 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3665c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3660110070 con 0x7f3660105e10 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f365ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660104260 0x7f366019a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f365ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660104260 0x7f366019a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47782/0 (socket says 172.21.15.146:47782) 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f365ea00640 1 -- 172.21.15.146:0/202060900 learned_addr learned my addr 172.21.15.146:0/202060900 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f3657e00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3660105450 0x7f366019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.659+0000 7f365f400640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105e10 0x7f36601a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f365ea00640 1 -- 172.21.15.146:0/202060900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105e10 msgr2=0x7f36601a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f365ea00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105e10 0x7f36601a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f365ea00640 1 -- 172.21.15.146:0/202060900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3660105450 msgr2=0x7f366019aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f365ea00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3660105450 0x7f366019aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f365ea00640 1 -- 172.21.15.146:0/202060900 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36601a2190 con 0x7f3660104260 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f365f400640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105e10 0x7f36601a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f3657e00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3660105450 0x7f366019aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f365ea00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660104260 0x7f366019a570 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f365000e9c0 tx=0x7f365000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f3657400640 1 -- 172.21.15.146:0/202060900 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365000cd80 con 0x7f3660104260 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36601a2480 con 0x7f3660104260 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f3657400640 1 -- 172.21.15.146:0/202060900 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36500035f0 con 0x7f3660104260 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f3657400640 1 -- 172.21.15.146:0/202060900 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650010690 con 0x7f3660104260 2024-04-24T08:18:47.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.660+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36601a2950 con 0x7f3660104260 2024-04-24T08:18:47.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.662+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f366010b580 con 0x7f3660104260 2024-04-24T08:18:47.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.664+0000 7f3657400640 1 -- 172.21.15.146:0/202060900 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f36500037a0 con 0x7f3660104260 2024-04-24T08:18:47.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.665+0000 7f3657400640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36440a3bb0 0x7f36440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.665+0000 7f3657400640 1 -- 172.21.15.146:0/202060900 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f36500d0800 con 0x7f3660104260 2024-04-24T08:18:47.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.665+0000 7f3657e00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36440a3bb0 0x7f36440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.666+0000 7f3657e00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36440a3bb0 0x7f36440a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3648009830 tx=0x7f3648030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.669+0000 7f3657400640 1 -- 172.21.15.146:0/202060900 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f365009d290 con 0x7f3660104260 2024-04-24T08:18:47.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T08:18:47.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T08:18:47.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: pgmap v311: 97 pgs: 9 creating+peering, 87 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2167817297' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:47 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T08:18:47.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T08:18:47.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: pgmap v311: 97 pgs: 9 creating+peering, 87 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:47.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2167817297' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:18:47.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:18:47.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-24T08:18:47.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T08:18:47.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:47 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: pgmap v311: 97 pgs: 9 creating+peering, 87 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2167817297' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:47 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:47.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.873+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f366002d620 con 0x7f3660104260 2024-04-24T08:18:47.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.876+0000 7f3657400640 1 -- 172.21.15.146:0/202060900 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22275 (secure 0 0 0) 0x7f365009cb90 con 0x7f3660104260 2024-04-24T08:18:47.877 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:47.877 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":87,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:46.894856+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36440a3bb0 msgr2=0x7f36440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36440a3bb0 0x7f36440a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3648009830 tx=0x7f3648030550 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660104260 msgr2=0x7f366019a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660104260 0x7f366019a570 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f365000e9c0 tx=0x7f365000ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 shutdown_connections 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3660105e10 0x7f36601a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36440a3bb0 0x7f36440a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3660105450 0x7f366019aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 --2- 172.21.15.146:0/202060900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3660104260 0x7f366019a570 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 >> 172.21.15.146:0/202060900 conn(0x7f36600ff7f0 msgr2=0x7f3660108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 shutdown_connections 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:47.879+0000 7f3665c00640 1 -- 172.21.15.146:0/202060900 wait complete. 2024-04-24T08:18:47.899 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:48.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.072+0000 7f5441400640 1 -- 172.21.15.146:0/3390197305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 msgr2=0x7f543c1058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.072+0000 7f5441400640 1 --2- 172.21.15.146:0/3390197305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 0x7f543c1058d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5424009900 tx=0x7f542402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.072+0000 7f5441400640 1 -- 172.21.15.146:0/3390197305 shutdown_connections 2024-04-24T08:18:48.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.072+0000 7f5441400640 1 --2- 172.21.15.146:0/3390197305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c105e10 0x7f543c10c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.072+0000 7f5441400640 1 --2- 172.21.15.146:0/3390197305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 0x7f543c1058d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.072+0000 7f5441400640 1 --2- 172.21.15.146:0/3390197305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f543c104260 0x7f543c104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.072+0000 7f5441400640 1 -- 172.21.15.146:0/3390197305 >> 172.21.15.146:0/3390197305 conn(0x7f543c0ff7f0 msgr2=0x7f543c101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.073+0000 7f5441400640 1 -- 172.21.15.146:0/3390197305 shutdown_connections 2024-04-24T08:18:48.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.073+0000 7f5441400640 1 -- 172.21.15.146:0/3390197305 wait complete. 2024-04-24T08:18:48.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 Processor -- start 2024-04-24T08:18:48.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 -- start start 2024-04-24T08:18:48.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c104260 0x7f543c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 0x7f543c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f543c105e10 0x7f543c1a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f543c10ff40 con 0x7f543c105e10 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f543c10fdc0 con 0x7f543c105450 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5441400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f543c1100c0 con 0x7f543c104260 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f543a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c104260 0x7f543c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f543b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f543c105e10 0x7f543c1a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f543a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c104260 0x7f543c19a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:49462/0 (socket says 172.21.15.146:49462) 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f543a800640 1 -- 172.21.15.146:0/3613017473 learned_addr learned my addr 172.21.15.146:0/3613017473 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.074+0000 7f5431e00640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 0x7f543c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f543a800640 1 -- 172.21.15.146:0/3613017473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 msgr2=0x7f543c19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f543a800640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 0x7f543c19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f543a800640 1 -- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f543c105e10 msgr2=0x7f543c1a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f543a800640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f543c105e10 0x7f543c1a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f543a800640 1 -- 172.21.15.146:0/3613017473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c1a2190 con 0x7f543c104260 2024-04-24T08:18:48.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f543a800640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c104260 0x7f543c19a570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f542c00ea80 tx=0x7f542c00ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f5433e00640 1 -- 172.21.15.146:0/3613017473 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c00ce40 con 0x7f543c104260 2024-04-24T08:18:48.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f5433e00640 1 -- 172.21.15.146:0/3613017473 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c011950 con 0x7f543c104260 2024-04-24T08:18:48.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543c1a2480 con 0x7f543c104260 2024-04-24T08:18:48.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.075+0000 7f5433e00640 1 -- 172.21.15.146:0/3613017473 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c01ab80 con 0x7f543c104260 2024-04-24T08:18:48.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.076+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f543c1a29a0 con 0x7f543c104260 2024-04-24T08:18:48.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.077+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543c10b580 con 0x7f543c104260 2024-04-24T08:18:48.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.078+0000 7f5433e00640 1 -- 172.21.15.146:0/3613017473 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f542c011b00 con 0x7f543c104260 2024-04-24T08:18:48.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.080+0000 7f5433e00640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54200a3bb0 0x7f54200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.080+0000 7f5433e00640 1 -- 172.21.15.146:0/3613017473 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f542c0d08e0 con 0x7f543c104260 2024-04-24T08:18:48.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.080+0000 7f5431e00640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54200a3bb0 0x7f54200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.081+0000 7f5431e00640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54200a3bb0 0x7f54200a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5424031440 tx=0x7f54240315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.082+0000 7f5433e00640 1 -- 172.21.15.146:0/3613017473 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f542c016030 con 0x7f543c104260 2024-04-24T08:18:48.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.289+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f543c062c00 con 0x7f543c104260 2024-04-24T08:18:48.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.296+0000 7f5433e00640 1 -- 172.21.15.146:0/3613017473 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22262 (secure 0 0 0) 0x7f542c09d370 con 0x7f543c104260 2024-04-24T08:18:48.297 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:48.298 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":88,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:48.273578+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:48.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.299+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54200a3bb0 msgr2=0x7f54200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.299+0000 7f5441400640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54200a3bb0 0x7f54200a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5424031440 tx=0x7f54240315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.299+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c104260 msgr2=0x7f543c19a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.299+0000 7f5441400640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c104260 0x7f543c19a570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f542c00ea80 tx=0x7f542c00ef40 comp rx=0 tx=0).stop 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 shutdown_connections 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f54200a3bb0 0x7f54200a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f543c105e10 0x7f543c1a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f543c105450 0x7f543c19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 --2- 172.21.15.146:0/3613017473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f543c104260 0x7f543c19a570 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 >> 172.21.15.146:0/3613017473 conn(0x7f543c0ff7f0 msgr2=0x7f543c108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 shutdown_connections 2024-04-24T08:18:48.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.300+0000 7f5441400640 1 -- 172.21.15.146:0/3613017473 wait complete. 2024-04-24T08:18:48.321 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:18:48.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.499+0000 7f642ee00640 1 -- 172.21.15.146:0/2986211043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6428108540 msgr2=0x7f6428108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.499+0000 7f642ee00640 1 --2- 172.21.15.146:0/2986211043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6428108540 0x7f6428108960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f641c009900 tx=0x7f641c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.499+0000 7f642ee00640 1 -- 172.21.15.146:0/2986211043 shutdown_connections 2024-04-24T08:18:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.500+0000 7f642ee00640 1 --2- 172.21.15.146:0/2986211043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428109730 0x7f6428109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.500+0000 7f642ee00640 1 --2- 172.21.15.146:0/2986211043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f642810a0f0 0x7f6428110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.500+0000 7f642ee00640 1 --2- 172.21.15.146:0/2986211043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6428108540 0x7f6428108960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.500+0000 7f642ee00640 1 -- 172.21.15.146:0/2986211043 >> 172.21.15.146:0/2986211043 conn(0x7f64280774d0 msgr2=0x7f6428079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.500+0000 7f642ee00640 1 -- 172.21.15.146:0/2986211043 shutdown_connections 2024-04-24T08:18:48.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.500+0000 7f642ee00640 1 -- 172.21.15.146:0/2986211043 wait complete. 2024-04-24T08:18:48.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 Processor -- start 2024-04-24T08:18:48.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 -- start start 2024-04-24T08:18:48.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428108540 0x7f642819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6428109730 0x7f642819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642810a0f0 0x7f64281a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f64281141d0 con 0x7f642810a0f0 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6428114050 con 0x7f6428109730 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642ee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6428114350 con 0x7f6428108540 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428108540 0x7f642819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428108540 0x7f642819e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:49472/0 (socket says 172.21.15.146:49472) 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642810a0f0 0x7f64281a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642da00640 1 -- 172.21.15.146:0/2097495982 learned_addr learned my addr 172.21.15.146:0/2097495982 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.501+0000 7f642d000640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6428109730 0x7f642819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642d000640 1 -- 172.21.15.146:0/2097495982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428108540 msgr2=0x7f642819e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642d000640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428108540 0x7f642819e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642d000640 1 -- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642810a0f0 msgr2=0x7f64281a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642d000640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642810a0f0 0x7f64281a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642d000640 1 -- 172.21.15.146:0/2097495982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64281a6450 con 0x7f6428109730 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642da00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428108540 0x7f642819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642d000640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6428109730 0x7f642819edc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f641800e9f0 tx=0x7f641800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f6426000640 1 -- 172.21.15.146:0/2097495982 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641800ccc0 con 0x7f6428109730 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f6426000640 1 -- 172.21.15.146:0/2097495982 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6418005020 con 0x7f6428109730 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64281a6740 con 0x7f6428109730 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f6426000640 1 -- 172.21.15.146:0/2097495982 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180049e0 con 0x7f6428109730 2024-04-24T08:18:48.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.502+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64281a6c90 con 0x7f6428109730 2024-04-24T08:18:48.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.503+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f642810f860 con 0x7f6428109730 2024-04-24T08:18:48.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.505+0000 7f6426000640 1 -- 172.21.15.146:0/2097495982 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6418005560 con 0x7f6428109730 2024-04-24T08:18:48.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.507+0000 7f6426000640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64100a3c30 0x7f64100a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.507+0000 7f6426000640 1 -- 172.21.15.146:0/2097495982 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f64180d0230 con 0x7f6428109730 2024-04-24T08:18:48.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.507+0000 7f642da00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64100a3c30 0x7f64100a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.508+0000 7f642da00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64100a3c30 0x7f64100a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f641c000c00 tx=0x7f641c0046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.511+0000 7f6426000640 1 -- 172.21.15.146:0/2097495982 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64180051d0 con 0x7f6428109730 2024-04-24T08:18:48.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/202060900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3613017473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/202060900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3613017473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:48.756+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f642802d620 con 0x7f6428109730 2024-04-24T08:18:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T08:18:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T08:18:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-24T08:18:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-24T08:18:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/202060900' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T08:18:48.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3613017473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:48.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:48.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:48.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:18:49.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.471+0000 7f6426000640 1 -- 172.21.15.146:0/2097495982 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f641809ccc0 con 0x7f6428109730 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64100a3c30 msgr2=0x7f64100a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64100a3c30 0x7f64100a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f641c000c00 tx=0x7f641c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6428109730 msgr2=0x7f642819edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6428109730 0x7f642819edc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f641800e9f0 tx=0x7f641800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 shutdown_connections 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64100a3c30 0x7f64100a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f642810a0f0 0x7f64281a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6428109730 0x7f642819edc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 --2- 172.21.15.146:0/2097495982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6428108540 0x7f642819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 >> 172.21.15.146:0/2097495982 conn(0x7f64280774d0 msgr2=0x7f642810ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 shutdown_connections 2024-04-24T08:18:49.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.475+0000 7f642ee00640 1 -- 172.21.15.146:0/2097495982 wait complete. 2024-04-24T08:18:49.498 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-24T08:18:49.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-24T08:18:49.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.707+0000 7f8969000640 1 -- 172.21.15.146:0/1168460981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f896410a0f0 msgr2=0x7f8964110980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.707+0000 7f8969000640 1 --2- 172.21.15.146:0/1168460981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f896410a0f0 0x7f8964110980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f895c009900 tx=0x7f895c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.708+0000 7f8969000640 1 -- 172.21.15.146:0/1168460981 shutdown_connections 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.708+0000 7f8969000640 1 --2- 172.21.15.146:0/1168460981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8964109730 0x7f8964109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.708+0000 7f8969000640 1 --2- 172.21.15.146:0/1168460981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f896410a0f0 0x7f8964110980 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.708+0000 7f8969000640 1 --2- 172.21.15.146:0/1168460981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8964108540 0x7f8964108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.708+0000 7f8969000640 1 -- 172.21.15.146:0/1168460981 >> 172.21.15.146:0/1168460981 conn(0x7f89640774d0 msgr2=0x7f8964079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.708+0000 7f8969000640 1 -- 172.21.15.146:0/1168460981 shutdown_connections 2024-04-24T08:18:49.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.708+0000 7f8969000640 1 -- 172.21.15.146:0/1168460981 wait complete. 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 Processor -- start 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 -- start start 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8964108540 0x7f896419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8964109730 0x7f896419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f896410a0f0 0x7f89641a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f89641141a0 con 0x7f8964108540 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8964114020 con 0x7f896410a0f0 2024-04-24T08:18:49.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8969000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8964114320 con 0x7f8964109730 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.709+0000 7f8963400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8964108540 0x7f896419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8964108540 0x7f896419e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47852/0 (socket says 172.21.15.146:47852) 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 -- 172.21.15.146:0/1474783904 learned_addr learned my addr 172.21.15.146:0/1474783904 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8962a00640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8964109730 0x7f896419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963e00640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f896410a0f0 0x7f89641a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 -- 172.21.15.146:0/1474783904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8964109730 msgr2=0x7f896419ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8964109730 0x7f896419ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 -- 172.21.15.146:0/1474783904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f896410a0f0 msgr2=0x7f89641a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f896410a0f0 0x7f89641a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 -- 172.21.15.146:0/1474783904 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89641a63e0 con 0x7f8964108540 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963e00640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f896410a0f0 0x7f89641a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f8963400640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8964108540 0x7f896419e7c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8954009d20 tx=0x7f895400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.710+0000 7f895be00640 1 -- 172.21.15.146:0/1474783904 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954010630 con 0x7f8964108540 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.711+0000 7f895be00640 1 -- 172.21.15.146:0/1474783904 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8954010cb0 con 0x7f8964108540 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.711+0000 7f895be00640 1 -- 172.21.15.146:0/1474783904 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954018de0 con 0x7f8964108540 2024-04-24T08:18:49.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.711+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89641a66d0 con 0x7f8964108540 2024-04-24T08:18:49.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.711+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89641a6c20 con 0x7f8964108540 2024-04-24T08:18:49.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: pgmap v314: 97 pgs: 15 creating+peering, 65 unknown, 17 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:49.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] up:active 2024-04-24T08:18:49.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:49.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:49.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2097495982' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:18:49.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:18:49.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-24T08:18:49.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:49 smithi153 ceph-mon[32746]: Cluster is now healthy 2024-04-24T08:18:49.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.712+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f896410f860 con 0x7f8964108540 2024-04-24T08:18:49.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.714+0000 7f895be00640 1 -- 172.21.15.146:0/1474783904 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f895401f020 con 0x7f8964108540 2024-04-24T08:18:49.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.715+0000 7f895be00640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89480a3bb0 0x7f89480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.715+0000 7f895be00640 1 -- 172.21.15.146:0/1474783904 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f89540d0880 con 0x7f8964108540 2024-04-24T08:18:49.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.715+0000 7f8962a00640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89480a3bb0 0x7f89480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.716+0000 7f8962a00640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89480a3bb0 0x7f89480a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89500095f0 tx=0x7f8950005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.720+0000 7f895be00640 1 -- 172.21.15.146:0/1474783904 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f895409d310 con 0x7f8964108540 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: pgmap v314: 97 pgs: 15 creating+peering, 65 unknown, 17 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] up:active 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2097495982' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-24T08:18:49.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:49 smithi146 ceph-mon[27805]: Cluster is now healthy 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: pgmap v314: 97 pgs: 15 creating+peering, 65 unknown, 17 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] up:active 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2097495982' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-24T08:18:49.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:49 smithi177 ceph-mon[33046]: Cluster is now healthy 2024-04-24T08:18:49.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:49.966+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f8964062c00 con 0x7f8964108540 2024-04-24T08:18:50.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.480+0000 7f895be00640 1 -- 172.21.15.146:0/1474783904 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f895409cc10 con 0x7f8964108540 2024-04-24T08:18:50.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89480a3bb0 msgr2=0x7f89480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89480a3bb0 0x7f89480a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89500095f0 tx=0x7f8950005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8964108540 msgr2=0x7f896419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8964108540 0x7f896419e7c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8954009d20 tx=0x7f895400c6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 shutdown_connections 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8964109730 0x7f896419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89480a3bb0 0x7f89480a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f896410a0f0 0x7f89641a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 --2- 172.21.15.146:0/1474783904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8964108540 0x7f896419e7c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 >> 172.21.15.146:0/1474783904 conn(0x7f89640774d0 msgr2=0x7f896410ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 shutdown_connections 2024-04-24T08:18:50.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.484+0000 7f8969000640 1 -- 172.21.15.146:0/1474783904 wait complete. 2024-04-24T08:18:50.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-24T08:18:50.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.697+0000 7f5107c00640 1 -- 172.21.15.146:0/3001379631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 msgr2=0x7f51001058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.697+0000 7f5107c00640 1 --2- 172.21.15.146:0/3001379631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 0x7f51001058d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009900 tx=0x7f50f002a410 comp rx=0 tx=0).stop 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.698+0000 7f5107c00640 1 -- 172.21.15.146:0/3001379631 shutdown_connections 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.698+0000 7f5107c00640 1 --2- 172.21.15.146:0/3001379631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5100105e10 0x7f510010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.698+0000 7f5107c00640 1 --2- 172.21.15.146:0/3001379631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 0x7f51001058d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.698+0000 7f5107c00640 1 --2- 172.21.15.146:0/3001379631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5100104260 0x7f5100104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.698+0000 7f5107c00640 1 -- 172.21.15.146:0/3001379631 >> 172.21.15.146:0/3001379631 conn(0x7f51000ff7f0 msgr2=0x7f5100101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.698+0000 7f5107c00640 1 -- 172.21.15.146:0/3001379631 shutdown_connections 2024-04-24T08:18:50.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.698+0000 7f5107c00640 1 -- 172.21.15.146:0/3001379631 wait complete. 2024-04-24T08:18:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 Processor -- start 2024-04-24T08:18:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 -- start start 2024-04-24T08:18:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5100104260 0x7f510019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 0x7f510019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5100105e10 0x7f51001a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f510010ff40 con 0x7f5100105450 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f510010fdc0 con 0x7f5100105e10 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5107c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51001100c0 con 0x7f5100104260 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f50ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 0x7f510019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f5105200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5100104260 0x7f510019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f50ffe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 0x7f510019aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47866/0 (socket says 172.21.15.146:47866) 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.699+0000 7f50ffe00640 1 -- 172.21.15.146:0/1192522834 learned_addr learned my addr 172.21.15.146:0/1192522834 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5100105e10 0x7f51001a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105200640 1 -- 172.21.15.146:0/1192522834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5100105e10 msgr2=0x7f51001a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105200640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5100105e10 0x7f51001a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105200640 1 -- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 msgr2=0x7f510019aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105200640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 0x7f510019aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105200640 1 -- 172.21.15.146:0/1192522834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51001a2160 con 0x7f5100104260 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5100105e10 0x7f51001a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:50.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5105200640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5100104260 0x7f510019a540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f50e8005fd0 tx=0x7f50e800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f50fd600640 1 -- 172.21.15.146:0/1192522834 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8003970 con 0x7f5100104260 2024-04-24T08:18:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.700+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51001a2450 con 0x7f5100104260 2024-04-24T08:18:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.701+0000 7f50fd600640 1 -- 172.21.15.146:0/1192522834 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50e8004030 con 0x7f5100104260 2024-04-24T08:18:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.701+0000 7f50fd600640 1 -- 172.21.15.146:0/1192522834 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e80107f0 con 0x7f5100104260 2024-04-24T08:18:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.701+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51001a29f0 con 0x7f5100104260 2024-04-24T08:18:50.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.702+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f510010b520 con 0x7f5100104260 2024-04-24T08:18:50.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.704+0000 7f50fd600640 1 -- 172.21.15.146:0/1192522834 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f50e80041e0 con 0x7f5100104260 2024-04-24T08:18:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.705+0000 7f50fd600640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f50e40a3bb0 0x7f50e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.705+0000 7f50fd600640 1 -- 172.21.15.146:0/1192522834 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f50e80d0e10 con 0x7f5100104260 2024-04-24T08:18:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.706+0000 7f50ffe00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f50e40a3bb0 0x7f50e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.706+0000 7f50ffe00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f50e40a3bb0 0x7f50e40a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f510019ba50 tx=0x7f50f00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.708+0000 7f50fd600640 1 -- 172.21.15.146:0/1192522834 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50e809d8a0 con 0x7f5100104260 2024-04-24T08:18:50.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:50 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T08:18:50.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:50 smithi146 ceph-mon[27805]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:50.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:50 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1474783904' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:18:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:50 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T08:18:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:50 smithi177 ceph-mon[33046]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:50 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1474783904' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:18:50.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:50.951+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f510002d620 con 0x7f5100104260 2024-04-24T08:18:50.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:50 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T08:18:50.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:50 smithi153 ceph-mon[32746]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:50.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:50 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1474783904' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:18:51.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.500+0000 7f50fd600640 1 -- 172.21.15.146:0/1192522834 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f50e809d1a0 con 0x7f5100104260 2024-04-24T08:18:51.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.504+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f50e40a3bb0 msgr2=0x7f50e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.504+0000 7f5107c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f50e40a3bb0 0x7f50e40a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f510019ba50 tx=0x7f50f00315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5100104260 msgr2=0x7f510019a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5100104260 0x7f510019a540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f50e8005fd0 tx=0x7f50e800ef90 comp rx=0 tx=0).stop 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 shutdown_connections 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f50e40a3bb0 0x7f50e40a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5100105e10 0x7f51001a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5100105450 0x7f510019aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 --2- 172.21.15.146:0/1192522834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5100104260 0x7f510019a540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 >> 172.21.15.146:0/1192522834 conn(0x7f51000ff7f0 msgr2=0x7f51001086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 shutdown_connections 2024-04-24T08:18:51.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.505+0000 7f5107c00640 1 -- 172.21.15.146:0/1192522834 wait complete. 2024-04-24T08:18:51.533 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 -- 172.21.15.146:0/1885077303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 msgr2=0x7f46f41058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1885077303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f41058d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009900 tx=0x7f46dc02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 -- 172.21.15.146:0/1885077303 shutdown_connections 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1885077303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46f4105e10 0x7f46f410c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1885077303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f41058d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1885077303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104260 0x7f46f4104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 -- 172.21.15.146:0/1885077303 >> 172.21.15.146:0/1885077303 conn(0x7f46f40ff7f0 msgr2=0x7f46f4101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:51.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 -- 172.21.15.146:0/1885077303 shutdown_connections 2024-04-24T08:18:51.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.713+0000 7f46f8a00640 1 -- 172.21.15.146:0/1885077303 wait complete. 2024-04-24T08:18:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.714+0000 7f46f8a00640 1 Processor -- start 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.714+0000 7f46f8a00640 1 -- start start 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.714+0000 7f46f8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104260 0x7f46f419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46f8a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46f8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46f4105e10 0x7f46f41a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46f8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410ff40 con 0x7f46f4105e10 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46f8a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410fdc0 con 0x7f46f4105450 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46f8a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46f41100c0 con 0x7f46f4104260 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46f2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46f4105e10 0x7f46f41a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f419aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54724/0 (socket says 172.21.15.146:54724) 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 -- 172.21.15.146:0/1254561525 learned_addr learned my addr 172.21.15.146:0/1254561525 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46f1e00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104260 0x7f46f419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 -- 172.21.15.146:0/1254561525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104260 msgr2=0x7f46f419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104260 0x7f46f419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 -- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46f4105e10 msgr2=0x7f46f41a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46f4105e10 0x7f46f41a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 -- 172.21.15.146:0/1254561525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f41a21d0 con 0x7f46f4105450 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.715+0000 7f46ebe00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f419aaf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46dc031440 tx=0x7f46dc032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.716+0000 7f46eaa00640 1 -- 172.21.15.146:0/1254561525 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc03f030 con 0x7f46f4105450 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.716+0000 7f46eaa00640 1 -- 172.21.15.146:0/1254561525 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46dc003ce0 con 0x7f46f4105450 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.716+0000 7f46eaa00640 1 -- 172.21.15.146:0/1254561525 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46dc035070 con 0x7f46f4105450 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.716+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f41a2460 con 0x7f46f4105450 2024-04-24T08:18:51.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.716+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f41a29a0 con 0x7f46f4105450 2024-04-24T08:18:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.717+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f410b520 con 0x7f46f4105450 2024-04-24T08:18:51.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.719+0000 7f46eaa00640 1 -- 172.21.15.146:0/1254561525 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f46dc042030 con 0x7f46f4105450 2024-04-24T08:18:51.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.720+0000 7f46eaa00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f46d80a3c30 0x7f46d80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:51.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.720+0000 7f46eaa00640 1 -- 172.21.15.146:0/1254561525 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f46dc0f0840 con 0x7f46f4105450 2024-04-24T08:18:51.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.720+0000 7f46f1e00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f46d80a3c30 0x7f46d80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:51.721 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.721+0000 7f46f1e00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f46d80a3c30 0x7f46d80a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46e4005d00 tx=0x7f46e4005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:51.725 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.724+0000 7f46eaa00640 1 -- 172.21.15.146:0/1254561525 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46dc0bd2d0 con 0x7f46f4105450 2024-04-24T08:18:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: pgmap v315: 97 pgs: 15 creating+peering, 28 unknown, 54 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-24T08:18:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T08:18:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1474783904' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-24T08:18:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T08:18:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:51.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-24T08:18:51.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-24T08:18:51.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1192522834' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:18:51.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:51 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: pgmap v315: 97 pgs: 15 creating+peering, 28 unknown, 54 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1474783904' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1192522834' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:18:51.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:51 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:18:51.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: pgmap v315: 97 pgs: 15 creating+peering, 28 unknown, 54 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1474783904' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1192522834' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:18:51.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:51 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:18:51.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:51.968+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f46f4062c00 con 0x7f46f4105450 2024-04-24T08:18:52.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.513+0000 7f46eaa00640 1 -- 172.21.15.146:0/1254561525 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f46dc0bcbd0 con 0x7f46f4105450 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f46d80a3c30 msgr2=0x7f46d80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f46d80a3c30 0x7f46d80a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f46e4005d00 tx=0x7f46e4005c50 comp rx=0 tx=0).stop 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 msgr2=0x7f46f419aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f419aaf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46dc031440 tx=0x7f46dc032ce0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 shutdown_connections 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f46d80a3c30 0x7f46d80a60f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46f4105e10 0x7f46f41a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46f4105450 0x7f46f419aaf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 --2- 172.21.15.146:0/1254561525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104260 0x7f46f419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 >> 172.21.15.146:0/1254561525 conn(0x7f46f40ff7f0 msgr2=0x7f46f41086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 shutdown_connections 2024-04-24T08:18:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.519+0000 7f46f8a00640 1 -- 172.21.15.146:0/1254561525 wait complete. 2024-04-24T08:18:52.542 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-24T08:18:52.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-24T08:18:52.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 -- 172.21.15.146:0/4175770181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa048101880 msgr2=0x7fa048101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 --2- 172.21.15.146:0/4175770181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa048101880 0x7fa048101d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa034009900 tx=0x7fa03402a410 comp rx=0 tx=0).stop 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 -- 172.21.15.146:0/4175770181 shutdown_connections 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 --2- 172.21.15.146:0/4175770181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa048106580 0x7fa048108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 --2- 172.21.15.146:0/4175770181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa048101880 0x7fa048101d00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 --2- 172.21.15.146:0/4175770181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480684b0 0x7fa048101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 -- 172.21.15.146:0/4175770181 >> 172.21.15.146:0/4175770181 conn(0x7fa0480fb6c0 msgr2=0x7fa0480fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 -- 172.21.15.146:0/4175770181 shutdown_connections 2024-04-24T08:18:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.737+0000 7fa047e00640 1 -- 172.21.15.146:0/4175770181 wait complete. 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 Processor -- start 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 -- start start 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0480684b0 0x7fa04819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa048101880 0x7fa04819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048106580 0x7fa0481a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa04810dbe0 con 0x7fa0480684b0 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa04810da60 con 0x7fa048106580 2024-04-24T08:18:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa04810dd60 con 0x7fa048101880 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa046a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0480684b0 0x7fa04819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa046a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0480684b0 0x7fa04819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59216/0 (socket says 172.21.15.146:59216) 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa047400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048106580 0x7fa0481a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa046a00640 1 -- 172.21.15.146:0/1632305018 learned_addr learned my addr 172.21.15.146:0/1632305018 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.738+0000 7fa046000640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa048101880 0x7fa04819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa046a00640 1 -- 172.21.15.146:0/1632305018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa048101880 msgr2=0x7fa04819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa046a00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa048101880 0x7fa04819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa046a00640 1 -- 172.21.15.146:0/1632305018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048106580 msgr2=0x7fa0481a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa046a00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048106580 0x7fa0481a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa046a00640 1 -- 172.21.15.146:0/1632305018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0481a20f0 con 0x7fa0480684b0 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa047400640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048106580 0x7fa0481a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa046a00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0480684b0 0x7fa04819a4d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa03800e9c0 tx=0x7fa03800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa03f400640 1 -- 172.21.15.146:0/1632305018 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03800cde0 con 0x7fa0480684b0 2024-04-24T08:18:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa03f400640 1 -- 172.21.15.146:0/1632305018 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa038010910 con 0x7fa0480684b0 2024-04-24T08:18:52.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa03f400640 1 -- 172.21.15.146:0/1632305018 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038018a00 con 0x7fa0480684b0 2024-04-24T08:18:52.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0481a23e0 con 0x7fa0480684b0 2024-04-24T08:18:52.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.739+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0481a28b0 con 0x7fa0480684b0 2024-04-24T08:18:52.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.741+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0481a2b10 con 0x7fa0480684b0 2024-04-24T08:18:52.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.742+0000 7fa03f400640 1 -- 172.21.15.146:0/1632305018 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa03801f020 con 0x7fa0480684b0 2024-04-24T08:18:52.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.745+0000 7fa03f400640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa02c0a3c30 0x7fa02c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:52.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.745+0000 7fa03f400640 1 -- 172.21.15.146:0/1632305018 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fa0380d04a0 con 0x7fa0480684b0 2024-04-24T08:18:52.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.745+0000 7fa046000640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa02c0a3c30 0x7fa02c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:52.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.746+0000 7fa046000640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa02c0a3c30 0x7fa02c0a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa034009230 tx=0x7fa0340315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:52.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.749+0000 7fa03f400640 1 -- 172.21.15.146:0/1632305018 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa03809cf30 con 0x7fa0480684b0 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] up:active 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: Cluster is now healthy 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1254561525' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:18:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:52 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:18:52.960 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: 2024-04-24T08:18:52.585+0000 7f8bddeba700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: 2024-04-24T08:18:52.585+0000 7f8bddeba700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: 2024-04-24T08:18:52.585+0000 7f8bddeba700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] up:active 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: Cluster is now healthy 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-24T08:18:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-04-24T08:18:52.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-24T08:18:52.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1254561525' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:18:52.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:52 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:18:52.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:52.986+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa0480fd4e0 con 0x7fa02c0a3c30 2024-04-24T08:18:52.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] up:active 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: Cluster is now healthy 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1254561525' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:18:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:18:53.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.016+0000 7fa03f400640 1 -- 172.21.15.146:0/1632305018 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa0480fd4e0 con 0x7fa02c0a3c30 2024-04-24T08:18:53.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa02c0a3c30 msgr2=0x7fa02c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa02c0a3c30 0x7fa02c0a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa034009230 tx=0x7fa0340315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0480684b0 msgr2=0x7fa04819a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0480684b0 0x7fa04819a4d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa03800e9c0 tx=0x7fa03800ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 shutdown_connections 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa048101880 0x7fa04819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa02c0a3c30 0x7fa02c0a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa048106580 0x7fa0481a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 --2- 172.21.15.146:0/1632305018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0480684b0 0x7fa04819a4d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 >> 172.21.15.146:0/1632305018 conn(0x7fa0480fb6c0 msgr2=0x7fa0480fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 shutdown_connections 2024-04-24T08:18:53.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.019+0000 7fa047e00640 1 -- 172.21.15.146:0/1632305018 wait complete. 2024-04-24T08:18:53.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 -- 172.21.15.146:0/1722247182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 msgr2=0x7fe170076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 --2- 172.21.15.146:0/1722247182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe170076de0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe160009900 tx=0x7fe16002a410 comp rx=0 tx=0).stop 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 -- 172.21.15.146:0/1722247182 shutdown_connections 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 --2- 172.21.15.146:0/1722247182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe170077c00 0x7fe170078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 --2- 172.21.15.146:0/1722247182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1700785a0 0x7fe170078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 --2- 172.21.15.146:0/1722247182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe170076de0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 -- 172.21.15.146:0/1722247182 >> 172.21.15.146:0/1722247182 conn(0x7fe1700ff6b0 msgr2=0x7fe170101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 -- 172.21.15.146:0/1722247182 shutdown_connections 2024-04-24T08:18:53.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.228+0000 7fe16f200640 1 -- 172.21.15.146:0/1722247182 wait complete. 2024-04-24T08:18:53.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 Processor -- start 2024-04-24T08:18:53.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 -- start start 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe17019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe170077c00 0x7fe17019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1700785a0 0x7fe1701a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe170114270 con 0x7fe170077c00 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701140f0 con 0x7fe1700769c0 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.229+0000 7fe16f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe1701143f0 con 0x7fe1700785a0 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe17019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe17019e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54794/0 (socket says 172.21.15.146:54794) 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 -- 172.21.15.146:0/3313437325 learned_addr learned my addr 172.21.15.146:0/3313437325 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe167e00640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe170077c00 0x7fe17019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16e800640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1700785a0 0x7fe1701a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 -- 172.21.15.146:0/3313437325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1700785a0 msgr2=0x7fe1701a5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1700785a0 0x7fe1701a5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 -- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe170077c00 msgr2=0x7fe17019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe170077c00 0x7fe17019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 -- 172.21.15.146:0/3313437325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1701a64d0 con 0x7fe1700769c0 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.230+0000 7fe16de00640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe17019e8b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe160031440 tx=0x7fe160032e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.231+0000 7fe166a00640 1 -- 172.21.15.146:0/3313437325 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16003f030 con 0x7fe1700769c0 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.231+0000 7fe166a00640 1 -- 172.21.15.146:0/3313437325 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe160003ce0 con 0x7fe1700769c0 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.231+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1701a6760 con 0x7fe1700769c0 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.231+0000 7fe166a00640 1 -- 172.21.15.146:0/3313437325 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160035070 con 0x7fe1700769c0 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.231+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1701a6cd0 con 0x7fe1700769c0 2024-04-24T08:18:53.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.232+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe170110ab0 con 0x7fe1700769c0 2024-04-24T08:18:53.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.234+0000 7fe166a00640 1 -- 172.21.15.146:0/3313437325 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe160042030 con 0x7fe1700769c0 2024-04-24T08:18:53.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.235+0000 7fe166a00640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe1540a3bb0 0x7fe1540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.235+0000 7fe166a00640 1 -- 172.21.15.146:0/3313437325 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fe1600f0aa0 con 0x7fe1700769c0 2024-04-24T08:18:53.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.235+0000 7fe167e00640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe1540a3bb0 0x7fe1540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.236+0000 7fe167e00640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe1540a3bb0 0x7fe1540a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe158005fd0 tx=0x7fe158005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:53.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.244+0000 7fe166a00640 1 -- 172.21.15.146:0/3313437325 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1600bd530 con 0x7fe1700769c0 2024-04-24T08:18:53.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.480+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe1700624d0 con 0x7fe1540a3bb0 2024-04-24T08:18:53.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:18:52.995+0000 7f8689200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:18:53.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:18:52.995+0000 7f8689200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:18:53.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:18:52.995+0000 7f8689200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:18:53.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:18:52.995+0000 7f8689200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:18:53.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:18:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:18:52.995+0000 7f8689200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:18:53.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.520+0000 7fe166a00640 1 -- 172.21.15.146:0/3313437325 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe1700624d0 con 0x7fe1540a3bb0 2024-04-24T08:18:53.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe1540a3bb0 msgr2=0x7fe1540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe1540a3bb0 0x7fe1540a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe158005fd0 tx=0x7fe158005850 comp rx=0 tx=0).stop 2024-04-24T08:18:53.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 msgr2=0x7fe17019e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe17019e8b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe160031440 tx=0x7fe160032e20 comp rx=0 tx=0).stop 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 shutdown_connections 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1700785a0 0x7fe1701a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe1540a3bb0 0x7fe1540a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe170077c00 0x7fe17019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 --2- 172.21.15.146:0/3313437325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe1700769c0 0x7fe17019e8b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 >> 172.21.15.146:0/3313437325 conn(0x7fe1700ff6b0 msgr2=0x7fe1701011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 shutdown_connections 2024-04-24T08:18:53.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.523+0000 7fe16f200640 1 -- 172.21.15.146:0/3313437325 wait complete. 2024-04-24T08:18:53.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 -- 172.21.15.146:0/1694684843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 msgr2=0x7fbf0c109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 --2- 172.21.15.146:0/1694684843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c109bb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009860 tx=0x7fbef402a340 comp rx=0 tx=0).stop 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 -- 172.21.15.146:0/1694684843 shutdown_connections 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 --2- 172.21.15.146:0/1694684843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 0x7fbf0c110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 --2- 172.21.15.146:0/1694684843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c109bb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 --2- 172.21.15.146:0/1694684843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c108540 0x7fbf0c108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 -- 172.21.15.146:0/1694684843 >> 172.21.15.146:0/1694684843 conn(0x7fbf0c0774d0 msgr2=0x7fbf0c079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:53.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 -- 172.21.15.146:0/1694684843 shutdown_connections 2024-04-24T08:18:53.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.745+0000 7fbf0b200640 1 -- 172.21.15.146:0/1694684843 wait complete. 2024-04-24T08:18:53.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.746+0000 7fbf0b200640 1 Processor -- start 2024-04-24T08:18:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0b200640 1 -- start start 2024-04-24T08:18:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c108540 0x7fbf0c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 0x7fbf0c1a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1141d0 con 0x7fbf0c10a0f0 2024-04-24T08:18:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c114050 con 0x7fbf0c109730 2024-04-24T08:18:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c114350 con 0x7fbf0c108540 2024-04-24T08:18:53.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf01400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 0x7fbf0c1a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf09e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c108540 0x7fbf0c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf01400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c19edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54820/0 (socket says 172.21.15.146:54820) 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf01400640 1 -- 172.21.15.146:0/2653387403 learned_addr learned my addr 172.21.15.146:0/2653387403 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.747+0000 7fbf0a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 0x7fbf0c1a5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59270/0 (socket says 172.21.15.146:59270) 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf01400640 1 -- 172.21.15.146:0/2653387403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c108540 msgr2=0x7fbf0c19e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf01400640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c108540 0x7fbf0c19e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf01400640 1 -- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 msgr2=0x7fbf0c1a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf01400640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 0x7fbf0c1a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf01400640 1 -- 172.21.15.146:0/2653387403 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c1a6450 con 0x7fbf0c109730 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf09e00640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c108540 0x7fbf0c19e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf0a800640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 0x7fbf0c1a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.748+0000 7fbf01400640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c19edc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbef4035040 tx=0x7fbef4033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.749+0000 7fbf03400640 1 -- 172.21.15.146:0/2653387403 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4037040 con 0x7fbf0c109730 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.749+0000 7fbf03400640 1 -- 172.21.15.146:0/2653387403 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef4030850 con 0x7fbf0c109730 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.749+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c1a66e0 con 0x7fbf0c109730 2024-04-24T08:18:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.749+0000 7fbf03400640 1 -- 172.21.15.146:0/2653387403 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4004920 con 0x7fbf0c109730 2024-04-24T08:18:53.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.749+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c1a6bd0 con 0x7fbf0c109730 2024-04-24T08:18:53.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.750+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0c10f860 con 0x7fbf0c109730 2024-04-24T08:18:53.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.753+0000 7fbf03400640 1 -- 172.21.15.146:0/2653387403 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fbef4004050 con 0x7fbf0c109730 2024-04-24T08:18:53.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.754+0000 7fbf03400640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbef00a3bb0 0x7fbef00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:53.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.754+0000 7fbf03400640 1 -- 172.21.15.146:0/2653387403 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fbef40f02b0 con 0x7fbf0c109730 2024-04-24T08:18:53.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.754+0000 7fbf09e00640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbef00a3bb0 0x7fbef00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:53.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.755+0000 7fbf09e00640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbef00a3bb0 0x7fbef00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c109590 tx=0x7fbefc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:53.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.758+0000 7fbf03400640 1 -- 172.21.15.146:0/2653387403 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbef40bcd40 con 0x7fbf0c109730 2024-04-24T08:18:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:53 smithi177 ceph-mon[33046]: pgmap v316: 97 pgs: 97 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 12 op/s 2024-04-24T08:18:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:53 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T08:18:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:53 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] up:active 2024-04-24T08:18:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:53 smithi177 ceph-mon[33046]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-04-24T08:18:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:53 smithi177 ceph-mon[33046]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:18:53.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:53 smithi153 ceph-mon[32746]: pgmap v316: 97 pgs: 97 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 12 op/s 2024-04-24T08:18:53.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:53 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T08:18:53.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:53 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] up:active 2024-04-24T08:18:53.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:53 smithi153 ceph-mon[32746]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-04-24T08:18:53.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:53 smithi153 ceph-mon[32746]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:18:53.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:53 smithi146 ceph-mon[27805]: pgmap v316: 97 pgs: 97 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 12 op/s 2024-04-24T08:18:53.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:53 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T08:18:53.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:53 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] up:active 2024-04-24T08:18:53.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:53 smithi146 ceph-mon[27805]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-04-24T08:18:53.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:53 smithi146 ceph-mon[27805]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:18:54.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:53.999+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fbf0c0624d0 con 0x7fbef00a3bb0 2024-04-24T08:18:54.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.013+0000 7fbf03400640 1 -- 172.21.15.146:0/2653387403 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fbf0c0624d0 con 0x7fbef00a3bb0 2024-04-24T08:18:54.015 INFO:teuthology.orchestra.run.smithi146.stdout:/volumes/qa/sv_0/1c506dab-21e4-4763-a873-96c14606511b 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbef00a3bb0 msgr2=0x7fbef00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbef00a3bb0 0x7fbef00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c109590 tx=0x7fbefc005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 msgr2=0x7fbf0c19edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c19edc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbef4035040 tx=0x7fbef4033d90 comp rx=0 tx=0).stop 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 shutdown_connections 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbef00a3bb0 0x7fbef00a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c10a0f0 0x7fbf0c1a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c109730 0x7fbf0c19edc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 --2- 172.21.15.146:0/2653387403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c108540 0x7fbf0c19e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 >> 172.21.15.146:0/2653387403 conn(0x7fbf0c0774d0 msgr2=0x7fbf0c10ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:54.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.016+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 shutdown_connections 2024-04-24T08:18:54.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.017+0000 7fbf0b200640 1 -- 172.21.15.146:0/2653387403 wait complete. 2024-04-24T08:18:54.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-24T08:18:54.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.246+0000 7fb960c00640 1 -- 172.21.15.146:0/224949923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c108540 msgr2=0x7fb95c108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.246+0000 7fb960c00640 1 --2- 172.21.15.146:0/224949923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c108540 0x7fb95c108960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009900 tx=0x7fb94c02a400 comp rx=0 tx=0).stop 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.247+0000 7fb960c00640 1 -- 172.21.15.146:0/224949923 shutdown_connections 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.247+0000 7fb960c00640 1 --2- 172.21.15.146:0/224949923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb95c10a0f0 0x7fb95c110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.247+0000 7fb960c00640 1 --2- 172.21.15.146:0/224949923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 0x7fb95c109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.247+0000 7fb960c00640 1 --2- 172.21.15.146:0/224949923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c108540 0x7fb95c108960 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.247+0000 7fb960c00640 1 -- 172.21.15.146:0/224949923 >> 172.21.15.146:0/224949923 conn(0x7fb95c0774d0 msgr2=0x7fb95c079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.247+0000 7fb960c00640 1 -- 172.21.15.146:0/224949923 shutdown_connections 2024-04-24T08:18:54.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.247+0000 7fb960c00640 1 -- 172.21.15.146:0/224949923 wait complete. 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 Processor -- start 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 -- start start 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb95c108540 0x7fb95c19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 0x7fb95c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c10a0f0 0x7fb95c1a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c114220 con 0x7fb95c109730 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1140a0 con 0x7fb95c108540 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.248+0000 7fb960c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1143a0 con 0x7fb95c10a0f0 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 0x7fb95c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 0x7fb95c19ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59290/0 (socket says 172.21.15.146:59290) 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb95c108540 0x7fb95c19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 -- 172.21.15.146:0/1004353216 learned_addr learned my addr 172.21.15.146:0/1004353216 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95be00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c10a0f0 0x7fb95c1a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 -- 172.21.15.146:0/1004353216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c10a0f0 msgr2=0x7fb95c1a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c10a0f0 0x7fb95c1a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 -- 172.21.15.146:0/1004353216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb95c108540 msgr2=0x7fb95c19e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb95c108540 0x7fb95c19e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95aa00640 1 -- 172.21.15.146:0/1004353216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c1a63f0 con 0x7fb95c109730 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.249+0000 7fb95b400640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb95c108540 0x7fb95c19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:54.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.250+0000 7fb95aa00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 0x7fb95c19ee50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb94800e9f0 tx=0x7fb94800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.250+0000 7fb953e00640 1 -- 172.21.15.146:0/1004353216 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94800cd80 con 0x7fb95c109730 2024-04-24T08:18:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.250+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c1a66e0 con 0x7fb95c109730 2024-04-24T08:18:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.250+0000 7fb953e00640 1 -- 172.21.15.146:0/1004353216 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb948010910 con 0x7fb95c109730 2024-04-24T08:18:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.250+0000 7fb953e00640 1 -- 172.21.15.146:0/1004353216 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9480188f0 con 0x7fb95c109730 2024-04-24T08:18:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.250+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c1a6c80 con 0x7fb95c109730 2024-04-24T08:18:54.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.251+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95c10f800 con 0x7fb95c109730 2024-04-24T08:18:54.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.254+0000 7fb953e00640 1 -- 172.21.15.146:0/1004353216 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fb948010ac0 con 0x7fb95c109730 2024-04-24T08:18:54.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.255+0000 7fb953e00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9400a3c30 0x7fb9400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.255+0000 7fb953e00640 1 -- 172.21.15.146:0/1004353216 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fb9480d06a0 con 0x7fb95c109730 2024-04-24T08:18:54.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.256+0000 7fb95b400640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9400a3c30 0x7fb9400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.256+0000 7fb95b400640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9400a3c30 0x7fb9400a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb94c02a9f0 tx=0x7fb94c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:54.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.259+0000 7fb953e00640 1 -- 172.21.15.146:0/1004353216 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb94809cae0 con 0x7fb95c109730 2024-04-24T08:18:54.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.498+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb95c062440 con 0x7fb9400a3c30 2024-04-24T08:18:54.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.541+0000 7fb953e00640 1 -- 172.21.15.146:0/1004353216 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb95c062440 con 0x7fb9400a3c30 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9400a3c30 msgr2=0x7fb9400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9400a3c30 0x7fb9400a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb94c02a9f0 tx=0x7fb94c0315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 msgr2=0x7fb95c19ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 0x7fb95c19ee50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb94800e9f0 tx=0x7fb94800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 shutdown_connections 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb95c10a0f0 0x7fb95c1a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9400a3c30 0x7fb9400a60f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c109730 0x7fb95c19ee50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 --2- 172.21.15.146:0/1004353216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb95c108540 0x7fb95c19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.544+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 >> 172.21.15.146:0/1004353216 conn(0x7fb95c0774d0 msgr2=0x7fb95c10c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:54.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.545+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 shutdown_connections 2024-04-24T08:18:54.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.545+0000 7fb960c00640 1 -- 172.21.15.146:0/1004353216 wait complete. 2024-04-24T08:18:54.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 -- 172.21.15.146:0/76311560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 msgr2=0x7f4ca0074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 --2- 172.21.15.146:0/76311560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 0x7f4ca0074490 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009c10 tx=0x7f4c9002a6f0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 -- 172.21.15.146:0/76311560 shutdown_connections 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 --2- 172.21.15.146:0/76311560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca010e560 0x7f4ca0110950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 --2- 172.21.15.146:0/76311560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 0x7f4ca0074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 --2- 172.21.15.146:0/76311560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 0x7f4ca0074490 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 -- 172.21.15.146:0/76311560 >> 172.21.15.146:0/76311560 conn(0x7f4ca00fd500 msgr2=0x7f4ca00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 -- 172.21.15.146:0/76311560 shutdown_connections 2024-04-24T08:18:54.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.771+0000 7f4ca5000640 1 -- 172.21.15.146:0/76311560 wait complete. 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.772+0000 7f4ca5000640 1 Processor -- start 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.772+0000 7f4ca5000640 1 -- start start 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4ca5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 0x7f4ca019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4ca5000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 0x7f4ca019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4ca5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca010e560 0x7f4ca01a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4ca5000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0114120 con 0x7f4ca0074a60 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4ca5000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca0113fa0 con 0x7f4ca0076040 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4ca5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca01142a0 con 0x7f4ca010e560 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 0x7f4ca019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 0x7f4ca019e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59302/0 (socket says 172.21.15.146:59302) 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9f400640 1 -- 172.21.15.146:0/221145725 learned_addr learned my addr 172.21.15.146:0/221145725 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9ea00640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 0x7f4ca019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9fe00640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca010e560 0x7f4ca01a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9f400640 1 -- 172.21.15.146:0/221145725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca010e560 msgr2=0x7f4ca01a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9f400640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca010e560 0x7f4ca01a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9f400640 1 -- 172.21.15.146:0/221145725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 msgr2=0x7f4ca019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.773+0000 7f4c9f400640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 0x7f4ca019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4c9f400640 1 -- 172.21.15.146:0/221145725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca01a6480 con 0x7f4ca0074a60 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4c9ea00640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 0x7f4ca019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4c9f400640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 0x7f4ca019e860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009be0 tx=0x7f4c900305b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4c97e00640 1 -- 172.21.15.146:0/221145725 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90041020 con 0x7f4ca0074a60 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca01a6710 con 0x7f4ca0074a60 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4c97e00640 1 -- 172.21.15.146:0/221145725 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c90037040 con 0x7f4ca0074a60 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4c97e00640 1 -- 172.21.15.146:0/221145725 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c90036070 con 0x7f4ca0074a60 2024-04-24T08:18:54.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.774+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca01a6c50 con 0x7f4ca0074a60 2024-04-24T08:18:54.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.775+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca0104730 con 0x7f4ca0074a60 2024-04-24T08:18:54.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.777+0000 7f4c97e00640 1 -- 172.21.15.146:0/221145725 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4c90044020 con 0x7f4ca0074a60 2024-04-24T08:18:54.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.779+0000 7f4c97e00640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c840a3bb0 0x7f4c840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:54.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.779+0000 7f4c97e00640 1 -- 172.21.15.146:0/221145725 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f4c900f0ef0 con 0x7f4ca0074a60 2024-04-24T08:18:54.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.779+0000 7f4c9ea00640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c840a3bb0 0x7f4c840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:54.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.780+0000 7f4c9ea00640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c840a3bb0 0x7f4c840a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ca019fd70 tx=0x7f4c8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:54.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:54.783+0000 7f4c97e00640 1 -- 172.21.15.146:0/221145725 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c9004b020 con 0x7f4ca0074a60 2024-04-24T08:18:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:54 smithi177 ceph-mon[33046]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:54 smithi177 ceph-mon[33046]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:54.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:54 smithi153 ceph-mon[32746]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:54.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:54 smithi153 ceph-mon[32746]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:54.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:54 smithi146 ceph-mon[27805]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:54.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:54 smithi146 ceph-mon[27805]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:55.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.015+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f4ca00624d0 con 0x7f4c840a3bb0 2024-04-24T08:18:55.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.025+0000 7f4c97e00640 1 -- 172.21.15.146:0/221145725 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4ca00624d0 con 0x7f4c840a3bb0 2024-04-24T08:18:55.027 INFO:teuthology.orchestra.run.smithi146.stdout:/volumes/qa/sv_1/dbf0f2a2-34a9-43a0-b34c-6ccd54f4bf23 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c840a3bb0 msgr2=0x7f4c840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c840a3bb0 0x7f4c840a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ca019fd70 tx=0x7f4c8c009260 comp rx=0 tx=0).stop 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 msgr2=0x7f4ca019e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 0x7f4ca019e860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009be0 tx=0x7f4c900305b0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 shutdown_connections 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca010e560 0x7f4ca01a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f4c840a3bb0 0x7f4c840a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ca0076040 0x7f4ca019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 --2- 172.21.15.146:0/221145725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ca0074a60 0x7f4ca019e860 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 >> 172.21.15.146:0/221145725 conn(0x7f4ca00fd500 msgr2=0x7f4ca0101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:55.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 shutdown_connections 2024-04-24T08:18:55.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.028+0000 7f4ca5000640 1 -- 172.21.15.146:0/221145725 wait complete. 2024-04-24T08:18:55.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:55.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 -- 172.21.15.146:0/1681830512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140105450 msgr2=0x7fd1401058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 --2- 172.21.15.146:0/1681830512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140105450 0x7fd1401058d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd128009860 tx=0x7fd12802a340 comp rx=0 tx=0).stop 2024-04-24T08:18:55.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 -- 172.21.15.146:0/1681830512 shutdown_connections 2024-04-24T08:18:55.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 --2- 172.21.15.146:0/1681830512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd140105e10 0x7fd14010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 --2- 172.21.15.146:0/1681830512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140105450 0x7fd1401058d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 --2- 172.21.15.146:0/1681830512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd140104260 0x7fd140104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 -- 172.21.15.146:0/1681830512 >> 172.21.15.146:0/1681830512 conn(0x7fd1400ff7f0 msgr2=0x7fd140101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:55.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 -- 172.21.15.146:0/1681830512 shutdown_connections 2024-04-24T08:18:55.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.248+0000 7fd145000640 1 -- 172.21.15.146:0/1681830512 wait complete. 2024-04-24T08:18:55.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.249+0000 7fd145000640 1 Processor -- start 2024-04-24T08:18:55.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.249+0000 7fd145000640 1 -- start start 2024-04-24T08:18:55.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.249+0000 7fd145000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140104260 0x7fd14019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd145000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd140105450 0x7fd14019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd145000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd140105e10 0x7fd1401a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd145000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd14010fef0 con 0x7fd140104260 2024-04-24T08:18:55.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd145000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd14010fd70 con 0x7fd140105450 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd145000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd140110070 con 0x7fd140105e10 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140104260 0x7fd14019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140104260 0x7fd14019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59314/0 (socket says 172.21.15.146:59314) 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 -- 172.21.15.146:0/2976384532 learned_addr learned my addr 172.21.15.146:0/2976384532 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd135400640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd140105450 0x7fd14019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13e800640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd140105e10 0x7fd1401a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 -- 172.21.15.146:0/2976384532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd140105e10 msgr2=0x7fd1401a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd140105e10 0x7fd1401a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 -- 172.21.15.146:0/2976384532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd140105450 msgr2=0x7fd14019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd140105450 0x7fd14019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 -- 172.21.15.146:0/2976384532 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1401a2140 con 0x7fd140104260 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.250+0000 7fd13de00640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140104260 0x7fd14019a520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9f0 tx=0x7fd13000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.251+0000 7fd137400640 1 -- 172.21.15.146:0/2976384532 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13000ccc0 con 0x7fd140104260 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.251+0000 7fd137400640 1 -- 172.21.15.146:0/2976384532 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1300035f0 con 0x7fd140104260 2024-04-24T08:18:55.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.251+0000 7fd137400640 1 -- 172.21.15.146:0/2976384532 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13000ce60 con 0x7fd140104260 2024-04-24T08:18:55.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.251+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1401a2430 con 0x7fd140104260 2024-04-24T08:18:55.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.251+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1401a2900 con 0x7fd140104260 2024-04-24T08:18:55.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.252+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14010b600 con 0x7fd140104260 2024-04-24T08:18:55.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.254+0000 7fd137400640 1 -- 172.21.15.146:0/2976384532 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fd1300037a0 con 0x7fd140104260 2024-04-24T08:18:55.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.256+0000 7fd137400640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd1240a3c30 0x7fd1240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.256+0000 7fd137400640 1 -- 172.21.15.146:0/2976384532 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fd1300d0680 con 0x7fd140104260 2024-04-24T08:18:55.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.256+0000 7fd135400640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd1240a3c30 0x7fd1240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.257+0000 7fd135400640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd1240a3c30 0x7fd1240a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd128000c00 tx=0x7fd128030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:55.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.260+0000 7fd137400640 1 -- 172.21.15.146:0/2976384532 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd13009d110 con 0x7fd140104260 2024-04-24T08:18:55.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.514+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd14002d620 con 0x7fd140104260 2024-04-24T08:18:55.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.515+0000 7fd137400640 1 -- 172.21.15.146:0/2976384532 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11891 (secure 0 0 0) 0x7fd13009ca10 con 0x7fd140104260 2024-04-24T08:18:55.517 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:55.517 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:18:52.518710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:18:55.518 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-24T08:18:55.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd1240a3c30 msgr2=0x7fd1240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd1240a3c30 0x7fd1240a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd128000c00 tx=0x7fd128030550 comp rx=0 tx=0).stop 2024-04-24T08:18:55.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140104260 msgr2=0x7fd14019a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140104260 0x7fd14019a520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9f0 tx=0x7fd13000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 shutdown_connections 2024-04-24T08:18:55.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd140105e10 0x7fd1401a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd1240a3c30 0x7fd1240a60f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd140105450 0x7fd14019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 --2- 172.21.15.146:0/2976384532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd140104260 0x7fd14019a520 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.518+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 >> 172.21.15.146:0/2976384532 conn(0x7fd1400ff7f0 msgr2=0x7fd1401087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.519+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 shutdown_connections 2024-04-24T08:18:55.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.519+0000 7fd145000640 1 -- 172.21.15.146:0/2976384532 wait complete. 2024-04-24T08:18:55.544 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:55.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.733+0000 7f23b3600640 1 -- 172.21.15.146:0/2827893413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac10a210 msgr2=0x7f23ac110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.733+0000 7f23b3600640 1 --2- 172.21.15.146:0/2827893413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac10a210 0x7f23ac110aa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009ca0 tx=0x7f23a402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.734+0000 7f23b3600640 1 -- 172.21.15.146:0/2827893413 shutdown_connections 2024-04-24T08:18:55.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.734+0000 7f23b3600640 1 --2- 172.21.15.146:0/2827893413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23ac109850 0x7f23ac109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.734+0000 7f23b3600640 1 --2- 172.21.15.146:0/2827893413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac10a210 0x7f23ac110aa0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.734+0000 7f23b3600640 1 --2- 172.21.15.146:0/2827893413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 0x7f23ac108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.734+0000 7f23b3600640 1 -- 172.21.15.146:0/2827893413 >> 172.21.15.146:0/2827893413 conn(0x7f23ac077620 msgr2=0x7f23ac079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:55.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.734+0000 7f23b3600640 1 -- 172.21.15.146:0/2827893413 shutdown_connections 2024-04-24T08:18:55.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.734+0000 7f23b3600640 1 -- 172.21.15.146:0/2827893413 wait complete. 2024-04-24T08:18:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 Processor -- start 2024-04-24T08:18:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 -- start start 2024-04-24T08:18:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 0x7f23ac19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac109850 0x7f23ac19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23ac10a210 0x7f23ac1a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac114330 con 0x7f23ac109850 2024-04-24T08:18:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1141b0 con 0x7f23ac108660 2024-04-24T08:18:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b3600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac1144b0 con 0x7f23ac10a210 2024-04-24T08:18:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23abe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac109850 0x7f23ac19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b0c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 0x7f23ac19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b1600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23ac10a210 0x7f23ac1a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b0c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 0x7f23ac19e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54906/0 (socket says 172.21.15.146:54906) 2024-04-24T08:18:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.735+0000 7f23b0c00640 1 -- 172.21.15.146:0/3093784944 learned_addr learned my addr 172.21.15.146:0/3093784944 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23b0c00640 1 -- 172.21.15.146:0/3093784944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23ac10a210 msgr2=0x7f23ac1a5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23b0c00640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23ac10a210 0x7f23ac1a5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23b0c00640 1 -- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac109850 msgr2=0x7f23ac19ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23b0c00640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac109850 0x7f23ac19ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23b0c00640 1 -- 172.21.15.146:0/3093784944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ac1a6540 con 0x7f23ac108660 2024-04-24T08:18:55.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23b0c00640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 0x7f23ac19e920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f23a000e9c0 tx=0x7f23a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:55.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23a9600640 1 -- 172.21.15.146:0/3093784944 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a000cd80 con 0x7f23ac108660 2024-04-24T08:18:55.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23a9600640 1 -- 172.21.15.146:0/3093784944 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23a00035f0 con 0x7f23ac108660 2024-04-24T08:18:55.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ac1a6830 con 0x7f23ac108660 2024-04-24T08:18:55.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.736+0000 7f23a9600640 1 -- 172.21.15.146:0/3093784944 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a0010690 con 0x7f23ac108660 2024-04-24T08:18:55.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.737+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ac1a6d80 con 0x7f23ac108660 2024-04-24T08:18:55.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.738+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23ac10f920 con 0x7f23ac108660 2024-04-24T08:18:55.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.740+0000 7f23a9600640 1 -- 172.21.15.146:0/3093784944 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f23a00037a0 con 0x7f23ac108660 2024-04-24T08:18:55.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.741+0000 7f23a9600640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f23940a3c40 0x7f23940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:55.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.741+0000 7f23a9600640 1 -- 172.21.15.146:0/3093784944 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f23a00d0a10 con 0x7f23ac108660 2024-04-24T08:18:55.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.741+0000 7f23abe00640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f23940a3c40 0x7f23940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:55.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.742+0000 7f23abe00640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f23940a3c40 0x7f23940a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f23ac19fe30 tx=0x7f2398005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:55.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.744+0000 7f23a9600640 1 -- 172.21.15.146:0/3093784944 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23a009d340 con 0x7f23ac108660 2024-04-24T08:18:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:55 smithi177 ceph-mon[33046]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:55 smithi177 ceph-mon[33046]: pgmap v317: 97 pgs: 97 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-04-24T08:18:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:55 smithi177 ceph-mon[33046]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:55 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2976384532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:55.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.940+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23ac062c00 con 0x7f23ac108660 2024-04-24T08:18:55.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.943+0000 7f23a9600640 1 -- 172.21.15.146:0/3093784944 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22262 (secure 0 0 0) 0x7f23a009cc40 con 0x7f23ac108660 2024-04-24T08:18:55.945 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:55.945 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":88,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:48.273578+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:55.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f23940a3c40 msgr2=0x7f23940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f23940a3c40 0x7f23940a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f23ac19fe30 tx=0x7f2398005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 msgr2=0x7f23ac19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 0x7f23ac19e920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f23a000e9c0 tx=0x7f23a000ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 shutdown_connections 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23ac10a210 0x7f23ac1a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f23940a3c40 0x7f23940a6100 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ac109850 0x7f23ac19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 --2- 172.21.15.146:0/3093784944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23ac108660 0x7f23ac19e920 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 >> 172.21.15.146:0/3093784944 conn(0x7f23ac077620 msgr2=0x7f23ac10caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 shutdown_connections 2024-04-24T08:18:55.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:55.946+0000 7f23b3600640 1 -- 172.21.15.146:0/3093784944 wait complete. 2024-04-24T08:18:55.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:55 smithi153 ceph-mon[32746]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:55.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:55 smithi153 ceph-mon[32746]: pgmap v317: 97 pgs: 97 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-04-24T08:18:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:55 smithi153 ceph-mon[32746]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:55 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2976384532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:55.963 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:55 smithi146 ceph-mon[27805]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:55.963 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:55 smithi146 ceph-mon[27805]: pgmap v317: 97 pgs: 97 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.6 KiB/s wr, 13 op/s 2024-04-24T08:18:55.963 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:55 smithi146 ceph-mon[27805]: from='client.14571 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:55.963 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:55 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2976384532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:55.966 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:18:55.966 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:18:56.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.153+0000 7fdda2000640 1 -- 172.21.15.146:0/1113370447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c10a030 msgr2=0x7fdd9c1108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.153+0000 7fdda2000640 1 --2- 172.21.15.146:0/1113370447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c10a030 0x7fdd9c1108c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009ca0 tx=0x7fdd9402a8b0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.154+0000 7fdda2000640 1 -- 172.21.15.146:0/1113370447 shutdown_connections 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.154+0000 7fdda2000640 1 --2- 172.21.15.146:0/1113370447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd9c109670 0x7fdd9c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.154+0000 7fdda2000640 1 --2- 172.21.15.146:0/1113370447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c10a030 0x7fdd9c1108c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.154+0000 7fdda2000640 1 --2- 172.21.15.146:0/1113370447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c108480 0x7fdd9c1088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.154+0000 7fdda2000640 1 -- 172.21.15.146:0/1113370447 >> 172.21.15.146:0/1113370447 conn(0x7fdd9c077440 msgr2=0x7fdd9c079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.154+0000 7fdda2000640 1 -- 172.21.15.146:0/1113370447 shutdown_connections 2024-04-24T08:18:56.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.154+0000 7fdda2000640 1 -- 172.21.15.146:0/1113370447 wait complete. 2024-04-24T08:18:56.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 Processor -- start 2024-04-24T08:18:56.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 -- start start 2024-04-24T08:18:56.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c108480 0x7fdd9c19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd9c109670 0x7fdd9c19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c10a030 0x7fdd9c1a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c114110 con 0x7fdd9c108480 2024-04-24T08:18:56.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c113f90 con 0x7fdd9c10a030 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda2000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9c114290 con 0x7fdd9c109670 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c108480 0x7fdd9c19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c108480 0x7fdd9c19e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59354/0 (socket says 172.21.15.146:59354) 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.155+0000 7fdda0c00640 1 -- 172.21.15.146:0/1008537431 learned_addr learned my addr 172.21.15.146:0/1008537431 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdd93e00640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd9c109670 0x7fdd9c19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda1600640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c10a030 0x7fdd9c1a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda0c00640 1 -- 172.21.15.146:0/1008537431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd9c109670 msgr2=0x7fdd9c19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda0c00640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd9c109670 0x7fdd9c19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda0c00640 1 -- 172.21.15.146:0/1008537431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c10a030 msgr2=0x7fdd9c1a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda0c00640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c10a030 0x7fdd9c1a5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda0c00640 1 -- 172.21.15.146:0/1008537431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9c1a62e0 con 0x7fdd9c108480 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda1600640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c10a030 0x7fdd9c1a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda0c00640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c108480 0x7fdd9c19e800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e9c0 tx=0x7fdd8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdd9a000640 1 -- 172.21.15.146:0/1008537431 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00cde0 con 0x7fdd9c108480 2024-04-24T08:18:56.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdd9a000640 1 -- 172.21.15.146:0/1008537431 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd8c010910 con 0x7fdd9c108480 2024-04-24T08:18:56.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdd9a000640 1 -- 172.21.15.146:0/1008537431 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c018a00 con 0x7fdd9c108480 2024-04-24T08:18:56.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.156+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9c1a65d0 con 0x7fdd9c108480 2024-04-24T08:18:56.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.157+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd9c1a6ad0 con 0x7fdd9c108480 2024-04-24T08:18:56.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.158+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd9c10f740 con 0x7fdd9c108480 2024-04-24T08:18:56.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.160+0000 7fdd9a000640 1 -- 172.21.15.146:0/1008537431 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fdd8c01f020 con 0x7fdd9c108480 2024-04-24T08:18:56.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.161+0000 7fdd9a000640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdd800a3cc0 0x7fdd800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.161+0000 7fdd9a000640 1 -- 172.21.15.146:0/1008537431 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fdd8c0d06e0 con 0x7fdd9c108480 2024-04-24T08:18:56.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.162+0000 7fdd93e00640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdd800a3cc0 0x7fdd800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.162+0000 7fdd93e00640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdd800a3cc0 0x7fdd800a6180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c19fbd0 tx=0x7fdd84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:56.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.166+0000 7fdd9a000640 1 -- 172.21.15.146:0/1008537431 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd8c09d010 con 0x7fdd9c108480 2024-04-24T08:18:56.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.366+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd9c062440 con 0x7fdd800a3cc0 2024-04-24T08:18:56.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.394+0000 7fdd9a000640 1 -- 172.21.15.146:0/1008537431 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258673 (secure 0 0 0) 0x7fdd9c062440 con 0x7fdd800a3cc0 2024-04-24T08:18:56.400 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:56.402 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-24T08:18:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.399+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdd800a3cc0 msgr2=0x7fdd800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.399+0000 7fdda2000640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdd800a3cc0 0x7fdd800a6180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c19fbd0 tx=0x7fdd84005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.399+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c108480 msgr2=0x7fdd9c19e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.399+0000 7fdda2000640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c108480 0x7fdd9c19e800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e9c0 tx=0x7fdd8c00ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.399+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 shutdown_connections 2024-04-24T08:18:56.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.400+0000 7fdda2000640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd9c109670 0x7fdd9c19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.400+0000 7fdda2000640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdd800a3cc0 0x7fdd800a6180 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.400+0000 7fdda2000640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd9c10a030 0x7fdd9c1a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.400+0000 7fdda2000640 1 --2- 172.21.15.146:0/1008537431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd9c108480 0x7fdd9c19e800 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.400+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 >> 172.21.15.146:0/1008537431 conn(0x7fdd9c077440 msgr2=0x7fdd9c10c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:56.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.400+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 shutdown_connections 2024-04-24T08:18:56.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.400+0000 7fdda2000640 1 -- 172.21.15.146:0/1008537431 wait complete. 2024-04-24T08:18:56.420 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-04-24T08:18:54.444536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595309,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":151,"num_read_kb":130,"num_write":273,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":259,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374668,"kb_used_data":12996,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567156,"statfs":{"total":1151940427776,"available":1151556767744,"internally_reserved":0,"allocated":13307904,"data_stored":7205215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4941,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":92,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003233"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893273+0000","last_change":"2024-04-24T08:18:47.309666+0000","last_active":"2024-04-24T08:18:48.893273+0000","last_peered":"2024-04-24T08:18:48.893273+0000","last_clean":"2024-04-24T08:18:48.893273+0000","last_became_active":"2024-04-24T08:18:47.309413+0000","last_became_peered":"2024-04-24T08:18:47.309413+0000","last_unstale":"2024-04-24T08:18:48.893273+0000","last_undegraded":"2024-04-24T08:18:48.893273+0000","last_fullsized":"2024-04-24T08:18:48.893273+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:29.662505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287787+0000","last_change":"2024-04-24T08:18:47.310321+0000","last_active":"2024-04-24T08:18:48.287787+0000","last_peered":"2024-04-24T08:18:48.287787+0000","last_clean":"2024-04-24T08:18:48.287787+0000","last_became_active":"2024-04-24T08:18:47.309956+0000","last_became_peered":"2024-04-24T08:18:47.309956+0000","last_unstale":"2024-04-24T08:18:48.287787+0000","last_undegraded":"2024-04-24T08:18:48.287787+0000","last_fullsized":"2024-04-24T08:18:48.287787+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:51:48.432725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287956+0000","last_change":"2024-04-24T08:18:47.312153+0000","last_active":"2024-04-24T08:18:48.287956+0000","last_peered":"2024-04-24T08:18:48.287956+0000","last_clean":"2024-04-24T08:18:48.287956+0000","last_became_active":"2024-04-24T08:18:47.309465+0000","last_became_peered":"2024-04-24T08:18:47.309465+0000","last_unstale":"2024-04-24T08:18:48.287956+0000","last_undegraded":"2024-04-24T08:18:48.287956+0000","last_fullsized":"2024-04-24T08:18:48.287956+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:55.829143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892547+0000","last_change":"2024-04-24T08:18:47.297017+0000","last_active":"2024-04-24T08:18:48.892547+0000","last_peered":"2024-04-24T08:18:48.892547+0000","last_clean":"2024-04-24T08:18:48.892547+0000","last_became_active":"2024-04-24T08:18:47.296693+0000","last_became_peered":"2024-04-24T08:18:47.296693+0000","last_unstale":"2024-04-24T08:18:48.892547+0000","last_undegraded":"2024-04-24T08:18:48.892547+0000","last_fullsized":"2024-04-24T08:18:48.892547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:23.400403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467594+0000","last_change":"2024-04-24T08:18:47.315376+0000","last_active":"2024-04-24T08:18:48.467594+0000","last_peered":"2024-04-24T08:18:48.467594+0000","last_clean":"2024-04-24T08:18:48.467594+0000","last_became_active":"2024-04-24T08:18:47.315027+0000","last_became_peered":"2024-04-24T08:18:47.315027+0000","last_unstale":"2024-04-24T08:18:48.467594+0000","last_undegraded":"2024-04-24T08:18:48.467594+0000","last_fullsized":"2024-04-24T08:18:48.467594+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:12.504515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282490+0000","last_change":"2024-04-24T08:18:47.306388+0000","last_active":"2024-04-24T08:18:48.282490+0000","last_peered":"2024-04-24T08:18:48.282490+0000","last_clean":"2024-04-24T08:18:48.282490+0000","last_became_active":"2024-04-24T08:18:47.306082+0000","last_became_peered":"2024-04-24T08:18:47.306082+0000","last_unstale":"2024-04-24T08:18:48.282490+0000","last_undegraded":"2024-04-24T08:18:48.282490+0000","last_fullsized":"2024-04-24T08:18:48.282490+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:34:52.612799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221440+0000","last_change":"2024-04-24T08:18:47.303336+0000","last_active":"2024-04-24T08:18:49.221440+0000","last_peered":"2024-04-24T08:18:49.221440+0000","last_clean":"2024-04-24T08:18:49.221440+0000","last_became_active":"2024-04-24T08:18:47.303008+0000","last_became_peered":"2024-04-24T08:18:47.303008+0000","last_unstale":"2024-04-24T08:18:49.221440+0000","last_undegraded":"2024-04-24T08:18:49.221440+0000","last_fullsized":"2024-04-24T08:18:49.221440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:57.404693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.469361+0000","last_change":"2024-04-24T08:18:47.316672+0000","last_active":"2024-04-24T08:18:48.469361+0000","last_peered":"2024-04-24T08:18:48.469361+0000","last_clean":"2024-04-24T08:18:48.469361+0000","last_became_active":"2024-04-24T08:18:47.315741+0000","last_became_peered":"2024-04-24T08:18:47.315741+0000","last_unstale":"2024-04-24T08:18:48.469361+0000","last_undegraded":"2024-04-24T08:18:48.469361+0000","last_fullsized":"2024-04-24T08:18:48.469361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:29.156710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892767+0000","last_change":"2024-04-24T08:18:47.291733+0000","last_active":"2024-04-24T08:18:48.892767+0000","last_peered":"2024-04-24T08:18:48.892767+0000","last_clean":"2024-04-24T08:18:48.892767+0000","last_became_active":"2024-04-24T08:18:47.291432+0000","last_became_peered":"2024-04-24T08:18:47.291432+0000","last_unstale":"2024-04-24T08:18:48.892767+0000","last_undegraded":"2024-04-24T08:18:48.892767+0000","last_fullsized":"2024-04-24T08:18:48.892767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:38:03.601540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282862+0000","last_change":"2024-04-24T08:18:47.294544+0000","last_active":"2024-04-24T08:18:48.282862+0000","last_peered":"2024-04-24T08:18:48.282862+0000","last_clean":"2024-04-24T08:18:48.282862+0000","last_became_active":"2024-04-24T08:18:47.294183+0000","last_became_peered":"2024-04-24T08:18:47.294183+0000","last_unstale":"2024-04-24T08:18:48.282862+0000","last_undegraded":"2024-04-24T08:18:48.282862+0000","last_fullsized":"2024-04-24T08:18:48.282862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:53.082644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221517+0000","last_change":"2024-04-24T08:18:47.292615+0000","last_active":"2024-04-24T08:18:49.221517+0000","last_peered":"2024-04-24T08:18:49.221517+0000","last_clean":"2024-04-24T08:18:49.221517+0000","last_became_active":"2024-04-24T08:18:47.292300+0000","last_became_peered":"2024-04-24T08:18:47.292300+0000","last_unstale":"2024-04-24T08:18:49.221517+0000","last_undegraded":"2024-04-24T08:18:49.221517+0000","last_fullsized":"2024-04-24T08:18:49.221517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:58.293253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892467+0000","last_change":"2024-04-24T08:18:47.298388+0000","last_active":"2024-04-24T08:18:48.892467+0000","last_peered":"2024-04-24T08:18:48.892467+0000","last_clean":"2024-04-24T08:18:48.892467+0000","last_became_active":"2024-04-24T08:18:47.298008+0000","last_became_peered":"2024-04-24T08:18:47.298008+0000","last_unstale":"2024-04-24T08:18:48.892467+0000","last_undegraded":"2024-04-24T08:18:48.892467+0000","last_fullsized":"2024-04-24T08:18:48.892467+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:10:00.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284912+0000","last_change":"2024-04-24T08:18:47.298809+0000","last_active":"2024-04-24T08:18:48.284912+0000","last_peered":"2024-04-24T08:18:48.284912+0000","last_clean":"2024-04-24T08:18:48.284912+0000","last_became_active":"2024-04-24T08:18:47.298103+0000","last_became_peered":"2024-04-24T08:18:47.298103+0000","last_unstale":"2024-04-24T08:18:48.284912+0000","last_undegraded":"2024-04-24T08:18:48.284912+0000","last_fullsized":"2024-04-24T08:18:48.284912+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:38:53.625721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287023+0000","last_change":"2024-04-24T08:18:47.314572+0000","last_active":"2024-04-24T08:18:48.287023+0000","last_peered":"2024-04-24T08:18:48.287023+0000","last_clean":"2024-04-24T08:18:48.287023+0000","last_became_active":"2024-04-24T08:18:47.313372+0000","last_became_peered":"2024-04-24T08:18:47.313372+0000","last_unstale":"2024-04-24T08:18:48.287023+0000","last_undegraded":"2024-04-24T08:18:48.287023+0000","last_fullsized":"2024-04-24T08:18:48.287023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:56:01.954909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.286869+0000","last_change":"2024-04-24T08:18:47.314822+0000","last_active":"2024-04-24T08:18:48.286869+0000","last_peered":"2024-04-24T08:18:48.286869+0000","last_clean":"2024-04-24T08:18:48.286869+0000","last_became_active":"2024-04-24T08:18:47.310774+0000","last_became_peered":"2024-04-24T08:18:47.310774+0000","last_unstale":"2024-04-24T08:18:48.286869+0000","last_undegraded":"2024-04-24T08:18:48.286869+0000","last_fullsized":"2024-04-24T08:18:48.286869+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:24.934100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.891934+0000","last_change":"2024-04-24T08:18:47.307208+0000","last_active":"2024-04-24T08:18:48.891934+0000","last_peered":"2024-04-24T08:18:48.891934+0000","last_clean":"2024-04-24T08:18:48.891934+0000","last_became_active":"2024-04-24T08:18:47.306889+0000","last_became_peered":"2024-04-24T08:18:47.306889+0000","last_unstale":"2024-04-24T08:18:48.891934+0000","last_undegraded":"2024-04-24T08:18:48.891934+0000","last_fullsized":"2024-04-24T08:18:48.891934+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:20:01.438388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893047+0000","last_change":"2024-04-24T08:18:47.322929+0000","last_active":"2024-04-24T08:18:48.893047+0000","last_peered":"2024-04-24T08:18:48.893047+0000","last_clean":"2024-04-24T08:18:48.893047+0000","last_became_active":"2024-04-24T08:18:47.322593+0000","last_became_peered":"2024-04-24T08:18:47.322593+0000","last_unstale":"2024-04-24T08:18:48.893047+0000","last_undegraded":"2024-04-24T08:18:48.893047+0000","last_fullsized":"2024-04-24T08:18:48.893047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:09.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.465031+0000","last_change":"2024-04-24T08:18:47.304886+0000","last_active":"2024-04-24T08:18:48.465031+0000","last_peered":"2024-04-24T08:18:48.465031+0000","last_clean":"2024-04-24T08:18:48.465031+0000","last_became_active":"2024-04-24T08:18:47.302631+0000","last_became_peered":"2024-04-24T08:18:47.302631+0000","last_unstale":"2024-04-24T08:18:48.465031+0000","last_undegraded":"2024-04-24T08:18:48.465031+0000","last_fullsized":"2024-04-24T08:18:48.465031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:13.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893769+0000","last_change":"2024-04-24T08:18:47.316828+0000","last_active":"2024-04-24T08:18:48.893769+0000","last_peered":"2024-04-24T08:18:48.893769+0000","last_clean":"2024-04-24T08:18:48.893769+0000","last_became_active":"2024-04-24T08:18:47.316540+0000","last_became_peered":"2024-04-24T08:18:47.316540+0000","last_unstale":"2024-04-24T08:18:48.893769+0000","last_undegraded":"2024-04-24T08:18:48.893769+0000","last_fullsized":"2024-04-24T08:18:48.893769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:41:42.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.464420+0000","last_change":"2024-04-24T08:18:46.444744+0000","last_active":"2024-04-24T08:18:48.464420+0000","last_peered":"2024-04-24T08:18:48.464420+0000","last_clean":"2024-04-24T08:18:48.464420+0000","last_became_active":"2024-04-24T08:18:46.442495+0000","last_became_peered":"2024-04-24T08:18:46.442495+0000","last_unstale":"2024-04-24T08:18:48.464420+0000","last_undegraded":"2024-04-24T08:18:48.464420+0000","last_fullsized":"2024-04-24T08:18:48.464420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:33:32.270984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.471046+0000","last_change":"2024-04-24T08:18:47.320462+0000","last_active":"2024-04-24T08:18:48.471046+0000","last_peered":"2024-04-24T08:18:48.471046+0000","last_clean":"2024-04-24T08:18:48.471046+0000","last_became_active":"2024-04-24T08:18:47.319267+0000","last_became_peered":"2024-04-24T08:18:47.319267+0000","last_unstale":"2024-04-24T08:18:48.471046+0000","last_undegraded":"2024-04-24T08:18:48.471046+0000","last_fullsized":"2024-04-24T08:18:48.471046+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:50:15.882077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.470748+0000","last_change":"2024-04-24T08:18:46.453278+0000","last_active":"2024-04-24T08:18:48.470748+0000","last_peered":"2024-04-24T08:18:48.470748+0000","last_clean":"2024-04-24T08:18:48.470748+0000","last_became_active":"2024-04-24T08:18:46.452913+0000","last_became_peered":"2024-04-24T08:18:46.452913+0000","last_unstale":"2024-04-24T08:18:48.470748+0000","last_undegraded":"2024-04-24T08:18:48.470748+0000","last_fullsized":"2024-04-24T08:18:48.470748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:48.272190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.288417+0000","last_change":"2024-04-24T08:18:47.323610+0000","last_active":"2024-04-24T08:18:48.288417+0000","last_peered":"2024-04-24T08:18:48.288417+0000","last_clean":"2024-04-24T08:18:48.288417+0000","last_became_active":"2024-04-24T08:18:47.322795+0000","last_became_peered":"2024-04-24T08:18:47.322795+0000","last_unstale":"2024-04-24T08:18:48.288417+0000","last_undegraded":"2024-04-24T08:18:48.288417+0000","last_fullsized":"2024-04-24T08:18:48.288417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:32.951564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.522260+0000","last_change":"2024-04-24T08:18:46.438815+0000","last_active":"2024-04-24T08:18:51.522260+0000","last_peered":"2024-04-24T08:18:51.522260+0000","last_clean":"2024-04-24T08:18:51.522260+0000","last_became_active":"2024-04-24T08:18:46.438523+0000","last_became_peered":"2024-04-24T08:18:46.438523+0000","last_unstale":"2024-04-24T08:18:51.522260+0000","last_undegraded":"2024-04-24T08:18:51.522260+0000","last_fullsized":"2024-04-24T08:18:51.522260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:50:57.222540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892006+0000","last_change":"2024-04-24T08:18:47.304884+0000","last_active":"2024-04-24T08:18:48.892006+0000","last_peered":"2024-04-24T08:18:48.892006+0000","last_clean":"2024-04-24T08:18:48.892006+0000","last_became_active":"2024-04-24T08:18:47.304520+0000","last_became_peered":"2024-04-24T08:18:47.304520+0000","last_unstale":"2024-04-24T08:18:48.892006+0000","last_undegraded":"2024-04-24T08:18:48.892006+0000","last_fullsized":"2024-04-24T08:18:48.892006+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:42.325010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467940+0000","last_change":"2024-04-24T08:18:46.441073+0000","last_active":"2024-04-24T08:18:48.467940+0000","last_peered":"2024-04-24T08:18:48.467940+0000","last_clean":"2024-04-24T08:18:48.467940+0000","last_became_active":"2024-04-24T08:18:46.440721+0000","last_became_peered":"2024-04-24T08:18:46.440721+0000","last_unstale":"2024-04-24T08:18:48.467940+0000","last_undegraded":"2024-04-24T08:18:48.467940+0000","last_fullsized":"2024-04-24T08:18:48.467940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:51:39.516458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893555+0000","last_change":"2024-04-24T08:18:47.293947+0000","last_active":"2024-04-24T08:18:48.893555+0000","last_peered":"2024-04-24T08:18:48.893555+0000","last_clean":"2024-04-24T08:18:48.893555+0000","last_became_active":"2024-04-24T08:18:47.293623+0000","last_became_peered":"2024-04-24T08:18:47.293623+0000","last_unstale":"2024-04-24T08:18:48.893555+0000","last_undegraded":"2024-04-24T08:18:48.893555+0000","last_fullsized":"2024-04-24T08:18:48.893555+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:23.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:53.529149+0000","last_change":"2024-04-24T08:18:46.439909+0000","last_active":"2024-04-24T08:18:53.529149+0000","last_peered":"2024-04-24T08:18:53.529149+0000","last_clean":"2024-04-24T08:18:53.529149+0000","last_became_active":"2024-04-24T08:18:46.439597+0000","last_became_peered":"2024-04-24T08:18:46.439597+0000","last_unstale":"2024-04-24T08:18:53.529149+0000","last_undegraded":"2024-04-24T08:18:53.529149+0000","last_fullsized":"2024-04-24T08:18:53.529149+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:49:11.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467643+0000","last_change":"2024-04-24T08:18:47.306480+0000","last_active":"2024-04-24T08:18:48.467643+0000","last_peered":"2024-04-24T08:18:48.467643+0000","last_clean":"2024-04-24T08:18:48.467643+0000","last_became_active":"2024-04-24T08:18:47.306104+0000","last_became_peered":"2024-04-24T08:18:47.306104+0000","last_unstale":"2024-04-24T08:18:48.467643+0000","last_undegraded":"2024-04-24T08:18:48.467643+0000","last_fullsized":"2024-04-24T08:18:48.467643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:56.340836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282363+0000","last_change":"2024-04-24T08:18:46.439381+0000","last_active":"2024-04-24T08:18:48.282363+0000","last_peered":"2024-04-24T08:18:48.282363+0000","last_clean":"2024-04-24T08:18:48.282363+0000","last_became_active":"2024-04-24T08:18:46.439033+0000","last_became_peered":"2024-04-24T08:18:46.439033+0000","last_unstale":"2024-04-24T08:18:48.282363+0000","last_undegraded":"2024-04-24T08:18:48.282363+0000","last_fullsized":"2024-04-24T08:18:48.282363+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:48:28.833066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893176+0000","last_change":"2024-04-24T08:18:47.315402+0000","last_active":"2024-04-24T08:18:48.893176+0000","last_peered":"2024-04-24T08:18:48.893176+0000","last_clean":"2024-04-24T08:18:48.893176+0000","last_became_active":"2024-04-24T08:18:47.315050+0000","last_became_peered":"2024-04-24T08:18:47.315050+0000","last_unstale":"2024-04-24T08:18:48.893176+0000","last_undegraded":"2024-04-24T08:18:48.893176+0000","last_fullsized":"2024-04-24T08:18:48.893176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:59:49.948239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284237+0000","last_change":"2024-04-24T08:18:46.448151+0000","last_active":"2024-04-24T08:18:48.284237+0000","last_peered":"2024-04-24T08:18:48.284237+0000","last_clean":"2024-04-24T08:18:48.284237+0000","last_became_active":"2024-04-24T08:18:46.447077+0000","last_became_peered":"2024-04-24T08:18:46.447077+0000","last_unstale":"2024-04-24T08:18:48.284237+0000","last_undegraded":"2024-04-24T08:18:48.284237+0000","last_fullsized":"2024-04-24T08:18:48.284237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:00.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.469089+0000","last_change":"2024-04-24T08:18:47.319423+0000","last_active":"2024-04-24T08:18:48.469089+0000","last_peered":"2024-04-24T08:18:48.469089+0000","last_clean":"2024-04-24T08:18:48.469089+0000","last_became_active":"2024-04-24T08:18:47.316284+0000","last_became_peered":"2024-04-24T08:18:47.316284+0000","last_unstale":"2024-04-24T08:18:48.469089+0000","last_undegraded":"2024-04-24T08:18:48.469089+0000","last_fullsized":"2024-04-24T08:18:48.469089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:55.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.018584+0000","last_change":"2024-04-24T08:18:46.436152+0000","last_active":"2024-04-24T08:18:49.018584+0000","last_peered":"2024-04-24T08:18:49.018584+0000","last_clean":"2024-04-24T08:18:49.018584+0000","last_became_active":"2024-04-24T08:18:46.435835+0000","last_became_peered":"2024-04-24T08:18:46.435835+0000","last_unstale":"2024-04-24T08:18:49.018584+0000","last_undegraded":"2024-04-24T08:18:49.018584+0000","last_fullsized":"2024-04-24T08:18:49.018584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:12:30.536455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.463156+0000","last_change":"2024-04-24T08:18:47.326408+0000","last_active":"2024-04-24T08:18:48.463156+0000","last_peered":"2024-04-24T08:18:48.463156+0000","last_clean":"2024-04-24T08:18:48.463156+0000","last_became_active":"2024-04-24T08:18:47.325991+0000","last_became_peered":"2024-04-24T08:18:47.325991+0000","last_unstale":"2024-04-24T08:18:48.463156+0000","last_undegraded":"2024-04-24T08:18:48.463156+0000","last_fullsized":"2024-04-24T08:18:48.463156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:59.727066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.507536+0000","last_change":"2024-04-24T08:18:46.446652+0000","last_active":"2024-04-24T08:18:50.507536+0000","last_peered":"2024-04-24T08:18:50.507536+0000","last_clean":"2024-04-24T08:18:50.507536+0000","last_became_active":"2024-04-24T08:18:46.446348+0000","last_became_peered":"2024-04-24T08:18:46.446348+0000","last_unstale":"2024-04-24T08:18:50.507536+0000","last_undegraded":"2024-04-24T08:18:50.507536+0000","last_fullsized":"2024-04-24T08:18:50.507536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:25.791359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892560+0000","last_change":"2024-04-24T08:18:47.309072+0000","last_active":"2024-04-24T08:18:48.892560+0000","last_peered":"2024-04-24T08:18:48.892560+0000","last_clean":"2024-04-24T08:18:48.892560+0000","last_became_active":"2024-04-24T08:18:47.308729+0000","last_became_peered":"2024-04-24T08:18:47.308729+0000","last_unstale":"2024-04-24T08:18:48.892560+0000","last_undegraded":"2024-04-24T08:18:48.892560+0000","last_fullsized":"2024-04-24T08:18:48.892560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:58.256876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.505650+0000","last_change":"2024-04-24T08:18:46.439190+0000","last_active":"2024-04-24T08:18:50.505650+0000","last_peered":"2024-04-24T08:18:50.505650+0000","last_clean":"2024-04-24T08:18:50.505650+0000","last_became_active":"2024-04-24T08:18:46.438442+0000","last_became_peered":"2024-04-24T08:18:46.438442+0000","last_unstale":"2024-04-24T08:18:50.505650+0000","last_undegraded":"2024-04-24T08:18:50.505650+0000","last_fullsized":"2024-04-24T08:18:50.505650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:42.614563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.288228+0000","last_change":"2024-04-24T08:18:47.311447+0000","last_active":"2024-04-24T08:18:48.288228+0000","last_peered":"2024-04-24T08:18:48.288228+0000","last_clean":"2024-04-24T08:18:48.288228+0000","last_became_active":"2024-04-24T08:18:47.310939+0000","last_became_peered":"2024-04-24T08:18:47.310939+0000","last_unstale":"2024-04-24T08:18:48.288228+0000","last_undegraded":"2024-04-24T08:18:48.288228+0000","last_fullsized":"2024-04-24T08:18:48.288228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:42.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.508794+0000","last_change":"2024-04-24T08:18:46.450596+0000","last_active":"2024-04-24T08:18:50.508794+0000","last_peered":"2024-04-24T08:18:50.508794+0000","last_clean":"2024-04-24T08:18:50.508794+0000","last_became_active":"2024-04-24T08:18:46.450226+0000","last_became_peered":"2024-04-24T08:18:46.450226+0000","last_unstale":"2024-04-24T08:18:50.508794+0000","last_undegraded":"2024-04-24T08:18:50.508794+0000","last_fullsized":"2024-04-24T08:18:50.508794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:05.722507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892288+0000","last_change":"2024-04-24T08:18:47.295266+0000","last_active":"2024-04-24T08:18:48.892288+0000","last_peered":"2024-04-24T08:18:48.892288+0000","last_clean":"2024-04-24T08:18:48.892288+0000","last_became_active":"2024-04-24T08:18:47.294963+0000","last_became_peered":"2024-04-24T08:18:47.294963+0000","last_unstale":"2024-04-24T08:18:48.892288+0000","last_undegraded":"2024-04-24T08:18:48.892288+0000","last_fullsized":"2024-04-24T08:18:48.892288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:07.638921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'8","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.504612+0000","last_change":"2024-04-24T08:18:46.435524+0000","last_active":"2024-04-24T08:18:50.504612+0000","last_peered":"2024-04-24T08:18:50.504612+0000","last_clean":"2024-04-24T08:18:50.504612+0000","last_became_active":"2024-04-24T08:18:46.435212+0000","last_became_peered":"2024-04-24T08:18:46.435212+0000","last_unstale":"2024-04-24T08:18:50.504612+0000","last_undegraded":"2024-04-24T08:18:50.504612+0000","last_fullsized":"2024-04-24T08:18:50.504612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:45.575905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467911+0000","last_change":"2024-04-24T08:18:47.293648+0000","last_active":"2024-04-24T08:18:48.467911+0000","last_peered":"2024-04-24T08:18:48.467911+0000","last_clean":"2024-04-24T08:18:48.467911+0000","last_became_active":"2024-04-24T08:18:47.293260+0000","last_became_peered":"2024-04-24T08:18:47.293260+0000","last_unstale":"2024-04-24T08:18:48.467911+0000","last_undegraded":"2024-04-24T08:18:48.467911+0000","last_fullsized":"2024-04-24T08:18:48.467911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:36.462606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892977+0000","last_change":"2024-04-24T08:18:46.441401+0000","last_active":"2024-04-24T08:18:48.892977+0000","last_peered":"2024-04-24T08:18:48.892977+0000","last_clean":"2024-04-24T08:18:48.892977+0000","last_became_active":"2024-04-24T08:18:46.440985+0000","last_became_peered":"2024-04-24T08:18:46.440985+0000","last_unstale":"2024-04-24T08:18:48.892977+0000","last_undegraded":"2024-04-24T08:18:48.892977+0000","last_fullsized":"2024-04-24T08:18:48.892977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:45.201518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284200+0000","last_change":"2024-04-24T08:18:47.320928+0000","last_active":"2024-04-24T08:18:48.284200+0000","last_peered":"2024-04-24T08:18:48.284200+0000","last_clean":"2024-04-24T08:18:48.284200+0000","last_became_active":"2024-04-24T08:18:47.320515+0000","last_became_peered":"2024-04-24T08:18:47.320515+0000","last_unstale":"2024-04-24T08:18:48.284200+0000","last_undegraded":"2024-04-24T08:18:48.284200+0000","last_fullsized":"2024-04-24T08:18:48.284200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:55:42.899438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.508954+0000","last_change":"2024-04-24T08:18:46.437677+0000","last_active":"2024-04-24T08:18:50.508954+0000","last_peered":"2024-04-24T08:18:50.508954+0000","last_clean":"2024-04-24T08:18:50.508954+0000","last_became_active":"2024-04-24T08:18:46.437364+0000","last_became_peered":"2024-04-24T08:18:46.437364+0000","last_unstale":"2024-04-24T08:18:50.508954+0000","last_undegraded":"2024-04-24T08:18:50.508954+0000","last_fullsized":"2024-04-24T08:18:50.508954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:25:55.408887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221573+0000","last_change":"2024-04-24T08:18:47.298396+0000","last_active":"2024-04-24T08:18:49.221573+0000","last_peered":"2024-04-24T08:18:49.221573+0000","last_clean":"2024-04-24T08:18:49.221573+0000","last_became_active":"2024-04-24T08:18:47.298021+0000","last_became_peered":"2024-04-24T08:18:47.298021+0000","last_unstale":"2024-04-24T08:18:49.221573+0000","last_undegraded":"2024-04-24T08:18:49.221573+0000","last_fullsized":"2024-04-24T08:18:49.221573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:24.015897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:52.533062+0000","last_change":"2024-04-24T08:18:46.436612+0000","last_active":"2024-04-24T08:18:52.533062+0000","last_peered":"2024-04-24T08:18:52.533062+0000","last_clean":"2024-04-24T08:18:52.533062+0000","last_became_active":"2024-04-24T08:18:46.436211+0000","last_became_peered":"2024-04-24T08:18:46.436211+0000","last_unstale":"2024-04-24T08:18:52.533062+0000","last_undegraded":"2024-04-24T08:18:52.533062+0000","last_fullsized":"2024-04-24T08:18:52.533062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:49:05.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221294+0000","last_change":"2024-04-24T08:18:47.290849+0000","last_active":"2024-04-24T08:18:49.221294+0000","last_peered":"2024-04-24T08:18:49.221294+0000","last_clean":"2024-04-24T08:18:49.221294+0000","last_became_active":"2024-04-24T08:18:47.290514+0000","last_became_peered":"2024-04-24T08:18:47.290514+0000","last_unstale":"2024-04-24T08:18:49.221294+0000","last_undegraded":"2024-04-24T08:18:49.221294+0000","last_fullsized":"2024-04-24T08:18:49.221294+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:39:12.527477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.524099+0000","last_change":"2024-04-24T08:18:46.451815+0000","last_active":"2024-04-24T08:18:51.524099+0000","last_peered":"2024-04-24T08:18:51.524099+0000","last_clean":"2024-04-24T08:18:51.524099+0000","last_became_active":"2024-04-24T08:18:46.451478+0000","last_became_peered":"2024-04-24T08:18:46.451478+0000","last_unstale":"2024-04-24T08:18:51.524099+0000","last_undegraded":"2024-04-24T08:18:51.524099+0000","last_fullsized":"2024-04-24T08:18:51.524099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:54:22.907143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892342+0000","last_change":"2024-04-24T08:18:47.295181+0000","last_active":"2024-04-24T08:18:48.892342+0000","last_peered":"2024-04-24T08:18:48.892342+0000","last_clean":"2024-04-24T08:18:48.892342+0000","last_became_active":"2024-04-24T08:18:47.294842+0000","last_became_peered":"2024-04-24T08:18:47.294842+0000","last_unstale":"2024-04-24T08:18:48.892342+0000","last_undegraded":"2024-04-24T08:18:48.892342+0000","last_fullsized":"2024-04-24T08:18:48.892342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:37.058025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.468046+0000","last_change":"2024-04-24T08:18:46.437967+0000","last_active":"2024-04-24T08:18:48.468046+0000","last_peered":"2024-04-24T08:18:48.468046+0000","last_clean":"2024-04-24T08:18:48.468046+0000","last_became_active":"2024-04-24T08:18:46.437617+0000","last_became_peered":"2024-04-24T08:18:46.437617+0000","last_unstale":"2024-04-24T08:18:48.468046+0000","last_undegraded":"2024-04-24T08:18:48.468046+0000","last_fullsized":"2024-04-24T08:18:48.468046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:07.303827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892483+0000","last_change":"2024-04-24T08:18:47.296975+0000","last_active":"2024-04-24T08:18:48.892483+0000","last_peered":"2024-04-24T08:18:48.892483+0000","last_clean":"2024-04-24T08:18:48.892483+0000","last_became_active":"2024-04-24T08:18:47.296691+0000","last_became_peered":"2024-04-24T08:18:47.296691+0000","last_unstale":"2024-04-24T08:18:48.892483+0000","last_undegraded":"2024-04-24T08:18:48.892483+0000","last_fullsized":"2024-04-24T08:18:48.892483+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:20:04.537185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.463919+0000","last_change":"2024-04-24T08:18:46.453216+0000","last_active":"2024-04-24T08:18:48.463919+0000","last_peered":"2024-04-24T08:18:48.463919+0000","last_clean":"2024-04-24T08:18:48.463919+0000","last_became_active":"2024-04-24T08:18:46.452814+0000","last_became_peered":"2024-04-24T08:18:46.452814+0000","last_unstale":"2024-04-24T08:18:48.463919+0000","last_undegraded":"2024-04-24T08:18:48.463919+0000","last_fullsized":"2024-04-24T08:18:48.463919+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:18.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282694+0000","last_change":"2024-04-24T08:18:47.316791+0000","last_active":"2024-04-24T08:18:48.282694+0000","last_peered":"2024-04-24T08:18:48.282694+0000","last_clean":"2024-04-24T08:18:48.282694+0000","last_became_active":"2024-04-24T08:18:47.316486+0000","last_became_peered":"2024-04-24T08:18:47.316486+0000","last_unstale":"2024-04-24T08:18:48.282694+0000","last_undegraded":"2024-04-24T08:18:48.282694+0000","last_fullsized":"2024-04-24T08:18:48.282694+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:36:37.190732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221379+0000","last_change":"2024-04-24T08:18:46.432831+0000","last_active":"2024-04-24T08:18:49.221379+0000","last_peered":"2024-04-24T08:18:49.221379+0000","last_clean":"2024-04-24T08:18:49.221379+0000","last_became_active":"2024-04-24T08:18:46.432483+0000","last_became_peered":"2024-04-24T08:18:46.432483+0000","last_unstale":"2024-04-24T08:18:49.221379+0000","last_undegraded":"2024-04-24T08:18:49.221379+0000","last_fullsized":"2024-04-24T08:18:49.221379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:43:24.802099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282548+0000","last_change":"2024-04-24T08:18:47.309408+0000","last_active":"2024-04-24T08:18:48.282548+0000","last_peered":"2024-04-24T08:18:48.282548+0000","last_clean":"2024-04-24T08:18:48.282548+0000","last_became_active":"2024-04-24T08:18:47.309083+0000","last_became_peered":"2024-04-24T08:18:47.309083+0000","last_unstale":"2024-04-24T08:18:48.282548+0000","last_undegraded":"2024-04-24T08:18:48.282548+0000","last_fullsized":"2024-04-24T08:18:48.282548+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:15:44.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892623+0000","last_change":"2024-04-24T08:18:47.291060+0000","last_active":"2024-04-24T08:18:48.892623+0000","last_peered":"2024-04-24T08:18:48.892623+0000","last_clean":"2024-04-24T08:18:48.892623+0000","last_became_active":"2024-04-24T08:18:47.290635+0000","last_became_peered":"2024-04-24T08:18:47.290635+0000","last_unstale":"2024-04-24T08:18:48.892623+0000","last_undegraded":"2024-04-24T08:18:48.892623+0000","last_fullsized":"2024-04-24T08:18:48.892623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:20.372853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.283955+0000","last_change":"2024-04-24T08:18:46.449850+0000","last_active":"2024-04-24T08:18:48.283955+0000","last_peered":"2024-04-24T08:18:48.283955+0000","last_clean":"2024-04-24T08:18:48.283955+0000","last_became_active":"2024-04-24T08:18:46.448486+0000","last_became_peered":"2024-04-24T08:18:46.448486+0000","last_unstale":"2024-04-24T08:18:48.283955+0000","last_undegraded":"2024-04-24T08:18:48.283955+0000","last_fullsized":"2024-04-24T08:18:48.283955+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:54.611867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.470613+0000","last_change":"2024-04-24T08:18:47.328051+0000","last_active":"2024-04-24T08:18:48.470613+0000","last_peered":"2024-04-24T08:18:48.470613+0000","last_clean":"2024-04-24T08:18:48.470613+0000","last_became_active":"2024-04-24T08:18:47.327474+0000","last_became_peered":"2024-04-24T08:18:47.327474+0000","last_unstale":"2024-04-24T08:18:48.470613+0000","last_undegraded":"2024-04-24T08:18:48.470613+0000","last_fullsized":"2024-04-24T08:18:48.470613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:07.577741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:52.524284+0000","last_change":"2024-04-24T08:18:46.444375+0000","last_active":"2024-04-24T08:18:52.524284+0000","last_peered":"2024-04-24T08:18:52.524284+0000","last_clean":"2024-04-24T08:18:52.524284+0000","last_became_active":"2024-04-24T08:18:46.443963+0000","last_became_peered":"2024-04-24T08:18:46.443963+0000","last_unstale":"2024-04-24T08:18:52.524284+0000","last_undegraded":"2024-04-24T08:18:52.524284+0000","last_fullsized":"2024-04-24T08:18:52.524284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:22.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221019+0000","last_change":"2024-04-24T08:17:29.632219+0000","last_active":"2024-04-24T08:18:49.221019+0000","last_peered":"2024-04-24T08:18:49.221019+0000","last_clean":"2024-04-24T08:18:49.221019+0000","last_became_active":"2024-04-24T08:17:29.631841+0000","last_became_peered":"2024-04-24T08:17:29.631841+0000","last_unstale":"2024-04-24T08:18:49.221019+0000","last_undegraded":"2024-04-24T08:18:49.221019+0000","last_fullsized":"2024-04-24T08:18:49.221019+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_clean_scrub_stamp":"2024-04-24T08:13:41.228801+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:02:55.415959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.283803+0000","last_change":"2024-04-24T08:18:47.301876+0000","last_active":"2024-04-24T08:18:48.283803+0000","last_peered":"2024-04-24T08:18:48.283803+0000","last_clean":"2024-04-24T08:18:48.283803+0000","last_became_active":"2024-04-24T08:18:47.301545+0000","last_became_peered":"2024-04-24T08:18:47.301545+0000","last_unstale":"2024-04-24T08:18:48.283803+0000","last_undegraded":"2024-04-24T08:18:48.283803+0000","last_fullsized":"2024-04-24T08:18:48.283803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:34.650014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.220998+0000","last_change":"2024-04-24T08:18:46.437915+0000","last_active":"2024-04-24T08:18:49.220998+0000","last_peered":"2024-04-24T08:18:49.220998+0000","last_clean":"2024-04-24T08:18:49.220998+0000","last_became_active":"2024-04-24T08:18:46.437567+0000","last_became_peered":"2024-04-24T08:18:46.437567+0000","last_unstale":"2024-04-24T08:18:49.220998+0000","last_undegraded":"2024-04-24T08:18:49.220998+0000","last_fullsized":"2024-04-24T08:18:49.220998+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:36.801535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.468006+0000","last_change":"2024-04-24T08:18:46.451834+0000","last_active":"2024-04-24T08:18:48.468006+0000","last_peered":"2024-04-24T08:18:48.468006+0000","last_clean":"2024-04-24T08:18:48.468006+0000","last_became_active":"2024-04-24T08:18:46.451505+0000","last_became_peered":"2024-04-24T08:18:46.451505+0000","last_unstale":"2024-04-24T08:18:48.468006+0000","last_undegraded":"2024-04-24T08:18:48.468006+0000","last_fullsized":"2024-04-24T08:18:48.468006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:26.468176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.285077+0000","last_change":"2024-04-24T08:18:47.310490+0000","last_active":"2024-04-24T08:18:48.285077+0000","last_peered":"2024-04-24T08:18:48.285077+0000","last_clean":"2024-04-24T08:18:48.285077+0000","last_became_active":"2024-04-24T08:18:47.309615+0000","last_became_peered":"2024-04-24T08:18:47.309615+0000","last_unstale":"2024-04-24T08:18:48.285077+0000","last_undegraded":"2024-04-24T08:18:48.285077+0000","last_fullsized":"2024-04-24T08:18:48.285077+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:22:55.063485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.464798+0000","last_change":"2024-04-24T08:18:46.445274+0000","last_active":"2024-04-24T08:18:48.464798+0000","last_peered":"2024-04-24T08:18:48.464798+0000","last_clean":"2024-04-24T08:18:48.464798+0000","last_became_active":"2024-04-24T08:18:46.444949+0000","last_became_peered":"2024-04-24T08:18:46.444949+0000","last_unstale":"2024-04-24T08:18:48.464798+0000","last_undegraded":"2024-04-24T08:18:48.464798+0000","last_fullsized":"2024-04-24T08:18:48.464798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:59:59.062657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.471098+0000","last_change":"2024-04-24T08:18:47.319022+0000","last_active":"2024-04-24T08:18:48.471098+0000","last_peered":"2024-04-24T08:18:48.471098+0000","last_clean":"2024-04-24T08:18:48.471098+0000","last_became_active":"2024-04-24T08:18:47.318167+0000","last_became_peered":"2024-04-24T08:18:47.318167+0000","last_unstale":"2024-04-24T08:18:48.471098+0000","last_undegraded":"2024-04-24T08:18:48.471098+0000","last_fullsized":"2024-04-24T08:18:48.471098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:18.980856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.507488+0000","last_change":"2024-04-24T08:18:46.433018+0000","last_active":"2024-04-24T08:18:50.507488+0000","last_peered":"2024-04-24T08:18:50.507488+0000","last_clean":"2024-04-24T08:18:50.507488+0000","last_became_active":"2024-04-24T08:18:46.432681+0000","last_became_peered":"2024-04-24T08:18:46.432681+0000","last_unstale":"2024-04-24T08:18:50.507488+0000","last_undegraded":"2024-04-24T08:18:50.507488+0000","last_fullsized":"2024-04-24T08:18:50.507488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:53.691684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.288121+0000","last_change":"2024-04-24T08:18:47.326055+0000","last_active":"2024-04-24T08:18:48.288121+0000","last_peered":"2024-04-24T08:18:48.288121+0000","last_clean":"2024-04-24T08:18:48.288121+0000","last_became_active":"2024-04-24T08:18:47.324956+0000","last_became_peered":"2024-04-24T08:18:47.324956+0000","last_unstale":"2024-04-24T08:18:48.288121+0000","last_undegraded":"2024-04-24T08:18:48.288121+0000","last_fullsized":"2024-04-24T08:18:48.288121+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:44:37.431264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.508788+0000","last_change":"2024-04-24T08:18:46.450710+0000","last_active":"2024-04-24T08:18:50.508788+0000","last_peered":"2024-04-24T08:18:50.508788+0000","last_clean":"2024-04-24T08:18:50.508788+0000","last_became_active":"2024-04-24T08:18:46.450391+0000","last_became_peered":"2024-04-24T08:18:46.450391+0000","last_unstale":"2024-04-24T08:18:50.508788+0000","last_undegraded":"2024-04-24T08:18:50.508788+0000","last_fullsized":"2024-04-24T08:18:50.508788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:19.905524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467654+0000","last_change":"2024-04-24T08:18:47.292114+0000","last_active":"2024-04-24T08:18:48.467654+0000","last_peered":"2024-04-24T08:18:48.467654+0000","last_clean":"2024-04-24T08:18:48.467654+0000","last_became_active":"2024-04-24T08:18:47.291787+0000","last_became_peered":"2024-04-24T08:18:47.291787+0000","last_unstale":"2024-04-24T08:18:48.467654+0000","last_undegraded":"2024-04-24T08:18:48.467654+0000","last_fullsized":"2024-04-24T08:18:48.467654+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:37.020368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.509331+0000","last_change":"2024-04-24T08:18:46.446027+0000","last_active":"2024-04-24T08:18:50.509331+0000","last_peered":"2024-04-24T08:18:50.509331+0000","last_clean":"2024-04-24T08:18:50.509331+0000","last_became_active":"2024-04-24T08:18:46.445634+0000","last_became_peered":"2024-04-24T08:18:46.445634+0000","last_unstale":"2024-04-24T08:18:50.509331+0000","last_undegraded":"2024-04-24T08:18:50.509331+0000","last_fullsized":"2024-04-24T08:18:50.509331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:46.208560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892558+0000","last_change":"2024-04-24T08:18:47.307800+0000","last_active":"2024-04-24T08:18:48.892558+0000","last_peered":"2024-04-24T08:18:48.892558+0000","last_clean":"2024-04-24T08:18:48.892558+0000","last_became_active":"2024-04-24T08:18:47.307501+0000","last_became_peered":"2024-04-24T08:18:47.307501+0000","last_unstale":"2024-04-24T08:18:48.892558+0000","last_undegraded":"2024-04-24T08:18:48.892558+0000","last_fullsized":"2024-04-24T08:18:48.892558+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:04:05.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893119+0000","last_change":"2024-04-24T08:18:46.442811+0000","last_active":"2024-04-24T08:18:48.893119+0000","last_peered":"2024-04-24T08:18:48.893119+0000","last_clean":"2024-04-24T08:18:48.893119+0000","last_became_active":"2024-04-24T08:18:46.442479+0000","last_became_peered":"2024-04-24T08:18:46.442479+0000","last_unstale":"2024-04-24T08:18:48.893119+0000","last_undegraded":"2024-04-24T08:18:48.893119+0000","last_fullsized":"2024-04-24T08:18:48.893119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:24:07.236529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.470069+0000","last_change":"2024-04-24T08:18:47.325046+0000","last_active":"2024-04-24T08:18:48.470069+0000","last_peered":"2024-04-24T08:18:48.470069+0000","last_clean":"2024-04-24T08:18:48.470069+0000","last_became_active":"2024-04-24T08:18:47.324720+0000","last_became_peered":"2024-04-24T08:18:47.324720+0000","last_unstale":"2024-04-24T08:18:48.470069+0000","last_undegraded":"2024-04-24T08:18:48.470069+0000","last_fullsized":"2024-04-24T08:18:48.470069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:11.240388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.503814+0000","last_change":"2024-04-24T08:18:46.440572+0000","last_active":"2024-04-24T08:18:50.503814+0000","last_peered":"2024-04-24T08:18:50.503814+0000","last_clean":"2024-04-24T08:18:50.503814+0000","last_became_active":"2024-04-24T08:18:46.440216+0000","last_became_peered":"2024-04-24T08:18:46.440216+0000","last_unstale":"2024-04-24T08:18:50.503814+0000","last_undegraded":"2024-04-24T08:18:50.503814+0000","last_fullsized":"2024-04-24T08:18:50.503814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:51.352874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221375+0000","last_change":"2024-04-24T08:18:47.293465+0000","last_active":"2024-04-24T08:18:49.221375+0000","last_peered":"2024-04-24T08:18:49.221375+0000","last_clean":"2024-04-24T08:18:49.221375+0000","last_became_active":"2024-04-24T08:18:47.293062+0000","last_became_peered":"2024-04-24T08:18:47.293062+0000","last_unstale":"2024-04-24T08:18:49.221375+0000","last_undegraded":"2024-04-24T08:18:49.221375+0000","last_fullsized":"2024-04-24T08:18:49.221375+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:36.029511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893467+0000","last_change":"2024-04-24T08:18:46.451007+0000","last_active":"2024-04-24T08:18:48.893467+0000","last_peered":"2024-04-24T08:18:48.893467+0000","last_clean":"2024-04-24T08:18:48.893467+0000","last_became_active":"2024-04-24T08:18:46.450676+0000","last_became_peered":"2024-04-24T08:18:46.450676+0000","last_unstale":"2024-04-24T08:18:48.893467+0000","last_undegraded":"2024-04-24T08:18:48.893467+0000","last_fullsized":"2024-04-24T08:18:48.893467+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:40:00.329311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221573+0000","last_change":"2024-04-24T08:18:47.293859+0000","last_active":"2024-04-24T08:18:49.221573+0000","last_peered":"2024-04-24T08:18:49.221573+0000","last_clean":"2024-04-24T08:18:49.221573+0000","last_became_active":"2024-04-24T08:18:47.293509+0000","last_became_peered":"2024-04-24T08:18:47.293509+0000","last_unstale":"2024-04-24T08:18:49.221573+0000","last_undegraded":"2024-04-24T08:18:49.221573+0000","last_fullsized":"2024-04-24T08:18:49.221573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:41.224037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.018673+0000","last_change":"2024-04-24T08:18:46.436661+0000","last_active":"2024-04-24T08:18:49.018673+0000","last_peered":"2024-04-24T08:18:49.018673+0000","last_clean":"2024-04-24T08:18:49.018673+0000","last_became_active":"2024-04-24T08:18:46.436353+0000","last_became_peered":"2024-04-24T08:18:46.436353+0000","last_unstale":"2024-04-24T08:18:49.018673+0000","last_undegraded":"2024-04-24T08:18:49.018673+0000","last_fullsized":"2024-04-24T08:18:49.018673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:57:25.559706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221435+0000","last_change":"2024-04-24T08:18:47.314372+0000","last_active":"2024-04-24T08:18:49.221435+0000","last_peered":"2024-04-24T08:18:49.221435+0000","last_clean":"2024-04-24T08:18:49.221435+0000","last_became_active":"2024-04-24T08:18:47.313979+0000","last_became_peered":"2024-04-24T08:18:47.313979+0000","last_unstale":"2024-04-24T08:18:49.221435+0000","last_undegraded":"2024-04-24T08:18:49.221435+0000","last_fullsized":"2024-04-24T08:18:49.221435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:05.364955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284824+0000","last_change":"2024-04-24T08:18:46.432499+0000","last_active":"2024-04-24T08:18:48.284824+0000","last_peered":"2024-04-24T08:18:48.284824+0000","last_clean":"2024-04-24T08:18:48.284824+0000","last_became_active":"2024-04-24T08:18:46.432085+0000","last_became_peered":"2024-04-24T08:18:46.432085+0000","last_unstale":"2024-04-24T08:18:48.284824+0000","last_undegraded":"2024-04-24T08:18:48.284824+0000","last_fullsized":"2024-04-24T08:18:48.284824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:03.794091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221585+0000","last_change":"2024-04-24T08:18:47.291193+0000","last_active":"2024-04-24T08:18:49.221585+0000","last_peered":"2024-04-24T08:18:49.221585+0000","last_clean":"2024-04-24T08:18:49.221585+0000","last_became_active":"2024-04-24T08:18:47.289903+0000","last_became_peered":"2024-04-24T08:18:47.289903+0000","last_unstale":"2024-04-24T08:18:49.221585+0000","last_undegraded":"2024-04-24T08:18:49.221585+0000","last_fullsized":"2024-04-24T08:18:49.221585+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:35:18.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892349+0000","last_change":"2024-04-24T08:18:47.292628+0000","last_active":"2024-04-24T08:18:48.892349+0000","last_peered":"2024-04-24T08:18:48.892349+0000","last_clean":"2024-04-24T08:18:48.892349+0000","last_became_active":"2024-04-24T08:18:47.292060+0000","last_became_peered":"2024-04-24T08:18:47.292060+0000","last_unstale":"2024-04-24T08:18:48.892349+0000","last_undegraded":"2024-04-24T08:18:48.892349+0000","last_fullsized":"2024-04-24T08:18:48.892349+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:53.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892627+0000","last_change":"2024-04-24T08:18:47.297121+0000","last_active":"2024-04-24T08:18:48.892627+0000","last_peered":"2024-04-24T08:18:48.892627+0000","last_clean":"2024-04-24T08:18:48.892627+0000","last_became_active":"2024-04-24T08:18:47.296787+0000","last_became_peered":"2024-04-24T08:18:47.296787+0000","last_unstale":"2024-04-24T08:18:48.892627+0000","last_undegraded":"2024-04-24T08:18:48.892627+0000","last_fullsized":"2024-04-24T08:18:48.892627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:44:10.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892995+0000","last_change":"2024-04-24T08:18:47.320669+0000","last_active":"2024-04-24T08:18:48.892995+0000","last_peered":"2024-04-24T08:18:48.892995+0000","last_clean":"2024-04-24T08:18:48.892995+0000","last_became_active":"2024-04-24T08:18:47.320287+0000","last_became_peered":"2024-04-24T08:18:47.320287+0000","last_unstale":"2024-04-24T08:18:48.892995+0000","last_undegraded":"2024-04-24T08:18:48.892995+0000","last_fullsized":"2024-04-24T08:18:48.892995+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:39:03.275189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287058+0000","last_change":"2024-04-24T08:18:47.320947+0000","last_active":"2024-04-24T08:18:48.287058+0000","last_peered":"2024-04-24T08:18:48.287058+0000","last_clean":"2024-04-24T08:18:48.287058+0000","last_became_active":"2024-04-24T08:18:47.314939+0000","last_became_peered":"2024-04-24T08:18:47.314939+0000","last_unstale":"2024-04-24T08:18:48.287058+0000","last_undegraded":"2024-04-24T08:18:48.287058+0000","last_fullsized":"2024-04-24T08:18:48.287058+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:40.183172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284657+0000","last_change":"2024-04-24T08:18:47.316068+0000","last_active":"2024-04-24T08:18:48.284657+0000","last_peered":"2024-04-24T08:18:48.284657+0000","last_clean":"2024-04-24T08:18:48.284657+0000","last_became_active":"2024-04-24T08:18:47.315647+0000","last_became_peered":"2024-04-24T08:18:47.315647+0000","last_unstale":"2024-04-24T08:18:48.284657+0000","last_undegraded":"2024-04-24T08:18:48.284657+0000","last_fullsized":"2024-04-24T08:18:48.284657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:33.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892833+0000","last_change":"2024-04-24T08:18:47.294966+0000","last_active":"2024-04-24T08:18:48.892833+0000","last_peered":"2024-04-24T08:18:48.892833+0000","last_clean":"2024-04-24T08:18:48.892833+0000","last_became_active":"2024-04-24T08:18:47.294672+0000","last_became_peered":"2024-04-24T08:18:47.294672+0000","last_unstale":"2024-04-24T08:18:48.892833+0000","last_undegraded":"2024-04-24T08:18:48.892833+0000","last_fullsized":"2024-04-24T08:18:48.892833+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:37.359114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893830+0000","last_change":"2024-04-24T08:18:47.294548+0000","last_active":"2024-04-24T08:18:48.893830+0000","last_peered":"2024-04-24T08:18:48.893830+0000","last_clean":"2024-04-24T08:18:48.893830+0000","last_became_active":"2024-04-24T08:18:47.293589+0000","last_became_peered":"2024-04-24T08:18:47.293589+0000","last_unstale":"2024-04-24T08:18:48.893830+0000","last_undegraded":"2024-04-24T08:18:48.893830+0000","last_fullsized":"2024-04-24T08:18:48.893830+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:34:56.208453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.471008+0000","last_change":"2024-04-24T08:18:47.313156+0000","last_active":"2024-04-24T08:18:48.471008+0000","last_peered":"2024-04-24T08:18:48.471008+0000","last_clean":"2024-04-24T08:18:48.471008+0000","last_became_active":"2024-04-24T08:18:47.312829+0000","last_became_peered":"2024-04-24T08:18:47.312829+0000","last_unstale":"2024-04-24T08:18:48.471008+0000","last_undegraded":"2024-04-24T08:18:48.471008+0000","last_fullsized":"2024-04-24T08:18:48.471008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:28:36.638514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892457+0000","last_change":"2024-04-24T08:18:47.320022+0000","last_active":"2024-04-24T08:18:48.892457+0000","last_peered":"2024-04-24T08:18:48.892457+0000","last_clean":"2024-04-24T08:18:48.892457+0000","last_became_active":"2024-04-24T08:18:47.319707+0000","last_became_peered":"2024-04-24T08:18:47.319707+0000","last_unstale":"2024-04-24T08:18:48.892457+0000","last_undegraded":"2024-04-24T08:18:48.892457+0000","last_fullsized":"2024-04-24T08:18:48.892457+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:51.330874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.469845+0000","last_change":"2024-04-24T08:18:47.323187+0000","last_active":"2024-04-24T08:18:48.469845+0000","last_peered":"2024-04-24T08:18:48.469845+0000","last_clean":"2024-04-24T08:18:48.469845+0000","last_became_active":"2024-04-24T08:18:47.320258+0000","last_became_peered":"2024-04-24T08:18:47.320258+0000","last_unstale":"2024-04-24T08:18:48.469845+0000","last_undegraded":"2024-04-24T08:18:48.469845+0000","last_fullsized":"2024-04-24T08:18:48.469845+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:42:32.770950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892073+0000","last_change":"2024-04-24T08:18:47.321926+0000","last_active":"2024-04-24T08:18:48.892073+0000","last_peered":"2024-04-24T08:18:48.892073+0000","last_clean":"2024-04-24T08:18:48.892073+0000","last_became_active":"2024-04-24T08:18:47.321584+0000","last_became_peered":"2024-04-24T08:18:47.321584+0000","last_unstale":"2024-04-24T08:18:48.892073+0000","last_undegraded":"2024-04-24T08:18:48.892073+0000","last_fullsized":"2024-04-24T08:18:48.892073+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:24.357157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284664+0000","last_change":"2024-04-24T08:18:47.313310+0000","last_active":"2024-04-24T08:18:48.284664+0000","last_peered":"2024-04-24T08:18:48.284664+0000","last_clean":"2024-04-24T08:18:48.284664+0000","last_became_active":"2024-04-24T08:18:47.312968+0000","last_became_peered":"2024-04-24T08:18:47.312968+0000","last_unstale":"2024-04-24T08:18:48.284664+0000","last_undegraded":"2024-04-24T08:18:48.284664+0000","last_fullsized":"2024-04-24T08:18:48.284664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:08.967577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892697+0000","last_change":"2024-04-24T08:18:47.296725+0000","last_active":"2024-04-24T08:18:48.892697+0000","last_peered":"2024-04-24T08:18:48.892697+0000","last_clean":"2024-04-24T08:18:48.892697+0000","last_became_active":"2024-04-24T08:18:47.296406+0000","last_became_peered":"2024-04-24T08:18:47.296406+0000","last_unstale":"2024-04-24T08:18:48.892697+0000","last_undegraded":"2024-04-24T08:18:48.892697+0000","last_fullsized":"2024-04-24T08:18:48.892697+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:04:59.487399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4941,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":92,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":11929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285582,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":452438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547218,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1576960,"data_stored":1044792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710745,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":452640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972384,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":452594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266723,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1552384,"data_stored":1043428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430249,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":451856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":453468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":451946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":453502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":954368,"data_stored":451960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":452500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691888,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1568768,"data_stored":1044091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:18:56.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 -- 172.21.15.146:0/4088364979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060104260 msgr2=0x7fd060104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 --2- 172.21.15.146:0/4088364979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060104260 0x7fd060104680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd050009900 tx=0x7fd05002a410 comp rx=0 tx=0).stop 2024-04-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 -- 172.21.15.146:0/4088364979 shutdown_connections 2024-04-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 --2- 172.21.15.146:0/4088364979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd060105e10 0x7fd06010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 --2- 172.21.15.146:0/4088364979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd060105450 0x7fd0601058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 --2- 172.21.15.146:0/4088364979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060104260 0x7fd060104680 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 -- 172.21.15.146:0/4088364979 >> 172.21.15.146:0/4088364979 conn(0x7fd0600ff7f0 msgr2=0x7fd060101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 -- 172.21.15.146:0/4088364979 shutdown_connections 2024-04-24T08:18:56.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.662+0000 7fd065600640 1 -- 172.21.15.146:0/4088364979 wait complete. 2024-04-24T08:18:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.663+0000 7fd065600640 1 Processor -- start 2024-04-24T08:18:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.663+0000 7fd065600640 1 -- start start 2024-04-24T08:18:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd065600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd060104260 0x7fd06019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd065600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060105450 0x7fd06019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd065600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd060105e10 0x7fd0601a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd065600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd06010ff40 con 0x7fd060105450 2024-04-24T08:18:56.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd065600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd06010fdc0 con 0x7fd060105e10 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd065600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0601100c0 con 0x7fd060104260 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060105450 0x7fd06019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060105450 0x7fd06019aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59362/0 (socket says 172.21.15.146:59362) 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 -- 172.21.15.146:0/1709803972 learned_addr learned my addr 172.21.15.146:0/1709803972 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd060104260 0x7fd06019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05f400640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd060105e10 0x7fd0601a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 -- 172.21.15.146:0/1709803972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd060104260 msgr2=0x7fd06019a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd060104260 0x7fd06019a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 -- 172.21.15.146:0/1709803972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd060105e10 msgr2=0x7fd0601a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd060105e10 0x7fd0601a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.664+0000 7fd05e000640 1 -- 172.21.15.146:0/1709803972 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0601a2150 con 0x7fd060105450 2024-04-24T08:18:56.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.665+0000 7fd05e000640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060105450 0x7fd06019aa70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00e9f0 tx=0x7fd04c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.665+0000 7fd057400640 1 -- 172.21.15.146:0/1709803972 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c00cd80 con 0x7fd060105450 2024-04-24T08:18:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.665+0000 7fd057400640 1 -- 172.21.15.146:0/1709803972 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd04c010910 con 0x7fd060105450 2024-04-24T08:18:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.665+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0601a2440 con 0x7fd060105450 2024-04-24T08:18:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.665+0000 7fd057400640 1 -- 172.21.15.146:0/1709803972 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c018990 con 0x7fd060105450 2024-04-24T08:18:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.665+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0601a2990 con 0x7fd060105450 2024-04-24T08:18:56.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.666+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd06010b580 con 0x7fd060105450 2024-04-24T08:18:56.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.669+0000 7fd057400640 1 -- 172.21.15.146:0/1709803972 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd04c010ac0 con 0x7fd060105450 2024-04-24T08:18:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.670+0000 7fd057400640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd0440a3c40 0x7fd0440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.670+0000 7fd057400640 1 -- 172.21.15.146:0/1709803972 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8240+0+0 (secure 0 0 0) 0x7fd04c0d0900 con 0x7fd060105450 2024-04-24T08:18:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.671+0000 7fd05ea00640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd0440a3c40 0x7fd0440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:56.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.671+0000 7fd05ea00640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd0440a3c40 0x7fd0440a6100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd05002aa00 tx=0x7fd0500315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:56.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.674+0000 7fd057400640 1 -- 172.21.15.146:0/1709803972 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd04c09caf0 con 0x7fd060105450 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3093784944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-04-24T08:18:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-24T08:18:56.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:56 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3093784944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-04-24T08:18:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-24T08:18:56.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:56 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-24T08:18:56.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.877+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd060062440 con 0x7fd0440a3c40 2024-04-24T08:18:56.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.900+0000 7fd057400640 1 -- 172.21.15.146:0/1709803972 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258743 (secure 0 0 0) 0x7fd060062440 con 0x7fd0440a3c40 2024-04-24T08:18:56.906 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:18:56.908 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-24T08:18:56.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd0440a3c40 msgr2=0x7fd0440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd0440a3c40 0x7fd0440a6100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd05002aa00 tx=0x7fd0500315b0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060105450 msgr2=0x7fd06019aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:56.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060105450 0x7fd06019aa70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00e9f0 tx=0x7fd04c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 shutdown_connections 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd0440a3c40 0x7fd0440a6100 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd060105e10 0x7fd0601a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd060105450 0x7fd06019aa70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 --2- 172.21.15.146:0/1709803972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd060104260 0x7fd06019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 >> 172.21.15.146:0/1709803972 conn(0x7fd0600ff7f0 msgr2=0x7fd060108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 shutdown_connections 2024-04-24T08:18:56.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:56.907+0000 7fd065600640 1 -- 172.21.15.146:0/1709803972 wait complete. 2024-04-24T08:18:56.928 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-04-24T08:18:56.559656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632796,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":173,"num_read_kb":182,"num_write":325,"num_write_kb":3020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374828,"kb_used_data":13196,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566996,"statfs":{"total":1151940427776,"available":1151556603904,"internally_reserved":0,"allocated":13512704,"data_stored":7320543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42428,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":37,"num_read_kb":63,"num_write":144,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.279020"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893273+0000","last_change":"2024-04-24T08:18:47.309666+0000","last_active":"2024-04-24T08:18:48.893273+0000","last_peered":"2024-04-24T08:18:48.893273+0000","last_clean":"2024-04-24T08:18:48.893273+0000","last_became_active":"2024-04-24T08:18:47.309413+0000","last_became_peered":"2024-04-24T08:18:47.309413+0000","last_unstale":"2024-04-24T08:18:48.893273+0000","last_undegraded":"2024-04-24T08:18:48.893273+0000","last_fullsized":"2024-04-24T08:18:48.893273+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:29.662505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287787+0000","last_change":"2024-04-24T08:18:47.310321+0000","last_active":"2024-04-24T08:18:48.287787+0000","last_peered":"2024-04-24T08:18:48.287787+0000","last_clean":"2024-04-24T08:18:48.287787+0000","last_became_active":"2024-04-24T08:18:47.309956+0000","last_became_peered":"2024-04-24T08:18:47.309956+0000","last_unstale":"2024-04-24T08:18:48.287787+0000","last_undegraded":"2024-04-24T08:18:48.287787+0000","last_fullsized":"2024-04-24T08:18:48.287787+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:51:48.432725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287956+0000","last_change":"2024-04-24T08:18:47.312153+0000","last_active":"2024-04-24T08:18:48.287956+0000","last_peered":"2024-04-24T08:18:48.287956+0000","last_clean":"2024-04-24T08:18:48.287956+0000","last_became_active":"2024-04-24T08:18:47.309465+0000","last_became_peered":"2024-04-24T08:18:47.309465+0000","last_unstale":"2024-04-24T08:18:48.287956+0000","last_undegraded":"2024-04-24T08:18:48.287956+0000","last_fullsized":"2024-04-24T08:18:48.287956+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:55.829143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892547+0000","last_change":"2024-04-24T08:18:47.297017+0000","last_active":"2024-04-24T08:18:48.892547+0000","last_peered":"2024-04-24T08:18:48.892547+0000","last_clean":"2024-04-24T08:18:48.892547+0000","last_became_active":"2024-04-24T08:18:47.296693+0000","last_became_peered":"2024-04-24T08:18:47.296693+0000","last_unstale":"2024-04-24T08:18:48.892547+0000","last_undegraded":"2024-04-24T08:18:48.892547+0000","last_fullsized":"2024-04-24T08:18:48.892547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:23.400403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467594+0000","last_change":"2024-04-24T08:18:47.315376+0000","last_active":"2024-04-24T08:18:48.467594+0000","last_peered":"2024-04-24T08:18:48.467594+0000","last_clean":"2024-04-24T08:18:48.467594+0000","last_became_active":"2024-04-24T08:18:47.315027+0000","last_became_peered":"2024-04-24T08:18:47.315027+0000","last_unstale":"2024-04-24T08:18:48.467594+0000","last_undegraded":"2024-04-24T08:18:48.467594+0000","last_fullsized":"2024-04-24T08:18:48.467594+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:12.504515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282490+0000","last_change":"2024-04-24T08:18:47.306388+0000","last_active":"2024-04-24T08:18:48.282490+0000","last_peered":"2024-04-24T08:18:48.282490+0000","last_clean":"2024-04-24T08:18:48.282490+0000","last_became_active":"2024-04-24T08:18:47.306082+0000","last_became_peered":"2024-04-24T08:18:47.306082+0000","last_unstale":"2024-04-24T08:18:48.282490+0000","last_undegraded":"2024-04-24T08:18:48.282490+0000","last_fullsized":"2024-04-24T08:18:48.282490+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:34:52.612799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221440+0000","last_change":"2024-04-24T08:18:47.303336+0000","last_active":"2024-04-24T08:18:49.221440+0000","last_peered":"2024-04-24T08:18:49.221440+0000","last_clean":"2024-04-24T08:18:49.221440+0000","last_became_active":"2024-04-24T08:18:47.303008+0000","last_became_peered":"2024-04-24T08:18:47.303008+0000","last_unstale":"2024-04-24T08:18:49.221440+0000","last_undegraded":"2024-04-24T08:18:49.221440+0000","last_fullsized":"2024-04-24T08:18:49.221440+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:57.404693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.469361+0000","last_change":"2024-04-24T08:18:47.316672+0000","last_active":"2024-04-24T08:18:48.469361+0000","last_peered":"2024-04-24T08:18:48.469361+0000","last_clean":"2024-04-24T08:18:48.469361+0000","last_became_active":"2024-04-24T08:18:47.315741+0000","last_became_peered":"2024-04-24T08:18:47.315741+0000","last_unstale":"2024-04-24T08:18:48.469361+0000","last_undegraded":"2024-04-24T08:18:48.469361+0000","last_fullsized":"2024-04-24T08:18:48.469361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:29.156710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892767+0000","last_change":"2024-04-24T08:18:47.291733+0000","last_active":"2024-04-24T08:18:48.892767+0000","last_peered":"2024-04-24T08:18:48.892767+0000","last_clean":"2024-04-24T08:18:48.892767+0000","last_became_active":"2024-04-24T08:18:47.291432+0000","last_became_peered":"2024-04-24T08:18:47.291432+0000","last_unstale":"2024-04-24T08:18:48.892767+0000","last_undegraded":"2024-04-24T08:18:48.892767+0000","last_fullsized":"2024-04-24T08:18:48.892767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:38:03.601540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282862+0000","last_change":"2024-04-24T08:18:47.294544+0000","last_active":"2024-04-24T08:18:48.282862+0000","last_peered":"2024-04-24T08:18:48.282862+0000","last_clean":"2024-04-24T08:18:48.282862+0000","last_became_active":"2024-04-24T08:18:47.294183+0000","last_became_peered":"2024-04-24T08:18:47.294183+0000","last_unstale":"2024-04-24T08:18:48.282862+0000","last_undegraded":"2024-04-24T08:18:48.282862+0000","last_fullsized":"2024-04-24T08:18:48.282862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:53.082644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221517+0000","last_change":"2024-04-24T08:18:47.292615+0000","last_active":"2024-04-24T08:18:49.221517+0000","last_peered":"2024-04-24T08:18:49.221517+0000","last_clean":"2024-04-24T08:18:49.221517+0000","last_became_active":"2024-04-24T08:18:47.292300+0000","last_became_peered":"2024-04-24T08:18:47.292300+0000","last_unstale":"2024-04-24T08:18:49.221517+0000","last_undegraded":"2024-04-24T08:18:49.221517+0000","last_fullsized":"2024-04-24T08:18:49.221517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:58.293253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892467+0000","last_change":"2024-04-24T08:18:47.298388+0000","last_active":"2024-04-24T08:18:48.892467+0000","last_peered":"2024-04-24T08:18:48.892467+0000","last_clean":"2024-04-24T08:18:48.892467+0000","last_became_active":"2024-04-24T08:18:47.298008+0000","last_became_peered":"2024-04-24T08:18:47.298008+0000","last_unstale":"2024-04-24T08:18:48.892467+0000","last_undegraded":"2024-04-24T08:18:48.892467+0000","last_fullsized":"2024-04-24T08:18:48.892467+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:10:00.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284912+0000","last_change":"2024-04-24T08:18:47.298809+0000","last_active":"2024-04-24T08:18:48.284912+0000","last_peered":"2024-04-24T08:18:48.284912+0000","last_clean":"2024-04-24T08:18:48.284912+0000","last_became_active":"2024-04-24T08:18:47.298103+0000","last_became_peered":"2024-04-24T08:18:47.298103+0000","last_unstale":"2024-04-24T08:18:48.284912+0000","last_undegraded":"2024-04-24T08:18:48.284912+0000","last_fullsized":"2024-04-24T08:18:48.284912+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:38:53.625721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287023+0000","last_change":"2024-04-24T08:18:47.314572+0000","last_active":"2024-04-24T08:18:48.287023+0000","last_peered":"2024-04-24T08:18:48.287023+0000","last_clean":"2024-04-24T08:18:48.287023+0000","last_became_active":"2024-04-24T08:18:47.313372+0000","last_became_peered":"2024-04-24T08:18:47.313372+0000","last_unstale":"2024-04-24T08:18:48.287023+0000","last_undegraded":"2024-04-24T08:18:48.287023+0000","last_fullsized":"2024-04-24T08:18:48.287023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:56:01.954909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.286869+0000","last_change":"2024-04-24T08:18:47.314822+0000","last_active":"2024-04-24T08:18:48.286869+0000","last_peered":"2024-04-24T08:18:48.286869+0000","last_clean":"2024-04-24T08:18:48.286869+0000","last_became_active":"2024-04-24T08:18:47.310774+0000","last_became_peered":"2024-04-24T08:18:47.310774+0000","last_unstale":"2024-04-24T08:18:48.286869+0000","last_undegraded":"2024-04-24T08:18:48.286869+0000","last_fullsized":"2024-04-24T08:18:48.286869+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:24.934100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.891934+0000","last_change":"2024-04-24T08:18:47.307208+0000","last_active":"2024-04-24T08:18:48.891934+0000","last_peered":"2024-04-24T08:18:48.891934+0000","last_clean":"2024-04-24T08:18:48.891934+0000","last_became_active":"2024-04-24T08:18:47.306889+0000","last_became_peered":"2024-04-24T08:18:47.306889+0000","last_unstale":"2024-04-24T08:18:48.891934+0000","last_undegraded":"2024-04-24T08:18:48.891934+0000","last_fullsized":"2024-04-24T08:18:48.891934+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:20:01.438388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893047+0000","last_change":"2024-04-24T08:18:47.322929+0000","last_active":"2024-04-24T08:18:48.893047+0000","last_peered":"2024-04-24T08:18:48.893047+0000","last_clean":"2024-04-24T08:18:48.893047+0000","last_became_active":"2024-04-24T08:18:47.322593+0000","last_became_peered":"2024-04-24T08:18:47.322593+0000","last_unstale":"2024-04-24T08:18:48.893047+0000","last_undegraded":"2024-04-24T08:18:48.893047+0000","last_fullsized":"2024-04-24T08:18:48.893047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:09.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.465031+0000","last_change":"2024-04-24T08:18:47.304886+0000","last_active":"2024-04-24T08:18:48.465031+0000","last_peered":"2024-04-24T08:18:48.465031+0000","last_clean":"2024-04-24T08:18:48.465031+0000","last_became_active":"2024-04-24T08:18:47.302631+0000","last_became_peered":"2024-04-24T08:18:47.302631+0000","last_unstale":"2024-04-24T08:18:48.465031+0000","last_undegraded":"2024-04-24T08:18:48.465031+0000","last_fullsized":"2024-04-24T08:18:48.465031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:13.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893769+0000","last_change":"2024-04-24T08:18:47.316828+0000","last_active":"2024-04-24T08:18:48.893769+0000","last_peered":"2024-04-24T08:18:48.893769+0000","last_clean":"2024-04-24T08:18:48.893769+0000","last_became_active":"2024-04-24T08:18:47.316540+0000","last_became_peered":"2024-04-24T08:18:47.316540+0000","last_unstale":"2024-04-24T08:18:48.893769+0000","last_undegraded":"2024-04-24T08:18:48.893769+0000","last_fullsized":"2024-04-24T08:18:48.893769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:41:42.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.526805+0000","last_change":"2024-04-24T08:18:46.444744+0000","last_active":"2024-04-24T08:18:51.526805+0000","last_peered":"2024-04-24T08:18:51.526805+0000","last_clean":"2024-04-24T08:18:51.526805+0000","last_became_active":"2024-04-24T08:18:46.442495+0000","last_became_peered":"2024-04-24T08:18:46.442495+0000","last_unstale":"2024-04-24T08:18:51.526805+0000","last_undegraded":"2024-04-24T08:18:51.526805+0000","last_fullsized":"2024-04-24T08:18:51.526805+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:33:32.270984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.471046+0000","last_change":"2024-04-24T08:18:47.320462+0000","last_active":"2024-04-24T08:18:48.471046+0000","last_peered":"2024-04-24T08:18:48.471046+0000","last_clean":"2024-04-24T08:18:48.471046+0000","last_became_active":"2024-04-24T08:18:47.319267+0000","last_became_peered":"2024-04-24T08:18:47.319267+0000","last_unstale":"2024-04-24T08:18:48.471046+0000","last_undegraded":"2024-04-24T08:18:48.471046+0000","last_fullsized":"2024-04-24T08:18:48.471046+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:50:15.882077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.470893+0000","last_change":"2024-04-24T08:18:46.453278+0000","last_active":"2024-04-24T08:18:48.470893+0000","last_peered":"2024-04-24T08:18:48.470893+0000","last_clean":"2024-04-24T08:18:48.470893+0000","last_became_active":"2024-04-24T08:18:46.452913+0000","last_became_peered":"2024-04-24T08:18:46.452913+0000","last_unstale":"2024-04-24T08:18:48.470893+0000","last_undegraded":"2024-04-24T08:18:48.470893+0000","last_fullsized":"2024-04-24T08:18:48.470893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:48.272190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.288417+0000","last_change":"2024-04-24T08:18:47.323610+0000","last_active":"2024-04-24T08:18:48.288417+0000","last_peered":"2024-04-24T08:18:48.288417+0000","last_clean":"2024-04-24T08:18:48.288417+0000","last_became_active":"2024-04-24T08:18:47.322795+0000","last_became_peered":"2024-04-24T08:18:47.322795+0000","last_unstale":"2024-04-24T08:18:48.288417+0000","last_undegraded":"2024-04-24T08:18:48.288417+0000","last_fullsized":"2024-04-24T08:18:48.288417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:32.951564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.522260+0000","last_change":"2024-04-24T08:18:46.438815+0000","last_active":"2024-04-24T08:18:51.522260+0000","last_peered":"2024-04-24T08:18:51.522260+0000","last_clean":"2024-04-24T08:18:51.522260+0000","last_became_active":"2024-04-24T08:18:46.438523+0000","last_became_peered":"2024-04-24T08:18:46.438523+0000","last_unstale":"2024-04-24T08:18:51.522260+0000","last_undegraded":"2024-04-24T08:18:51.522260+0000","last_fullsized":"2024-04-24T08:18:51.522260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:50:57.222540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892006+0000","last_change":"2024-04-24T08:18:47.304884+0000","last_active":"2024-04-24T08:18:48.892006+0000","last_peered":"2024-04-24T08:18:48.892006+0000","last_clean":"2024-04-24T08:18:48.892006+0000","last_became_active":"2024-04-24T08:18:47.304520+0000","last_became_peered":"2024-04-24T08:18:47.304520+0000","last_unstale":"2024-04-24T08:18:48.892006+0000","last_undegraded":"2024-04-24T08:18:48.892006+0000","last_fullsized":"2024-04-24T08:18:48.892006+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:42.325010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467940+0000","last_change":"2024-04-24T08:18:46.441073+0000","last_active":"2024-04-24T08:18:48.467940+0000","last_peered":"2024-04-24T08:18:48.467940+0000","last_clean":"2024-04-24T08:18:48.467940+0000","last_became_active":"2024-04-24T08:18:46.440721+0000","last_became_peered":"2024-04-24T08:18:46.440721+0000","last_unstale":"2024-04-24T08:18:48.467940+0000","last_undegraded":"2024-04-24T08:18:48.467940+0000","last_fullsized":"2024-04-24T08:18:48.467940+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:51:39.516458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893555+0000","last_change":"2024-04-24T08:18:47.293947+0000","last_active":"2024-04-24T08:18:48.893555+0000","last_peered":"2024-04-24T08:18:48.893555+0000","last_clean":"2024-04-24T08:18:48.893555+0000","last_became_active":"2024-04-24T08:18:47.293623+0000","last_became_peered":"2024-04-24T08:18:47.293623+0000","last_unstale":"2024-04-24T08:18:48.893555+0000","last_undegraded":"2024-04-24T08:18:48.893555+0000","last_fullsized":"2024-04-24T08:18:48.893555+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:23.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:53.529149+0000","last_change":"2024-04-24T08:18:46.439909+0000","last_active":"2024-04-24T08:18:53.529149+0000","last_peered":"2024-04-24T08:18:53.529149+0000","last_clean":"2024-04-24T08:18:53.529149+0000","last_became_active":"2024-04-24T08:18:46.439597+0000","last_became_peered":"2024-04-24T08:18:46.439597+0000","last_unstale":"2024-04-24T08:18:53.529149+0000","last_undegraded":"2024-04-24T08:18:53.529149+0000","last_fullsized":"2024-04-24T08:18:53.529149+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:49:11.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467643+0000","last_change":"2024-04-24T08:18:47.306480+0000","last_active":"2024-04-24T08:18:48.467643+0000","last_peered":"2024-04-24T08:18:48.467643+0000","last_clean":"2024-04-24T08:18:48.467643+0000","last_became_active":"2024-04-24T08:18:47.306104+0000","last_became_peered":"2024-04-24T08:18:47.306104+0000","last_unstale":"2024-04-24T08:18:48.467643+0000","last_undegraded":"2024-04-24T08:18:48.467643+0000","last_fullsized":"2024-04-24T08:18:48.467643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:56.340836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.525921+0000","last_change":"2024-04-24T08:18:46.439381+0000","last_active":"2024-04-24T08:18:51.525921+0000","last_peered":"2024-04-24T08:18:51.525921+0000","last_clean":"2024-04-24T08:18:51.525921+0000","last_became_active":"2024-04-24T08:18:46.439033+0000","last_became_peered":"2024-04-24T08:18:46.439033+0000","last_unstale":"2024-04-24T08:18:51.525921+0000","last_undegraded":"2024-04-24T08:18:51.525921+0000","last_fullsized":"2024-04-24T08:18:51.525921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:48:28.833066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893176+0000","last_change":"2024-04-24T08:18:47.315402+0000","last_active":"2024-04-24T08:18:48.893176+0000","last_peered":"2024-04-24T08:18:48.893176+0000","last_clean":"2024-04-24T08:18:48.893176+0000","last_became_active":"2024-04-24T08:18:47.315050+0000","last_became_peered":"2024-04-24T08:18:47.315050+0000","last_unstale":"2024-04-24T08:18:48.893176+0000","last_undegraded":"2024-04-24T08:18:48.893176+0000","last_fullsized":"2024-04-24T08:18:48.893176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:59:49.948239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.527219+0000","last_change":"2024-04-24T08:18:46.448151+0000","last_active":"2024-04-24T08:18:51.527219+0000","last_peered":"2024-04-24T08:18:51.527219+0000","last_clean":"2024-04-24T08:18:51.527219+0000","last_became_active":"2024-04-24T08:18:46.447077+0000","last_became_peered":"2024-04-24T08:18:46.447077+0000","last_unstale":"2024-04-24T08:18:51.527219+0000","last_undegraded":"2024-04-24T08:18:51.527219+0000","last_fullsized":"2024-04-24T08:18:51.527219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:00.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.469089+0000","last_change":"2024-04-24T08:18:47.319423+0000","last_active":"2024-04-24T08:18:48.469089+0000","last_peered":"2024-04-24T08:18:48.469089+0000","last_clean":"2024-04-24T08:18:48.469089+0000","last_became_active":"2024-04-24T08:18:47.316284+0000","last_became_peered":"2024-04-24T08:18:47.316284+0000","last_unstale":"2024-04-24T08:18:48.469089+0000","last_undegraded":"2024-04-24T08:18:48.469089+0000","last_fullsized":"2024-04-24T08:18:48.469089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:55.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.018584+0000","last_change":"2024-04-24T08:18:46.436152+0000","last_active":"2024-04-24T08:18:49.018584+0000","last_peered":"2024-04-24T08:18:49.018584+0000","last_clean":"2024-04-24T08:18:49.018584+0000","last_became_active":"2024-04-24T08:18:46.435835+0000","last_became_peered":"2024-04-24T08:18:46.435835+0000","last_unstale":"2024-04-24T08:18:49.018584+0000","last_undegraded":"2024-04-24T08:18:49.018584+0000","last_fullsized":"2024-04-24T08:18:49.018584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:12:30.536455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.463156+0000","last_change":"2024-04-24T08:18:47.326408+0000","last_active":"2024-04-24T08:18:48.463156+0000","last_peered":"2024-04-24T08:18:48.463156+0000","last_clean":"2024-04-24T08:18:48.463156+0000","last_became_active":"2024-04-24T08:18:47.325991+0000","last_became_peered":"2024-04-24T08:18:47.325991+0000","last_unstale":"2024-04-24T08:18:48.463156+0000","last_undegraded":"2024-04-24T08:18:48.463156+0000","last_fullsized":"2024-04-24T08:18:48.463156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:59.727066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:54.530365+0000","last_change":"2024-04-24T08:18:46.446652+0000","last_active":"2024-04-24T08:18:54.530365+0000","last_peered":"2024-04-24T08:18:54.530365+0000","last_clean":"2024-04-24T08:18:54.530365+0000","last_became_active":"2024-04-24T08:18:46.446348+0000","last_became_peered":"2024-04-24T08:18:46.446348+0000","last_unstale":"2024-04-24T08:18:54.530365+0000","last_undegraded":"2024-04-24T08:18:54.530365+0000","last_fullsized":"2024-04-24T08:18:54.530365+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:25.791359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892560+0000","last_change":"2024-04-24T08:18:47.309072+0000","last_active":"2024-04-24T08:18:48.892560+0000","last_peered":"2024-04-24T08:18:48.892560+0000","last_clean":"2024-04-24T08:18:48.892560+0000","last_became_active":"2024-04-24T08:18:47.308729+0000","last_became_peered":"2024-04-24T08:18:47.308729+0000","last_unstale":"2024-04-24T08:18:48.892560+0000","last_undegraded":"2024-04-24T08:18:48.892560+0000","last_fullsized":"2024-04-24T08:18:48.892560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:58.256876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.505650+0000","last_change":"2024-04-24T08:18:46.439190+0000","last_active":"2024-04-24T08:18:50.505650+0000","last_peered":"2024-04-24T08:18:50.505650+0000","last_clean":"2024-04-24T08:18:50.505650+0000","last_became_active":"2024-04-24T08:18:46.438442+0000","last_became_peered":"2024-04-24T08:18:46.438442+0000","last_unstale":"2024-04-24T08:18:50.505650+0000","last_undegraded":"2024-04-24T08:18:50.505650+0000","last_fullsized":"2024-04-24T08:18:50.505650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:42.614563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.288228+0000","last_change":"2024-04-24T08:18:47.311447+0000","last_active":"2024-04-24T08:18:48.288228+0000","last_peered":"2024-04-24T08:18:48.288228+0000","last_clean":"2024-04-24T08:18:48.288228+0000","last_became_active":"2024-04-24T08:18:47.310939+0000","last_became_peered":"2024-04-24T08:18:47.310939+0000","last_unstale":"2024-04-24T08:18:48.288228+0000","last_undegraded":"2024-04-24T08:18:48.288228+0000","last_fullsized":"2024-04-24T08:18:48.288228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:42.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.508794+0000","last_change":"2024-04-24T08:18:46.450596+0000","last_active":"2024-04-24T08:18:50.508794+0000","last_peered":"2024-04-24T08:18:50.508794+0000","last_clean":"2024-04-24T08:18:50.508794+0000","last_became_active":"2024-04-24T08:18:46.450226+0000","last_became_peered":"2024-04-24T08:18:46.450226+0000","last_unstale":"2024-04-24T08:18:50.508794+0000","last_undegraded":"2024-04-24T08:18:50.508794+0000","last_fullsized":"2024-04-24T08:18:50.508794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:05.722507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892288+0000","last_change":"2024-04-24T08:18:47.295266+0000","last_active":"2024-04-24T08:18:48.892288+0000","last_peered":"2024-04-24T08:18:48.892288+0000","last_clean":"2024-04-24T08:18:48.892288+0000","last_became_active":"2024-04-24T08:18:47.294963+0000","last_became_peered":"2024-04-24T08:18:47.294963+0000","last_unstale":"2024-04-24T08:18:48.892288+0000","last_undegraded":"2024-04-24T08:18:48.892288+0000","last_fullsized":"2024-04-24T08:18:48.892288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:07.638921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'19","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:55.539272+0000","last_change":"2024-04-24T08:18:46.435524+0000","last_active":"2024-04-24T08:18:55.539272+0000","last_peered":"2024-04-24T08:18:55.539272+0000","last_clean":"2024-04-24T08:18:55.539272+0000","last_became_active":"2024-04-24T08:18:46.435212+0000","last_became_peered":"2024-04-24T08:18:46.435212+0000","last_unstale":"2024-04-24T08:18:55.539272+0000","last_undegraded":"2024-04-24T08:18:55.539272+0000","last_fullsized":"2024-04-24T08:18:55.539272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:45.575905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":42,"num_write":20,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467911+0000","last_change":"2024-04-24T08:18:47.293648+0000","last_active":"2024-04-24T08:18:48.467911+0000","last_peered":"2024-04-24T08:18:48.467911+0000","last_clean":"2024-04-24T08:18:48.467911+0000","last_became_active":"2024-04-24T08:18:47.293260+0000","last_became_peered":"2024-04-24T08:18:47.293260+0000","last_unstale":"2024-04-24T08:18:48.467911+0000","last_undegraded":"2024-04-24T08:18:48.467911+0000","last_fullsized":"2024-04-24T08:18:48.467911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:36.462606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.523831+0000","last_change":"2024-04-24T08:18:46.441401+0000","last_active":"2024-04-24T08:18:51.523831+0000","last_peered":"2024-04-24T08:18:51.523831+0000","last_clean":"2024-04-24T08:18:51.523831+0000","last_became_active":"2024-04-24T08:18:46.440985+0000","last_became_peered":"2024-04-24T08:18:46.440985+0000","last_unstale":"2024-04-24T08:18:51.523831+0000","last_undegraded":"2024-04-24T08:18:51.523831+0000","last_fullsized":"2024-04-24T08:18:51.523831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:45.201518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284200+0000","last_change":"2024-04-24T08:18:47.320928+0000","last_active":"2024-04-24T08:18:48.284200+0000","last_peered":"2024-04-24T08:18:48.284200+0000","last_clean":"2024-04-24T08:18:48.284200+0000","last_became_active":"2024-04-24T08:18:47.320515+0000","last_became_peered":"2024-04-24T08:18:47.320515+0000","last_unstale":"2024-04-24T08:18:48.284200+0000","last_undegraded":"2024-04-24T08:18:48.284200+0000","last_fullsized":"2024-04-24T08:18:48.284200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:55:42.899438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.508954+0000","last_change":"2024-04-24T08:18:46.437677+0000","last_active":"2024-04-24T08:18:50.508954+0000","last_peered":"2024-04-24T08:18:50.508954+0000","last_clean":"2024-04-24T08:18:50.508954+0000","last_became_active":"2024-04-24T08:18:46.437364+0000","last_became_peered":"2024-04-24T08:18:46.437364+0000","last_unstale":"2024-04-24T08:18:50.508954+0000","last_undegraded":"2024-04-24T08:18:50.508954+0000","last_fullsized":"2024-04-24T08:18:50.508954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:25:55.408887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221573+0000","last_change":"2024-04-24T08:18:47.298396+0000","last_active":"2024-04-24T08:18:49.221573+0000","last_peered":"2024-04-24T08:18:49.221573+0000","last_clean":"2024-04-24T08:18:49.221573+0000","last_became_active":"2024-04-24T08:18:47.298021+0000","last_became_peered":"2024-04-24T08:18:47.298021+0000","last_unstale":"2024-04-24T08:18:49.221573+0000","last_undegraded":"2024-04-24T08:18:49.221573+0000","last_fullsized":"2024-04-24T08:18:49.221573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:24.015897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:52.533062+0000","last_change":"2024-04-24T08:18:46.436612+0000","last_active":"2024-04-24T08:18:52.533062+0000","last_peered":"2024-04-24T08:18:52.533062+0000","last_clean":"2024-04-24T08:18:52.533062+0000","last_became_active":"2024-04-24T08:18:46.436211+0000","last_became_peered":"2024-04-24T08:18:46.436211+0000","last_unstale":"2024-04-24T08:18:52.533062+0000","last_undegraded":"2024-04-24T08:18:52.533062+0000","last_fullsized":"2024-04-24T08:18:52.533062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:49:05.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221294+0000","last_change":"2024-04-24T08:18:47.290849+0000","last_active":"2024-04-24T08:18:49.221294+0000","last_peered":"2024-04-24T08:18:49.221294+0000","last_clean":"2024-04-24T08:18:49.221294+0000","last_became_active":"2024-04-24T08:18:47.290514+0000","last_became_peered":"2024-04-24T08:18:47.290514+0000","last_unstale":"2024-04-24T08:18:49.221294+0000","last_undegraded":"2024-04-24T08:18:49.221294+0000","last_fullsized":"2024-04-24T08:18:49.221294+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:39:12.527477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.524099+0000","last_change":"2024-04-24T08:18:46.451815+0000","last_active":"2024-04-24T08:18:51.524099+0000","last_peered":"2024-04-24T08:18:51.524099+0000","last_clean":"2024-04-24T08:18:51.524099+0000","last_became_active":"2024-04-24T08:18:46.451478+0000","last_became_peered":"2024-04-24T08:18:46.451478+0000","last_unstale":"2024-04-24T08:18:51.524099+0000","last_undegraded":"2024-04-24T08:18:51.524099+0000","last_fullsized":"2024-04-24T08:18:51.524099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:54:22.907143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892342+0000","last_change":"2024-04-24T08:18:47.295181+0000","last_active":"2024-04-24T08:18:48.892342+0000","last_peered":"2024-04-24T08:18:48.892342+0000","last_clean":"2024-04-24T08:18:48.892342+0000","last_became_active":"2024-04-24T08:18:47.294842+0000","last_became_peered":"2024-04-24T08:18:47.294842+0000","last_unstale":"2024-04-24T08:18:48.892342+0000","last_undegraded":"2024-04-24T08:18:48.892342+0000","last_fullsized":"2024-04-24T08:18:48.892342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:37.058025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.468046+0000","last_change":"2024-04-24T08:18:46.437967+0000","last_active":"2024-04-24T08:18:48.468046+0000","last_peered":"2024-04-24T08:18:48.468046+0000","last_clean":"2024-04-24T08:18:48.468046+0000","last_became_active":"2024-04-24T08:18:46.437617+0000","last_became_peered":"2024-04-24T08:18:46.437617+0000","last_unstale":"2024-04-24T08:18:48.468046+0000","last_undegraded":"2024-04-24T08:18:48.468046+0000","last_fullsized":"2024-04-24T08:18:48.468046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:07.303827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892483+0000","last_change":"2024-04-24T08:18:47.296975+0000","last_active":"2024-04-24T08:18:48.892483+0000","last_peered":"2024-04-24T08:18:48.892483+0000","last_clean":"2024-04-24T08:18:48.892483+0000","last_became_active":"2024-04-24T08:18:47.296691+0000","last_became_peered":"2024-04-24T08:18:47.296691+0000","last_unstale":"2024-04-24T08:18:48.892483+0000","last_undegraded":"2024-04-24T08:18:48.892483+0000","last_fullsized":"2024-04-24T08:18:48.892483+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:20:04.537185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.463919+0000","last_change":"2024-04-24T08:18:46.453216+0000","last_active":"2024-04-24T08:18:48.463919+0000","last_peered":"2024-04-24T08:18:48.463919+0000","last_clean":"2024-04-24T08:18:48.463919+0000","last_became_active":"2024-04-24T08:18:46.452814+0000","last_became_peered":"2024-04-24T08:18:46.452814+0000","last_unstale":"2024-04-24T08:18:48.463919+0000","last_undegraded":"2024-04-24T08:18:48.463919+0000","last_fullsized":"2024-04-24T08:18:48.463919+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:18.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282761+0000","last_change":"2024-04-24T08:18:47.316791+0000","last_active":"2024-04-24T08:18:48.282761+0000","last_peered":"2024-04-24T08:18:48.282761+0000","last_clean":"2024-04-24T08:18:48.282761+0000","last_became_active":"2024-04-24T08:18:47.316486+0000","last_became_peered":"2024-04-24T08:18:47.316486+0000","last_unstale":"2024-04-24T08:18:48.282761+0000","last_undegraded":"2024-04-24T08:18:48.282761+0000","last_fullsized":"2024-04-24T08:18:48.282761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:36:37.190732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221494+0000","last_change":"2024-04-24T08:18:46.432831+0000","last_active":"2024-04-24T08:18:49.221494+0000","last_peered":"2024-04-24T08:18:49.221494+0000","last_clean":"2024-04-24T08:18:49.221494+0000","last_became_active":"2024-04-24T08:18:46.432483+0000","last_became_peered":"2024-04-24T08:18:46.432483+0000","last_unstale":"2024-04-24T08:18:49.221494+0000","last_undegraded":"2024-04-24T08:18:49.221494+0000","last_fullsized":"2024-04-24T08:18:49.221494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:43:24.802099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.282548+0000","last_change":"2024-04-24T08:18:47.309408+0000","last_active":"2024-04-24T08:18:48.282548+0000","last_peered":"2024-04-24T08:18:48.282548+0000","last_clean":"2024-04-24T08:18:48.282548+0000","last_became_active":"2024-04-24T08:18:47.309083+0000","last_became_peered":"2024-04-24T08:18:47.309083+0000","last_unstale":"2024-04-24T08:18:48.282548+0000","last_undegraded":"2024-04-24T08:18:48.282548+0000","last_fullsized":"2024-04-24T08:18:48.282548+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:15:44.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892623+0000","last_change":"2024-04-24T08:18:47.291060+0000","last_active":"2024-04-24T08:18:48.892623+0000","last_peered":"2024-04-24T08:18:48.892623+0000","last_clean":"2024-04-24T08:18:48.892623+0000","last_became_active":"2024-04-24T08:18:47.290635+0000","last_became_peered":"2024-04-24T08:18:47.290635+0000","last_unstale":"2024-04-24T08:18:48.892623+0000","last_undegraded":"2024-04-24T08:18:48.892623+0000","last_fullsized":"2024-04-24T08:18:48.892623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:20.372853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:53.533980+0000","last_change":"2024-04-24T08:18:46.449850+0000","last_active":"2024-04-24T08:18:53.533980+0000","last_peered":"2024-04-24T08:18:53.533980+0000","last_clean":"2024-04-24T08:18:53.533980+0000","last_became_active":"2024-04-24T08:18:46.448486+0000","last_became_peered":"2024-04-24T08:18:46.448486+0000","last_unstale":"2024-04-24T08:18:53.533980+0000","last_undegraded":"2024-04-24T08:18:53.533980+0000","last_fullsized":"2024-04-24T08:18:53.533980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:54.611867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.470613+0000","last_change":"2024-04-24T08:18:47.328051+0000","last_active":"2024-04-24T08:18:48.470613+0000","last_peered":"2024-04-24T08:18:48.470613+0000","last_clean":"2024-04-24T08:18:48.470613+0000","last_became_active":"2024-04-24T08:18:47.327474+0000","last_became_peered":"2024-04-24T08:18:47.327474+0000","last_unstale":"2024-04-24T08:18:48.470613+0000","last_undegraded":"2024-04-24T08:18:48.470613+0000","last_fullsized":"2024-04-24T08:18:48.470613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:07.577741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:52.524284+0000","last_change":"2024-04-24T08:18:46.444375+0000","last_active":"2024-04-24T08:18:52.524284+0000","last_peered":"2024-04-24T08:18:52.524284+0000","last_clean":"2024-04-24T08:18:52.524284+0000","last_became_active":"2024-04-24T08:18:46.443963+0000","last_became_peered":"2024-04-24T08:18:46.443963+0000","last_unstale":"2024-04-24T08:18:52.524284+0000","last_undegraded":"2024-04-24T08:18:52.524284+0000","last_fullsized":"2024-04-24T08:18:52.524284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:22.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221019+0000","last_change":"2024-04-24T08:17:29.632219+0000","last_active":"2024-04-24T08:18:49.221019+0000","last_peered":"2024-04-24T08:18:49.221019+0000","last_clean":"2024-04-24T08:18:49.221019+0000","last_became_active":"2024-04-24T08:17:29.631841+0000","last_became_peered":"2024-04-24T08:17:29.631841+0000","last_unstale":"2024-04-24T08:18:49.221019+0000","last_undegraded":"2024-04-24T08:18:49.221019+0000","last_fullsized":"2024-04-24T08:18:49.221019+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_clean_scrub_stamp":"2024-04-24T08:13:41.228801+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:02:55.415959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.283803+0000","last_change":"2024-04-24T08:18:47.301876+0000","last_active":"2024-04-24T08:18:48.283803+0000","last_peered":"2024-04-24T08:18:48.283803+0000","last_clean":"2024-04-24T08:18:48.283803+0000","last_became_active":"2024-04-24T08:18:47.301545+0000","last_became_peered":"2024-04-24T08:18:47.301545+0000","last_unstale":"2024-04-24T08:18:48.283803+0000","last_undegraded":"2024-04-24T08:18:48.283803+0000","last_fullsized":"2024-04-24T08:18:48.283803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:34.650014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.520257+0000","last_change":"2024-04-24T08:18:46.437915+0000","last_active":"2024-04-24T08:18:51.520257+0000","last_peered":"2024-04-24T08:18:51.520257+0000","last_clean":"2024-04-24T08:18:51.520257+0000","last_became_active":"2024-04-24T08:18:46.437567+0000","last_became_peered":"2024-04-24T08:18:46.437567+0000","last_unstale":"2024-04-24T08:18:51.520257+0000","last_undegraded":"2024-04-24T08:18:51.520257+0000","last_fullsized":"2024-04-24T08:18:51.520257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:36.801535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.468006+0000","last_change":"2024-04-24T08:18:46.451834+0000","last_active":"2024-04-24T08:18:48.468006+0000","last_peered":"2024-04-24T08:18:48.468006+0000","last_clean":"2024-04-24T08:18:48.468006+0000","last_became_active":"2024-04-24T08:18:46.451505+0000","last_became_peered":"2024-04-24T08:18:46.451505+0000","last_unstale":"2024-04-24T08:18:48.468006+0000","last_undegraded":"2024-04-24T08:18:48.468006+0000","last_fullsized":"2024-04-24T08:18:48.468006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:26.468176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.285077+0000","last_change":"2024-04-24T08:18:47.310490+0000","last_active":"2024-04-24T08:18:48.285077+0000","last_peered":"2024-04-24T08:18:48.285077+0000","last_clean":"2024-04-24T08:18:48.285077+0000","last_became_active":"2024-04-24T08:18:47.309615+0000","last_became_peered":"2024-04-24T08:18:47.309615+0000","last_unstale":"2024-04-24T08:18:48.285077+0000","last_undegraded":"2024-04-24T08:18:48.285077+0000","last_fullsized":"2024-04-24T08:18:48.285077+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:22:55.063485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'6","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:54.460400+0000","last_change":"2024-04-24T08:18:46.445274+0000","last_active":"2024-04-24T08:18:54.460400+0000","last_peered":"2024-04-24T08:18:54.460400+0000","last_clean":"2024-04-24T08:18:54.460400+0000","last_became_active":"2024-04-24T08:18:46.444949+0000","last_became_peered":"2024-04-24T08:18:46.444949+0000","last_unstale":"2024-04-24T08:18:54.460400+0000","last_undegraded":"2024-04-24T08:18:54.460400+0000","last_fullsized":"2024-04-24T08:18:54.460400+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:59:59.062657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.471098+0000","last_change":"2024-04-24T08:18:47.319022+0000","last_active":"2024-04-24T08:18:48.471098+0000","last_peered":"2024-04-24T08:18:48.471098+0000","last_clean":"2024-04-24T08:18:48.471098+0000","last_became_active":"2024-04-24T08:18:47.318167+0000","last_became_peered":"2024-04-24T08:18:47.318167+0000","last_unstale":"2024-04-24T08:18:48.471098+0000","last_undegraded":"2024-04-24T08:18:48.471098+0000","last_fullsized":"2024-04-24T08:18:48.471098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:18.980856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.507488+0000","last_change":"2024-04-24T08:18:46.433018+0000","last_active":"2024-04-24T08:18:50.507488+0000","last_peered":"2024-04-24T08:18:50.507488+0000","last_clean":"2024-04-24T08:18:50.507488+0000","last_became_active":"2024-04-24T08:18:46.432681+0000","last_became_peered":"2024-04-24T08:18:46.432681+0000","last_unstale":"2024-04-24T08:18:50.507488+0000","last_undegraded":"2024-04-24T08:18:50.507488+0000","last_fullsized":"2024-04-24T08:18:50.507488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:53.691684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.288121+0000","last_change":"2024-04-24T08:18:47.326055+0000","last_active":"2024-04-24T08:18:48.288121+0000","last_peered":"2024-04-24T08:18:48.288121+0000","last_clean":"2024-04-24T08:18:48.288121+0000","last_became_active":"2024-04-24T08:18:47.324956+0000","last_became_peered":"2024-04-24T08:18:47.324956+0000","last_unstale":"2024-04-24T08:18:48.288121+0000","last_undegraded":"2024-04-24T08:18:48.288121+0000","last_fullsized":"2024-04-24T08:18:48.288121+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:44:37.431264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.508788+0000","last_change":"2024-04-24T08:18:46.450710+0000","last_active":"2024-04-24T08:18:50.508788+0000","last_peered":"2024-04-24T08:18:50.508788+0000","last_clean":"2024-04-24T08:18:50.508788+0000","last_became_active":"2024-04-24T08:18:46.450391+0000","last_became_peered":"2024-04-24T08:18:46.450391+0000","last_unstale":"2024-04-24T08:18:50.508788+0000","last_undegraded":"2024-04-24T08:18:50.508788+0000","last_fullsized":"2024-04-24T08:18:50.508788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:19.905524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.467654+0000","last_change":"2024-04-24T08:18:47.292114+0000","last_active":"2024-04-24T08:18:48.467654+0000","last_peered":"2024-04-24T08:18:48.467654+0000","last_clean":"2024-04-24T08:18:48.467654+0000","last_became_active":"2024-04-24T08:18:47.291787+0000","last_became_peered":"2024-04-24T08:18:47.291787+0000","last_unstale":"2024-04-24T08:18:48.467654+0000","last_undegraded":"2024-04-24T08:18:48.467654+0000","last_fullsized":"2024-04-24T08:18:48.467654+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:37.020368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.509331+0000","last_change":"2024-04-24T08:18:46.446027+0000","last_active":"2024-04-24T08:18:50.509331+0000","last_peered":"2024-04-24T08:18:50.509331+0000","last_clean":"2024-04-24T08:18:50.509331+0000","last_became_active":"2024-04-24T08:18:46.445634+0000","last_became_peered":"2024-04-24T08:18:46.445634+0000","last_unstale":"2024-04-24T08:18:50.509331+0000","last_undegraded":"2024-04-24T08:18:50.509331+0000","last_fullsized":"2024-04-24T08:18:50.509331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:46.208560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892558+0000","last_change":"2024-04-24T08:18:47.307800+0000","last_active":"2024-04-24T08:18:48.892558+0000","last_peered":"2024-04-24T08:18:48.892558+0000","last_clean":"2024-04-24T08:18:48.892558+0000","last_became_active":"2024-04-24T08:18:47.307501+0000","last_became_peered":"2024-04-24T08:18:47.307501+0000","last_unstale":"2024-04-24T08:18:48.892558+0000","last_undegraded":"2024-04-24T08:18:48.892558+0000","last_fullsized":"2024-04-24T08:18:48.892558+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:04:05.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.233142+0000","last_change":"2024-04-24T08:18:46.442811+0000","last_active":"2024-04-24T08:18:50.233142+0000","last_peered":"2024-04-24T08:18:50.233142+0000","last_clean":"2024-04-24T08:18:50.233142+0000","last_became_active":"2024-04-24T08:18:46.442479+0000","last_became_peered":"2024-04-24T08:18:46.442479+0000","last_unstale":"2024-04-24T08:18:50.233142+0000","last_undegraded":"2024-04-24T08:18:50.233142+0000","last_fullsized":"2024-04-24T08:18:50.233142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:24:07.236529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.470069+0000","last_change":"2024-04-24T08:18:47.325046+0000","last_active":"2024-04-24T08:18:48.470069+0000","last_peered":"2024-04-24T08:18:48.470069+0000","last_clean":"2024-04-24T08:18:48.470069+0000","last_became_active":"2024-04-24T08:18:47.324720+0000","last_became_peered":"2024-04-24T08:18:47.324720+0000","last_unstale":"2024-04-24T08:18:48.470069+0000","last_undegraded":"2024-04-24T08:18:48.470069+0000","last_fullsized":"2024-04-24T08:18:48.470069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:11.240388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:50.503814+0000","last_change":"2024-04-24T08:18:46.440572+0000","last_active":"2024-04-24T08:18:50.503814+0000","last_peered":"2024-04-24T08:18:50.503814+0000","last_clean":"2024-04-24T08:18:50.503814+0000","last_became_active":"2024-04-24T08:18:46.440216+0000","last_became_peered":"2024-04-24T08:18:46.440216+0000","last_unstale":"2024-04-24T08:18:50.503814+0000","last_undegraded":"2024-04-24T08:18:50.503814+0000","last_fullsized":"2024-04-24T08:18:50.503814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:51.352874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221375+0000","last_change":"2024-04-24T08:18:47.293465+0000","last_active":"2024-04-24T08:18:49.221375+0000","last_peered":"2024-04-24T08:18:49.221375+0000","last_clean":"2024-04-24T08:18:49.221375+0000","last_became_active":"2024-04-24T08:18:47.293062+0000","last_became_peered":"2024-04-24T08:18:47.293062+0000","last_unstale":"2024-04-24T08:18:49.221375+0000","last_undegraded":"2024-04-24T08:18:49.221375+0000","last_fullsized":"2024-04-24T08:18:49.221375+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:36.029511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.525296+0000","last_change":"2024-04-24T08:18:46.451007+0000","last_active":"2024-04-24T08:18:51.525296+0000","last_peered":"2024-04-24T08:18:51.525296+0000","last_clean":"2024-04-24T08:18:51.525296+0000","last_became_active":"2024-04-24T08:18:46.450676+0000","last_became_peered":"2024-04-24T08:18:46.450676+0000","last_unstale":"2024-04-24T08:18:51.525296+0000","last_undegraded":"2024-04-24T08:18:51.525296+0000","last_fullsized":"2024-04-24T08:18:51.525296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:40:00.329311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221573+0000","last_change":"2024-04-24T08:18:47.293859+0000","last_active":"2024-04-24T08:18:49.221573+0000","last_peered":"2024-04-24T08:18:49.221573+0000","last_clean":"2024-04-24T08:18:49.221573+0000","last_became_active":"2024-04-24T08:18:47.293509+0000","last_became_peered":"2024-04-24T08:18:47.293509+0000","last_unstale":"2024-04-24T08:18:49.221573+0000","last_undegraded":"2024-04-24T08:18:49.221573+0000","last_fullsized":"2024-04-24T08:18:49.221573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:41.224037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.018673+0000","last_change":"2024-04-24T08:18:46.436661+0000","last_active":"2024-04-24T08:18:49.018673+0000","last_peered":"2024-04-24T08:18:49.018673+0000","last_clean":"2024-04-24T08:18:49.018673+0000","last_became_active":"2024-04-24T08:18:46.436353+0000","last_became_peered":"2024-04-24T08:18:46.436353+0000","last_unstale":"2024-04-24T08:18:49.018673+0000","last_undegraded":"2024-04-24T08:18:49.018673+0000","last_fullsized":"2024-04-24T08:18:49.018673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:57:25.559706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:51.088384+0000","last_change":"2024-04-24T08:18:47.314372+0000","last_active":"2024-04-24T08:18:51.088384+0000","last_peered":"2024-04-24T08:18:51.088384+0000","last_clean":"2024-04-24T08:18:51.088384+0000","last_became_active":"2024-04-24T08:18:47.313979+0000","last_became_peered":"2024-04-24T08:18:47.313979+0000","last_unstale":"2024-04-24T08:18:51.088384+0000","last_undegraded":"2024-04-24T08:18:51.088384+0000","last_fullsized":"2024-04-24T08:18:51.088384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:05.364955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284824+0000","last_change":"2024-04-24T08:18:46.432499+0000","last_active":"2024-04-24T08:18:48.284824+0000","last_peered":"2024-04-24T08:18:48.284824+0000","last_clean":"2024-04-24T08:18:48.284824+0000","last_became_active":"2024-04-24T08:18:46.432085+0000","last_became_peered":"2024-04-24T08:18:46.432085+0000","last_unstale":"2024-04-24T08:18:48.284824+0000","last_undegraded":"2024-04-24T08:18:48.284824+0000","last_fullsized":"2024-04-24T08:18:48.284824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:03.794091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:49.221585+0000","last_change":"2024-04-24T08:18:47.291193+0000","last_active":"2024-04-24T08:18:49.221585+0000","last_peered":"2024-04-24T08:18:49.221585+0000","last_clean":"2024-04-24T08:18:49.221585+0000","last_became_active":"2024-04-24T08:18:47.289903+0000","last_became_peered":"2024-04-24T08:18:47.289903+0000","last_unstale":"2024-04-24T08:18:49.221585+0000","last_undegraded":"2024-04-24T08:18:49.221585+0000","last_fullsized":"2024-04-24T08:18:49.221585+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:35:18.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892349+0000","last_change":"2024-04-24T08:18:47.292628+0000","last_active":"2024-04-24T08:18:48.892349+0000","last_peered":"2024-04-24T08:18:48.892349+0000","last_clean":"2024-04-24T08:18:48.892349+0000","last_became_active":"2024-04-24T08:18:47.292060+0000","last_became_peered":"2024-04-24T08:18:47.292060+0000","last_unstale":"2024-04-24T08:18:48.892349+0000","last_undegraded":"2024-04-24T08:18:48.892349+0000","last_fullsized":"2024-04-24T08:18:48.892349+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:53.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892627+0000","last_change":"2024-04-24T08:18:47.297121+0000","last_active":"2024-04-24T08:18:48.892627+0000","last_peered":"2024-04-24T08:18:48.892627+0000","last_clean":"2024-04-24T08:18:48.892627+0000","last_became_active":"2024-04-24T08:18:47.296787+0000","last_became_peered":"2024-04-24T08:18:47.296787+0000","last_unstale":"2024-04-24T08:18:48.892627+0000","last_undegraded":"2024-04-24T08:18:48.892627+0000","last_fullsized":"2024-04-24T08:18:48.892627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:44:10.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892995+0000","last_change":"2024-04-24T08:18:47.320669+0000","last_active":"2024-04-24T08:18:48.892995+0000","last_peered":"2024-04-24T08:18:48.892995+0000","last_clean":"2024-04-24T08:18:48.892995+0000","last_became_active":"2024-04-24T08:18:47.320287+0000","last_became_peered":"2024-04-24T08:18:47.320287+0000","last_unstale":"2024-04-24T08:18:48.892995+0000","last_undegraded":"2024-04-24T08:18:48.892995+0000","last_fullsized":"2024-04-24T08:18:48.892995+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:39:03.275189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.287058+0000","last_change":"2024-04-24T08:18:47.320947+0000","last_active":"2024-04-24T08:18:48.287058+0000","last_peered":"2024-04-24T08:18:48.287058+0000","last_clean":"2024-04-24T08:18:48.287058+0000","last_became_active":"2024-04-24T08:18:47.314939+0000","last_became_peered":"2024-04-24T08:18:47.314939+0000","last_unstale":"2024-04-24T08:18:48.287058+0000","last_undegraded":"2024-04-24T08:18:48.287058+0000","last_fullsized":"2024-04-24T08:18:48.287058+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:40.183172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284657+0000","last_change":"2024-04-24T08:18:47.316068+0000","last_active":"2024-04-24T08:18:48.284657+0000","last_peered":"2024-04-24T08:18:48.284657+0000","last_clean":"2024-04-24T08:18:48.284657+0000","last_became_active":"2024-04-24T08:18:47.315647+0000","last_became_peered":"2024-04-24T08:18:47.315647+0000","last_unstale":"2024-04-24T08:18:48.284657+0000","last_undegraded":"2024-04-24T08:18:48.284657+0000","last_fullsized":"2024-04-24T08:18:48.284657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:33.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892833+0000","last_change":"2024-04-24T08:18:47.294966+0000","last_active":"2024-04-24T08:18:48.892833+0000","last_peered":"2024-04-24T08:18:48.892833+0000","last_clean":"2024-04-24T08:18:48.892833+0000","last_became_active":"2024-04-24T08:18:47.294672+0000","last_became_peered":"2024-04-24T08:18:47.294672+0000","last_unstale":"2024-04-24T08:18:48.892833+0000","last_undegraded":"2024-04-24T08:18:48.892833+0000","last_fullsized":"2024-04-24T08:18:48.892833+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:37.359114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.893830+0000","last_change":"2024-04-24T08:18:47.294548+0000","last_active":"2024-04-24T08:18:48.893830+0000","last_peered":"2024-04-24T08:18:48.893830+0000","last_clean":"2024-04-24T08:18:48.893830+0000","last_became_active":"2024-04-24T08:18:47.293589+0000","last_became_peered":"2024-04-24T08:18:47.293589+0000","last_unstale":"2024-04-24T08:18:48.893830+0000","last_undegraded":"2024-04-24T08:18:48.893830+0000","last_fullsized":"2024-04-24T08:18:48.893830+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:34:56.208453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.471008+0000","last_change":"2024-04-24T08:18:47.313156+0000","last_active":"2024-04-24T08:18:48.471008+0000","last_peered":"2024-04-24T08:18:48.471008+0000","last_clean":"2024-04-24T08:18:48.471008+0000","last_became_active":"2024-04-24T08:18:47.312829+0000","last_became_peered":"2024-04-24T08:18:47.312829+0000","last_unstale":"2024-04-24T08:18:48.471008+0000","last_undegraded":"2024-04-24T08:18:48.471008+0000","last_fullsized":"2024-04-24T08:18:48.471008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:28:36.638514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892457+0000","last_change":"2024-04-24T08:18:47.320022+0000","last_active":"2024-04-24T08:18:48.892457+0000","last_peered":"2024-04-24T08:18:48.892457+0000","last_clean":"2024-04-24T08:18:48.892457+0000","last_became_active":"2024-04-24T08:18:47.319707+0000","last_became_peered":"2024-04-24T08:18:47.319707+0000","last_unstale":"2024-04-24T08:18:48.892457+0000","last_undegraded":"2024-04-24T08:18:48.892457+0000","last_fullsized":"2024-04-24T08:18:48.892457+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:51.330874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.469845+0000","last_change":"2024-04-24T08:18:47.323187+0000","last_active":"2024-04-24T08:18:48.469845+0000","last_peered":"2024-04-24T08:18:48.469845+0000","last_clean":"2024-04-24T08:18:48.469845+0000","last_became_active":"2024-04-24T08:18:47.320258+0000","last_became_peered":"2024-04-24T08:18:47.320258+0000","last_unstale":"2024-04-24T08:18:48.469845+0000","last_undegraded":"2024-04-24T08:18:48.469845+0000","last_fullsized":"2024-04-24T08:18:48.469845+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:42:32.770950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892073+0000","last_change":"2024-04-24T08:18:47.321926+0000","last_active":"2024-04-24T08:18:48.892073+0000","last_peered":"2024-04-24T08:18:48.892073+0000","last_clean":"2024-04-24T08:18:48.892073+0000","last_became_active":"2024-04-24T08:18:47.321584+0000","last_became_peered":"2024-04-24T08:18:47.321584+0000","last_unstale":"2024-04-24T08:18:48.892073+0000","last_undegraded":"2024-04-24T08:18:48.892073+0000","last_fullsized":"2024-04-24T08:18:48.892073+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:24.357157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.284664+0000","last_change":"2024-04-24T08:18:47.313310+0000","last_active":"2024-04-24T08:18:48.284664+0000","last_peered":"2024-04-24T08:18:48.284664+0000","last_clean":"2024-04-24T08:18:48.284664+0000","last_became_active":"2024-04-24T08:18:47.312968+0000","last_became_peered":"2024-04-24T08:18:47.312968+0000","last_unstale":"2024-04-24T08:18:48.284664+0000","last_undegraded":"2024-04-24T08:18:48.284664+0000","last_fullsized":"2024-04-24T08:18:48.284664+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:08.967577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:18:48.892697+0000","last_change":"2024-04-24T08:18:47.296725+0000","last_active":"2024-04-24T08:18:48.892697+0000","last_peered":"2024-04-24T08:18:48.892697+0000","last_clean":"2024-04-24T08:18:48.892697+0000","last_became_active":"2024-04-24T08:18:47.296406+0000","last_became_peered":"2024-04-24T08:18:47.296406+0000","last_unstale":"2024-04-24T08:18:48.892697+0000","last_undegraded":"2024-04-24T08:18:48.892697+0000","last_fullsized":"2024-04-24T08:18:48.892697+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:04:59.487399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":42194,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":37,"num_read_kb":63,"num_write":142,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":126582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285582,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":452438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547219,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1082112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710746,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":452674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972384,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":452594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266724,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1044524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430250,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":452063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":490049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":453334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":491790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":452374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":452500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691888,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36868,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708284,"statfs":{"total":95995035648,"available":95957282816,"internally_reserved":0,"allocated":1568768,"data_stored":1044091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":38193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":39934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":39771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:18:56.934 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:18:56.935 INFO:tasks.cephadm:Setting up client nodes... 2024-04-24T08:18:56.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='client.14577 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3093784944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-04-24T08:18:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-24T08:18:56.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:56 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-24T08:18:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:18:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:18:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: pgmap v318: 97 pgs: 97 active+clean; 582 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-24T08:18:57.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:57 smithi177 ceph-mon[33046]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: pgmap v318: 97 pgs: 97 active+clean; 582 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-24T08:18:57.850 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:57 smithi146 ceph-mon[27805]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: pgmap v318: 97 pgs: 97 active+clean; 582 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-24T08:18:57.862 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-04-24T08:18:57.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-04-24T08:18:57.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-24T08:18:57.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-24T08:18:57.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:57 smithi153 ceph-mon[32746]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T08:18:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:58 smithi146 ceph-mon[27805]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:18:58.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:58 smithi146 ceph-mon[27805]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T08:18:58.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:58 smithi153 ceph-mon[32746]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:18:58.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:58 smithi153 ceph-mon[32746]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T08:18:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:58 smithi177 ceph-mon[33046]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:18:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:58 smithi177 ceph-mon[33046]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T08:18:59.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 -- 172.21.15.146:0/3718008927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f904810ad30 msgr2=0x7f90481115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:59.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 --2- 172.21.15.146:0/3718008927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f904810ad30 0x7f90481115e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9040009fc0 tx=0x7f904000b330 comp rx=0 tx=0).stop 2024-04-24T08:18:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 -- 172.21.15.146:0/3718008927 shutdown_connections 2024-04-24T08:18:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 --2- 172.21.15.146:0/3718008927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f904810ad30 0x7f90481115e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 --2- 172.21.15.146:0/3718008927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9048109190 0x7f90481095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 --2- 172.21.15.146:0/3718008927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f904810a390 0x7f904810a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 -- 172.21.15.146:0/3718008927 >> 172.21.15.146:0/3718008927 conn(0x7f9048078160 msgr2=0x7f904807a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 -- 172.21.15.146:0/3718008927 shutdown_connections 2024-04-24T08:18:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.680+0000 7f9050200700 1 -- 172.21.15.146:0/3718008927 wait complete. 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.681+0000 7f9050200700 1 Processor -- start 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.681+0000 7f9050200700 1 -- start start 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f9050200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9048109190 0x7f904819f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f9050200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904810a390 0x7f904819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f9050200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f904810ad30 0x7f90481a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f9050200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9048114dc0 con 0x7f904810ad30 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f9050200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9048114c40 con 0x7f9048109190 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f9050200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9048114f40 con 0x7f904810a390 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f904d800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9048109190 0x7f904819f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f904d800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9048109190 0x7f904819f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54982/0 (socket says 172.21.15.146:54982) 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f904d800700 1 -- 172.21.15.146:0/2501823173 learned_addr learned my addr 172.21.15.146:0/2501823173 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f904ce00700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904810a390 0x7f904819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.682+0000 7f904e200700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f904810ad30 0x7f90481a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f904d800700 1 -- 172.21.15.146:0/2501823173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904810a390 msgr2=0x7f904819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f904d800700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904810a390 0x7f904819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f904d800700 1 -- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f904810ad30 msgr2=0x7f90481a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f904d800700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f904810ad30 0x7f90481a6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f904d800700 1 -- 172.21.15.146:0/2501823173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90481a6ee0 con 0x7f9048109190 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f904d800700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9048109190 0x7f904819f280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f903c00e970 tx=0x7f903c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f9046000700 1 -- 172.21.15.146:0/2501823173 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c004220 con 0x7f9048109190 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f9046000700 1 -- 172.21.15.146:0/2501823173 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f903c0043c0 con 0x7f9048109190 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90481a71d0 con 0x7f9048109190 2024-04-24T08:18:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f9046000700 1 -- 172.21.15.146:0/2501823173 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c003df0 con 0x7f9048109190 2024-04-24T08:18:59.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.683+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90481a7760 con 0x7f9048109190 2024-04-24T08:18:59.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.685+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9048051030 con 0x7f9048109190 2024-04-24T08:18:59.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.686+0000 7f9046000700 1 -- 172.21.15.146:0/2501823173 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f903c00c7e0 con 0x7f9048109190 2024-04-24T08:18:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.688+0000 7f9046000700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f90300a3e00 0x7f90300a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.688+0000 7f9046000700 1 -- 172.21.15.146:0/2501823173 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f903c0d06f0 con 0x7f9048109190 2024-04-24T08:18:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.688+0000 7f904ce00700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f90300a3e00 0x7f90300a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.689+0000 7f904ce00700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f90300a3e00 0x7f90300a62c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9034009750 tx=0x7f9034009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:59.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:18:59.692+0000 7f9046000700 1 -- 172.21.15.146:0/2501823173 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f903c09cf30 con 0x7f9048109190 2024-04-24T08:18:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:18:59 smithi153 ceph-mon[32746]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 9.1 KiB/s wr, 21 op/s 2024-04-24T08:18:59.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:18:59 smithi146 ceph-mon[27805]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 9.1 KiB/s wr, 21 op/s 2024-04-24T08:19:00.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.006+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f904802dfe0 con 0x7f9048109190 2024-04-24T08:19:00.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.014+0000 7f9046000700 1 -- 172.21.15.146:0/2501823173 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f903c09c830 con 0x7f9048109190 2024-04-24T08:19:00.016 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-04-24T08:19:00.016 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQB0wChmKl2NABAAh4Mr5hRfzmxmQrhWfGsxRQ== 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f90300a3e00 msgr2=0x7f90300a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f90300a3e00 0x7f90300a62c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9034009750 tx=0x7f9034009310 comp rx=0 tx=0).stop 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9048109190 msgr2=0x7f904819f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9048109190 0x7f904819f280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f903c00e970 tx=0x7f903c00ec80 comp rx=0 tx=0).stop 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 shutdown_connections 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f90300a3e00 0x7f90300a62c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f904810ad30 0x7f90481a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9048109190 0x7f904819f280 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 --2- 172.21.15.146:0/2501823173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904810a390 0x7f904819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 >> 172.21.15.146:0/2501823173 conn(0x7f9048078160 msgr2=0x7f904810d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:00.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 shutdown_connections 2024-04-24T08:19:00.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:00.017+0000 7f9050200700 1 -- 172.21.15.146:0/2501823173 wait complete. 2024-04-24T08:19:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:18:59 smithi177 ceph-mon[33046]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 9.1 KiB/s wr, 21 op/s 2024-04-24T08:19:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:00 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2501823173' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:19:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:00 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:19:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:00 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T08:19:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:00 smithi146 ceph-mon[27805]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-24T08:19:00.895 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:19:00.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-24T08:19:00.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T08:19:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:00 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2501823173' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:19:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:00 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:19:00.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:00 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T08:19:00.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:00 smithi153 ceph-mon[32746]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-24T08:19:01.029 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T08:19:01.029 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-24T08:19:01.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-24T08:19:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:00 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2501823173' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:19:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:00 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:19:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:00 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T08:19:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:00 smithi177 ceph-mon[33046]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-24T08:19:01.523 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.770+0000 7f03f1600700 1 -- 172.21.15.146:0/4175956860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 msgr2=0x7f03ec06b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.770+0000 7f03f1600700 1 --2- 172.21.15.146:0/4175956860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec06b1a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f03d80098f0 tx=0x7f03d8009c00 comp rx=0 tx=0).stop 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.771+0000 7f03f1600700 1 -- 172.21.15.146:0/4175956860 shutdown_connections 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.771+0000 7f03f1600700 1 --2- 172.21.15.146:0/4175956860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03ec105530 0x7f03ec1059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.771+0000 7f03f1600700 1 --2- 172.21.15.146:0/4175956860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03ec107a20 0x7f03ec06a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.771+0000 7f03f1600700 1 --2- 172.21.15.146:0/4175956860 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec06b1a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.771+0000 7f03f1600700 1 -- 172.21.15.146:0/4175956860 >> 172.21.15.146:0/4175956860 conn(0x7f03ec0789a0 msgr2=0x7f03ec078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.771+0000 7f03f1600700 1 -- 172.21.15.146:0/4175956860 shutdown_connections 2024-04-24T08:19:03.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.771+0000 7f03f1600700 1 -- 172.21.15.146:0/4175956860 wait complete. 2024-04-24T08:19:03.774 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:03 smithi146 ceph-mon[27805]: pgmap v323: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.5 KiB/s wr, 14 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.772+0000 7f03f1600700 1 Processor -- start 2024-04-24T08:19:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.772+0000 7f03f1600700 1 -- start start 2024-04-24T08:19:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03f1600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec1a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03f1600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03ec105530 0x7f03ec1a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:03.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03f1600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03ec107a20 0x7f03ec19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03f1600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03ec119410 con 0x7f03ec105530 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03f1600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03ec119290 con 0x7f03ec06ad40 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03f1600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03ec119590 con 0x7f03ec107a20 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03ea000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03ec105530 0x7f03ec1a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eaa00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec1a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eaa00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec1a5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48480/0 (socket says 172.21.15.146:48480) 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eaa00700 1 -- 172.21.15.146:0/3544288575 learned_addr learned my addr 172.21.15.146:0/3544288575 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eb400700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03ec107a20 0x7f03ec19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eaa00700 1 -- 172.21.15.146:0/3544288575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03ec107a20 msgr2=0x7f03ec19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eaa00700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03ec107a20 0x7f03ec19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eaa00700 1 -- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03ec105530 msgr2=0x7f03ec1a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.773+0000 7f03eaa00700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03ec105530 0x7f03ec1a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.774+0000 7f03eaa00700 1 -- 172.21.15.146:0/3544288575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ec19ff70 con 0x7f03ec06ad40 2024-04-24T08:19:03.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.774+0000 7f03eaa00700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec1a5610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00e9a0 tx=0x7f03dc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:03.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.774+0000 7f03e3400700 1 -- 172.21.15.146:0/3544288575 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc00ca10 con 0x7f03ec06ad40 2024-04-24T08:19:03.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.774+0000 7f03e3400700 1 -- 172.21.15.146:0/3544288575 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03dc00cbb0 con 0x7f03ec06ad40 2024-04-24T08:19:03.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.774+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ec1a0260 con 0x7f03ec06ad40 2024-04-24T08:19:03.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.774+0000 7f03e3400700 1 -- 172.21.15.146:0/3544288575 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc018680 con 0x7f03ec06ad40 2024-04-24T08:19:03.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.774+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ec1ac470 con 0x7f03ec06ad40 2024-04-24T08:19:03.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.775+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03ec102010 con 0x7f03ec06ad40 2024-04-24T08:19:03.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.777+0000 7f03e3400700 1 -- 172.21.15.146:0/3544288575 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f03dc010430 con 0x7f03ec06ad40 2024-04-24T08:19:03.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.779+0000 7f03e3400700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d00a3d50 0x7f03d00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:03.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.779+0000 7f03e3400700 1 -- 172.21.15.146:0/3544288575 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f03dc0d0720 con 0x7f03ec06ad40 2024-04-24T08:19:03.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.779+0000 7f03ea000700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d00a3d50 0x7f03d00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:03.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.780+0000 7f03ea000700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d00a3d50 0x7f03d00a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009810 tx=0x7f03d8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:03.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:03.783+0000 7f03e3400700 1 -- 172.21.15.146:0/3544288575 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03dc09cf60 con 0x7f03ec06ad40 2024-04-24T08:19:03.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:03 smithi177 ceph-mon[33046]: pgmap v323: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.5 KiB/s wr, 14 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:03.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:03 smithi153 ceph-mon[32746]: pgmap v323: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.5 KiB/s wr, 14 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:04.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.069+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f03ec1ac6a0 con 0x7f03ec06ad40 2024-04-24T08:19:04.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.075+0000 7f03e3400700 1 -- 172.21.15.146:0/3544288575 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+265005 (secure 0 0 0) 0x7f03dc09c860 con 0x7f03ec06ad40 2024-04-24T08:19:04.078 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:19:04.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d00a3d50 msgr2=0x7f03d00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:04.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d00a3d50 0x7f03d00a6210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009810 tx=0x7f03d8019040 comp rx=0 tx=0).stop 2024-04-24T08:19:04.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 msgr2=0x7f03ec1a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec1a5610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00e9a0 tx=0x7f03dc00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 shutdown_connections 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03ec107a20 0x7f03ec19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03ec06ad40 0x7f03ec1a5610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d00a3d50 0x7f03d00a6210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 --2- 172.21.15.146:0/3544288575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03ec105530 0x7f03ec1a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 >> 172.21.15.146:0/3544288575 conn(0x7f03ec0789a0 msgr2=0x7f03ec104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 shutdown_connections 2024-04-24T08:19:04.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:04.082+0000 7f03f1600700 1 -- 172.21.15.146:0/3544288575 wait complete. 2024-04-24T08:19:04.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:04 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3544288575' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T08:19:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:04 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3544288575' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T08:19:04.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:04 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3544288575' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T08:19:04.968 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1475219332},{"type":"v1","addr":"172.21.15.146:6801","nonce":1475219332}]},"active_addr":"172.21.15.146:6801/1475219332","active_change":"2024-04-24T08:10:56.306974+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.146:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":393585939}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":4151754658}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":4023745007}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2355630587}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":516279323}]}]} 2024-04-24T08:19:04.973 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-24T08:19:04.974 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-24T08:19:04.974 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T08:19:05.438 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:05.795 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:05 smithi146 ceph-mon[27805]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.1 KiB/s wr, 12 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:05 smithi177 ceph-mon[33046]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.1 KiB/s wr, 12 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:05.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:05 smithi153 ceph-mon[32746]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.1 KiB/s wr, 12 op/s; 14 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:07.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 -- 172.21.15.146:0/3618998938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 msgr2=0x7f5ab810f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:07.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 --2- 172.21.15.146:0/3618998938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 0x7f5ab810f9d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009f90 tx=0x7f5ab000b330 comp rx=0 tx=0).stop 2024-04-24T08:19:07.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 -- 172.21.15.146:0/3618998938 shutdown_connections 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 --2- 172.21.15.146:0/3618998938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 0x7f5ab810f9d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 --2- 172.21.15.146:0/3618998938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab8108b90 0x7f5ab8108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 --2- 172.21.15.146:0/3618998938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab806b1c0 0x7f5ab806b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 -- 172.21.15.146:0/3618998938 >> 172.21.15.146:0/3618998938 conn(0x7f5ab80789a0 msgr2=0x7f5ab8078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 -- 172.21.15.146:0/3618998938 shutdown_connections 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.693+0000 7f5abcc00700 1 -- 172.21.15.146:0/3618998938 wait complete. 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.694+0000 7f5abcc00700 1 Processor -- start 2024-04-24T08:19:07.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5abcc00700 1 -- start start 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5abcc00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab806b1c0 0x7f5ab819b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5abcc00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 0x7f5ab819ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5abcc00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab8108b90 0x7f5ab819fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5abcc00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8112e40 con 0x7f5ab8108b90 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5abcc00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8112cc0 con 0x7f5ab806b1c0 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5abcc00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8112fc0 con 0x7f5ab806bb60 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5ab5e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab806b1c0 0x7f5ab819b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5ab5e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab806b1c0 0x7f5ab819b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48496/0 (socket says 172.21.15.146:48496) 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5ab5e00700 1 -- 172.21.15.146:0/2656208021 learned_addr learned my addr 172.21.15.146:0/2656208021 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:07.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.695+0000 7f5aafe00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 0x7f5ab819ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab6800700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab8108b90 0x7f5ab819fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab5e00700 1 -- 172.21.15.146:0/2656208021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 msgr2=0x7f5ab819ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab5e00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 0x7f5ab819ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab5e00700 1 -- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab8108b90 msgr2=0x7f5ab819fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab5e00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab8108b90 0x7f5ab819fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab5e00700 1 -- 172.21.15.146:0/2656208021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab81a04e0 con 0x7f5ab806b1c0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab6800700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab8108b90 0x7f5ab819fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5ab5e00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab806b1c0 0x7f5ab819b4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800e9f0 tx=0x7f5aa800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5aaea00700 1 -- 172.21.15.146:0/2656208021 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004a30 con 0x7f5ab806b1c0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5aaea00700 1 -- 172.21.15.146:0/2656208021 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa800cca0 con 0x7f5ab806b1c0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.696+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab81a07d0 con 0x7f5ab806b1c0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.697+0000 7f5aaea00700 1 -- 172.21.15.146:0/2656208021 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80056d0 con 0x7f5ab806b1c0 2024-04-24T08:19:07.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.697+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab81a8050 con 0x7f5ab806b1c0 2024-04-24T08:19:07.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.698+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab8104240 con 0x7f5ab806b1c0 2024-04-24T08:19:07.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.699+0000 7f5aaea00700 1 -- 172.21.15.146:0/2656208021 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f5aa800c7c0 con 0x7f5ab806b1c0 2024-04-24T08:19:07.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.701+0000 7f5aaea00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5a9c0a3ef0 0x7f5a9c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.701+0000 7f5aaea00700 1 -- 172.21.15.146:0/2656208021 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f5aa80d0910 con 0x7f5ab806b1c0 2024-04-24T08:19:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.701+0000 7f5aafe00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5a9c0a3ef0 0x7f5a9c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:07.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.702+0000 7f5aafe00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5a9c0a3ef0 0x7f5a9c0a63b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8103e50 tx=0x7f5aa0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:07.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.705+0000 7f5aaea00700 1 -- 172.21.15.146:0/2656208021 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5aa809d150 con 0x7f5ab806b1c0 2024-04-24T08:19:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:07 smithi177 ceph-mon[33046]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 724 B/s wr, 6 op/s; 11 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:07.936 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:07 smithi146 ceph-mon[27805]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 724 B/s wr, 6 op/s; 11 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:07.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.934+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ab80697e0 con 0x7f5ab806b1c0 2024-04-24T08:19:07.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.936+0000 7f5aaea00700 1 -- 172.21.15.146:0/2656208021 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7f5aa809ca50 con 0x7f5ab806b1c0 2024-04-24T08:19:07.939 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:19:07.939 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:19:07.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5a9c0a3ef0 msgr2=0x7f5a9c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5a9c0a3ef0 0x7f5a9c0a63b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8103e50 tx=0x7f5aa0009260 comp rx=0 tx=0).stop 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab806b1c0 msgr2=0x7f5ab819b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab806b1c0 0x7f5ab819b4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800e9f0 tx=0x7f5aa800ed00 comp rx=0 tx=0).stop 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 shutdown_connections 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5a9c0a3ef0 0x7f5a9c0a63b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab8108b90 0x7f5ab819fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5ab806b1c0 0x7f5ab819b4d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 --2- 172.21.15.146:0/2656208021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab806bb60 0x7f5ab819ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 >> 172.21.15.146:0/2656208021 conn(0x7f5ab80789a0 msgr2=0x7f5ab810dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:07.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 shutdown_connections 2024-04-24T08:19:07.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:07.940+0000 7f5abcc00700 1 -- 172.21.15.146:0/2656208021 wait complete. 2024-04-24T08:19:07.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:07 smithi153 ceph-mon[32746]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 724 B/s wr, 6 op/s; 11 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:19:08.602 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:08 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2656208021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:19:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:08 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2656208021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:19:08.828 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-24T08:19:08.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T08:19:08.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:08 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2656208021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:19:09.284 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:09.710 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:09 smithi146 ceph-mon[27805]: pgmap v326: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 10 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:09 smithi177 ceph-mon[33046]: pgmap v326: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 10 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:09.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:09 smithi153 ceph-mon[32746]: pgmap v326: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 10 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 -- 172.21.15.146:0/2122068835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90105830 msgr2=0x7f7b90109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2122068835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90105830 0x7f7b90109880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009990 tx=0x7f7b80009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 -- 172.21.15.146:0/2122068835 shutdown_connections 2024-04-24T08:19:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2122068835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b90109f60 0x7f7b90111b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2122068835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90104e80 0x7f7b90105260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2122068835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90105830 0x7f7b90109880 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 -- 172.21.15.146:0/2122068835 >> 172.21.15.146:0/2122068835 conn(0x7f7b90100710 msgr2=0x7f7b90102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 -- 172.21.15.146:0/2122068835 shutdown_connections 2024-04-24T08:19:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.542+0000 7f7b8f200700 1 -- 172.21.15.146:0/2122068835 wait complete. 2024-04-24T08:19:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 Processor -- start 2024-04-24T08:19:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 -- start start 2024-04-24T08:19:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b90104e80 0x7f7b9019b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90105830 0x7f7b9019b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90109f60 0x7f7b9019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7b90114f70 con 0x7f7b90105830 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7b90114df0 con 0x7f7b90109f60 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b8f200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7b901150f0 con 0x7f7b90104e80 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b85400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90105830 0x7f7b9019b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.544+0000 7f7b85400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90105830 0x7f7b9019b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:53654/0 (socket says 172.21.15.146:53654) 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b85400700 1 -- 172.21.15.146:0/2698489024 learned_addr learned my addr 172.21.15.146:0/2698489024 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b8de00700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b90104e80 0x7f7b9019b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b8e800700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90109f60 0x7f7b9019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b85400700 1 -- 172.21.15.146:0/2698489024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b90104e80 msgr2=0x7f7b9019b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b85400700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b90104e80 0x7f7b9019b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b85400700 1 -- 172.21.15.146:0/2698489024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90109f60 msgr2=0x7f7b9019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b85400700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90109f60 0x7f7b9019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b85400700 1 -- 172.21.15.146:0/2698489024 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b901a03e0 con 0x7f7b90105830 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b8de00700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b90104e80 0x7f7b9019b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.545+0000 7f7b85400700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90105830 0x7f7b9019b960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000c00 tx=0x7f7b80005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.546+0000 7f7b87400700 1 -- 172.21.15.146:0/2698489024 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b80027070 con 0x7f7b90105830 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.546+0000 7f7b87400700 1 -- 172.21.15.146:0/2698489024 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b8000fda0 con 0x7f7b90105830 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.546+0000 7f7b87400700 1 -- 172.21.15.146:0/2698489024 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b800173f0 con 0x7f7b90105830 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.546+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b901a0670 con 0x7f7b90105830 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.546+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b901a7f50 con 0x7f7b90105830 2024-04-24T08:19:11.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.547+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b900697e0 con 0x7f7b90105830 2024-04-24T08:19:11.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.549+0000 7f7b87400700 1 -- 172.21.15.146:0/2698489024 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f7b8002a070 con 0x7f7b90105830 2024-04-24T08:19:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.550+0000 7f7b87400700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7b740a3d50 0x7f7b740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.551+0000 7f7b87400700 1 -- 172.21.15.146:0/2698489024 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7b800d1360 con 0x7f7b90105830 2024-04-24T08:19:11.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.551+0000 7f7b8de00700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7b740a3d50 0x7f7b740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.551+0000 7f7b8de00700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7b740a3d50 0x7f7b740a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7b78011490 tx=0x7f7b78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.555+0000 7f7b87400700 1 -- 172.21.15.146:0/2698489024 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b8005eff0 con 0x7f7b90105830 2024-04-24T08:19:11.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:11 smithi146 ceph-mon[27805]: pgmap v327: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:11.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.782+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b9002dfe0 con 0x7f7b90105830 2024-04-24T08:19:11.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.784+0000 7f7b87400700 1 -- 172.21.15.146:0/2698489024 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7f7b80013070 con 0x7f7b90105830 2024-04-24T08:19:11.786 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:19:11.787 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:19:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.787+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7b740a3d50 msgr2=0x7f7b740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.787+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7b740a3d50 0x7f7b740a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7b78011490 tx=0x7f7b78009260 comp rx=0 tx=0).stop 2024-04-24T08:19:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.787+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90105830 msgr2=0x7f7b9019b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.787+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90105830 0x7f7b9019b960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7b80000c00 tx=0x7f7b80005590 comp rx=0 tx=0).stop 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.787+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 shutdown_connections 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.788+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b90109f60 0x7f7b9019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.788+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b90104e80 0x7f7b9019b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.788+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7b740a3d50 0x7f7b740a6210 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.788+0000 7f7b8f200700 1 --2- 172.21.15.146:0/2698489024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b90105830 0x7f7b9019b960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.788+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 >> 172.21.15.146:0/2698489024 conn(0x7f7b90100710 msgr2=0x7f7b901012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.788+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 shutdown_connections 2024-04-24T08:19:11.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:11.788+0000 7f7b8f200700 1 -- 172.21.15.146:0/2698489024 wait complete. 2024-04-24T08:19:11.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:11 smithi177 ceph-mon[33046]: pgmap v327: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:11.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:11 smithi153 ceph-mon[32746]: pgmap v327: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:12.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-24T08:19:12.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-24T08:19:12.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-24T08:19:12.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-24T08:19:12.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-24T08:19:12.525 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-24T08:19:12.525 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-24T08:19:12.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-24T08:19:12.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-24T08:19:12.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-24T08:19:12.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-24T08:19:12.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-24T08:19:12.696 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:12 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2698489024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:19:12.814 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:12 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2698489024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:19:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:12 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2698489024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:19:13.482 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:13.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:13 smithi177 ceph-mon[33046]: pgmap v328: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:13.842 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:13.843 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:13 smithi146 ceph-mon[27805]: pgmap v328: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:13.853 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:13.858 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:13.859 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:13.866 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:13.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:13 smithi153 ceph-mon[32746]: pgmap v328: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:19:15.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:15 smithi146 ceph-mon[27805]: pgmap v329: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-24T08:19:15.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:15 smithi177 ceph-mon[33046]: pgmap v329: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-24T08:19:15.825 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:15.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:15 smithi153 ceph-mon[32746]: pgmap v329: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-24T08:19:17.730 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:17.932 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:17 smithi146 ceph-mon[27805]: pgmap v330: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-24T08:19:17.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:17 smithi153 ceph-mon[32746]: pgmap v330: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-24T08:19:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:17 smithi177 ceph-mon[33046]: pgmap v330: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-24T08:19:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:19 smithi146 ceph-mon[27805]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-24T08:19:19.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:19 smithi153 ceph-mon[32746]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-24T08:19:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:19 smithi177 ceph-mon[33046]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-24T08:19:21.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 -- 172.21.15.146:0/4059292287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1810e460 msgr2=0x7f2e18116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 --2- 172.21.15.146:0/4059292287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1810e460 0x7f2e18116030 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009fc0 tx=0x7f2e1000b330 comp rx=0 tx=0).stop 2024-04-24T08:19:21.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 -- 172.21.15.146:0/4059292287 shutdown_connections 2024-04-24T08:19:21.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 --2- 172.21.15.146:0/4059292287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1810e460 0x7f2e18116030 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 --2- 172.21.15.146:0/4059292287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18109380 0x7f2e18109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 --2- 172.21.15.146:0/4059292287 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e18109d30 0x7f2e1810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 -- 172.21.15.146:0/4059292287 >> 172.21.15.146:0/4059292287 conn(0x7f2e180762c0 msgr2=0x7f2e18107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 -- 172.21.15.146:0/4059292287 shutdown_connections 2024-04-24T08:19:21.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.045+0000 7f2e1fa00700 1 -- 172.21.15.146:0/4059292287 wait complete. 2024-04-24T08:19:21.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.046+0000 7f2e1fa00700 1 Processor -- start 2024-04-24T08:19:21.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1fa00700 1 -- start start 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1fa00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18109380 0x7f2e181a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1fa00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18109d30 0x7f2e181a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1fa00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1810e460 0x7f2e1819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1fa00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2e181194a0 con 0x7f2e18109d30 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1fa00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e18119320 con 0x7f2e1810e460 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1fa00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2e18119620 con 0x7f2e18109380 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1d000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18109380 0x7f2e181a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e17e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18109d30 0x7f2e181a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1d000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18109380 0x7f2e181a55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57904/0 (socket says 172.21.15.146:57904) 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1d000700 1 -- 172.21.15.146:0/146820146 learned_addr learned my addr 172.21.15.146:0/146820146 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.047+0000 7f2e1da00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1810e460 0x7f2e1819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1d000700 1 -- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1810e460 msgr2=0x7f2e1819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1d000700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1810e460 0x7f2e1819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1d000700 1 -- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18109d30 msgr2=0x7f2e181a5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1d000700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18109d30 0x7f2e181a5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1d000700 1 -- 172.21.15.146:0/146820146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1819ff80 con 0x7f2e18109380 2024-04-24T08:19:21.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1da00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e1810e460 0x7f2e1819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e17e00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e18109d30 0x7f2e181a5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1d000700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18109380 0x7f2e181a55d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00e9a0 tx=0x7f2e0c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c00ca10 con 0x7f2e18109380 2024-04-24T08:19:21.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e0c00cbb0 con 0x7f2e18109380 2024-04-24T08:19:21.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e181a0270 con 0x7f2e18109380 2024-04-24T08:19:21.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.048+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c018680 con 0x7f2e18109380 2024-04-24T08:19:21.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.049+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e181ac550 con 0x7f2e18109380 2024-04-24T08:19:21.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.049+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e180771c0 con 0x7f2e18109380 2024-04-24T08:19:21.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.051+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2e0c010430 con 0x7f2e18109380 2024-04-24T08:19:21.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.053+0000 7f2e15600700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e000a3d50 0x7f2e000a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.053+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f2e0c0d0fc0 con 0x7f2e18109380 2024-04-24T08:19:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.053+0000 7f2e17e00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e000a3d50 0x7f2e000a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.053+0000 7f2e15600700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] conn(0x7f2e000ad910 0x7f2e000afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.053+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 --> [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e0c027e50 con 0x7f2e000ad910 2024-04-24T08:19:21.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.053+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f2e0c0d1370 con 0x7f2e18109380 2024-04-24T08:19:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.054+0000 7f2e17e00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e000a3d50 0x7f2e000a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2e181a0e60 tx=0x7f2e04009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.054+0000 7f2e1da00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] conn(0x7f2e000ad910 0x7f2e000afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.054+0000 7f2e1da00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] conn(0x7f2e000ad910 0x7f2e000afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.055+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== osd.4 v2:172.21.15.153:6808/3654502994 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e0c027e50 con 0x7f2e000ad910 2024-04-24T08:19:21.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.074+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 --> [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e1810a860 con 0x7f2e000ad910 2024-04-24T08:19:21.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.074+0000 7f2e15600700 1 -- 172.21.15.146:0/146820146 <== osd.4 v2:172.21.15.153:6808/3654502994 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2e1810a860 con 0x7f2e000ad910 2024-04-24T08:19:21.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.075+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] conn(0x7f2e000ad910 msgr2=0x7f2e000afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.075+0000 7f2e1fa00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] conn(0x7f2e000ad910 0x7f2e000afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.075+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e000a3d50 msgr2=0x7f2e000a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.075+0000 7f2e1fa00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e000a3d50 0x7f2e000a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2e181a0e60 tx=0x7f2e04009310 comp rx=0 tx=0).stop 2024-04-24T08:19:21.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.075+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18109380 msgr2=0x7f2e181a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.075+0000 7f2e1fa00700 1 --2- 172.21.15.146:0/146820146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e18109380 0x7f2e181a55d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00e9a0 tx=0x7f2e0c00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.075+0000 7f2e1da00700 1 -- 172.21.15.146:0/146820146 reap_dead start 2024-04-24T08:19:21.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.076+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 shutdown_connections 2024-04-24T08:19:21.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.076+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 >> 172.21.15.146:0/146820146 conn(0x7f2e180762c0 msgr2=0x7f2e180767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.076+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 shutdown_connections 2024-04-24T08:19:21.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:21.076+0000 7f2e1fa00700 1 -- 172.21.15.146:0/146820146 wait complete. 2024-04-24T08:19:21.576 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:21 smithi146 ceph-mon[27805]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:19:21.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:21 smithi153 ceph-mon[32746]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:19:22.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:21 smithi177 ceph-mon[33046]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:19:23.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.576+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1409114709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4108a20 msgr2=0x7fa6d4108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.576+0000 7fa6d9a00700 1 --2- 172.21.15.146:0/1409114709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4108a20 0x7fa6d4108e00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009990 tx=0x7fa6c4009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.576+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1409114709 shutdown_connections 2024-04-24T08:19:23.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.576+0000 7fa6d9a00700 1 --2- 172.21.15.146:0/1409114709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d41033e0 0x7fa6d410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.576+0000 7fa6d9a00700 1 --2- 172.21.15.146:0/1409114709 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4108a20 0x7fa6d4108e00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.576+0000 7fa6d9a00700 1 --2- 172.21.15.146:0/1409114709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4102a20 0x7fa6d4102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.576+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1409114709 >> 172.21.15.146:0/1409114709 conn(0x7fa6d40fe570 msgr2=0x7fa6d4100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.577+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1409114709 shutdown_connections 2024-04-24T08:19:23.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.577+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1409114709 wait complete. 2024-04-24T08:19:23.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.578+0000 7fa6d9a00700 1 Processor -- start 2024-04-24T08:19:23.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.578+0000 7fa6d9a00700 1 -- start start 2024-04-24T08:19:23.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d9a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4102a20 0x7fa6d4074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d9a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d41033e0 0x7fa6d4075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d9a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d4108a20 0x7fa6d4075870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d9a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d4112da0 con 0x7fa6d4102a20 2024-04-24T08:19:23.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d9a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d4112c20 con 0x7fa6d4108a20 2024-04-24T08:19:23.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d9a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d4112f20 con 0x7fa6d41033e0 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d3e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4102a20 0x7fa6d4074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d3e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4102a20 0x7fa6d4074df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51810/0 (socket says 172.21.15.146:51810) 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d3e00700 1 -- 172.21.15.146:0/1987880252 learned_addr learned my addr 172.21.15.146:0/1987880252 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d9000700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d4108a20 0x7fa6d4075870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.579+0000 7fa6d3400700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d41033e0 0x7fa6d4075330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d3e00700 1 -- 172.21.15.146:0/1987880252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d41033e0 msgr2=0x7fa6d4075330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d3e00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d41033e0 0x7fa6d4075330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d3e00700 1 -- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d4108a20 msgr2=0x7fa6d4075870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d3e00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d4108a20 0x7fa6d4075870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d3e00700 1 -- 172.21.15.146:0/1987880252 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d41a7720 con 0x7fa6d4102a20 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d3400700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d41033e0 0x7fa6d4075330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d9000700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6d4108a20 0x7fa6d4075870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:23.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.580+0000 7fa6d3e00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4102a20 0x7fa6d4074df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000c00 tx=0x7fa6c400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.582+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4027070 con 0x7fa6d4102a20 2024-04-24T08:19:23.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.582+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c4003d70 con 0x7fa6d4102a20 2024-04-24T08:19:23.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.582+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c400f460 con 0x7fa6d4102a20 2024-04-24T08:19:23.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.583+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d41a79b0 con 0x7fa6d4102a20 2024-04-24T08:19:23.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.583+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d41a7db0 con 0x7fa6d4102a20 2024-04-24T08:19:23.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.584+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa6d4063180 con 0x7fa6d4102a20 2024-04-24T08:19:23.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.586+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa6c402a070 con 0x7fa6d4102a20 2024-04-24T08:19:23.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.587+0000 7fa6d0c00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6b80a3d50 0x7fa6b80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.587+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fa6c40d1ad0 con 0x7fa6d4102a20 2024-04-24T08:19:23.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.587+0000 7fa6d3400700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6b80a3d50 0x7fa6b80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.588+0000 7fa6d0c00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] conn(0x7fa6b80ad910 0x7fa6b80afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.588+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 --> [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6c4003bc0 con 0x7fa6b80ad910 2024-04-24T08:19:23.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.588+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fa6c40d1e80 con 0x7fa6d4102a20 2024-04-24T08:19:23.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.588+0000 7fa6d9000700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] conn(0x7fa6b80ad910 0x7fa6b80afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.588+0000 7fa6d3400700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6b80a3d50 0x7fa6b80a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4107fc0 tx=0x7fa6c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.589+0000 7fa6d9000700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] conn(0x7fa6b80ad910 0x7fa6b80afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.589+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== osd.1 v2:172.21.15.153:6800/1792775739 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa6c4003bc0 con 0x7fa6b80ad910 2024-04-24T08:19:23.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.608+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 --> [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa6d4063650 con 0x7fa6b80ad910 2024-04-24T08:19:23.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.609+0000 7fa6d0c00700 1 -- 172.21.15.146:0/1987880252 <== osd.1 v2:172.21.15.153:6800/1792775739 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa6d4063650 con 0x7fa6b80ad910 2024-04-24T08:19:23.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.609+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] conn(0x7fa6b80ad910 msgr2=0x7fa6b80afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.609+0000 7fa6d9a00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] conn(0x7fa6b80ad910 0x7fa6b80afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.609+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6b80a3d50 msgr2=0x7fa6b80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.609+0000 7fa6d9a00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6b80a3d50 0x7fa6b80a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4107fc0 tx=0x7fa6c0009260 comp rx=0 tx=0).stop 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.610+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4102a20 msgr2=0x7fa6d4074df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.610+0000 7fa6d9a00700 1 --2- 172.21.15.146:0/1987880252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6d4102a20 0x7fa6d4074df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000c00 tx=0x7fa6c400bab0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.610+0000 7fa6d9000700 1 -- 172.21.15.146:0/1987880252 reap_dead start 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.610+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 shutdown_connections 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.610+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 >> 172.21.15.146:0/1987880252 conn(0x7fa6d40fe570 msgr2=0x7fa6d40feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.610+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 shutdown_connections 2024-04-24T08:19:23.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.610+0000 7fa6d9a00700 1 -- 172.21.15.146:0/1987880252 wait complete. 2024-04-24T08:19:23.840 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:23 smithi146 ceph-mon[27805]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-24T08:19:23.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:23 smithi153 ceph-mon[32746]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-24T08:19:24.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:23 smithi177 ceph-mon[33046]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-24T08:19:24.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 -- 172.21.15.146:0/1784333533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dac105920 msgr2=0x7f3dac109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:24.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 --2- 172.21.15.146:0/1784333533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dac105920 0x7f3dac109970 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3d980099c0 tx=0x7f3d98009cd0 comp rx=0 tx=0).stop 2024-04-24T08:19:24.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 -- 172.21.15.146:0/1784333533 shutdown_connections 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 --2- 172.21.15.146:0/1784333533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dac10a050 0x7f3dac111c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 --2- 172.21.15.146:0/1784333533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac104f70 0x7f3dac105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 --2- 172.21.15.146:0/1784333533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dac105920 0x7f3dac109970 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 -- 172.21.15.146:0/1784333533 >> 172.21.15.146:0/1784333533 conn(0x7f3dac1007c0 msgr2=0x7f3dac102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 -- 172.21.15.146:0/1784333533 shutdown_connections 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.811+0000 7f3db3400700 1 -- 172.21.15.146:0/1784333533 wait complete. 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.812+0000 7f3db3400700 1 Processor -- start 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.813+0000 7f3db3400700 1 -- start start 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.813+0000 7f3db3400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dac104f70 0x7f3dac19b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.813+0000 7f3db3400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac105920 0x7f3dac19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.813+0000 7f3db3400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dac10a050 0x7f3dac19fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.813+0000 7f3db3400700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac115080 con 0x7f3dac10a050 2024-04-24T08:19:24.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.813+0000 7f3db3400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac114f00 con 0x7f3dac104f70 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.813+0000 7f3db3400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac115200 con 0x7f3dac105920 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3db1400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dac10a050 0x7f3dac19fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac105920 0x7f3dac19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3db0a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dac104f70 0x7f3dac19b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac105920 0x7f3dac19ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57952/0 (socket says 172.21.15.146:57952) 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 -- 172.21.15.146:0/1180716010 learned_addr learned my addr 172.21.15.146:0/1180716010 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 -- 172.21.15.146:0/1180716010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dac104f70 msgr2=0x7f3dac19b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3dac104f70 0x7f3dac19b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 -- 172.21.15.146:0/1180716010 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dac10a050 msgr2=0x7f3dac19fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dac10a050 0x7f3dac19fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 -- 172.21.15.146:0/1180716010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac1a0520 con 0x7f3dac105920 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3dabe00700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac105920 0x7f3dac19ba50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d98005f50 tx=0x7f3d98003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.814+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98003680 con 0x7f3dac105920 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.815+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d980055a0 con 0x7f3dac105920 2024-04-24T08:19:24.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.815+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac1a07b0 con 0x7f3dac105920 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.815+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800fce0 con 0x7f3dac105920 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.815+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac1a8090 con 0x7f3dac105920 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.815+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3dac063280 con 0x7f3dac105920 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.818+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f3d9800f460 con 0x7f3dac105920 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.819+0000 7f3da9600700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3d940a3ef0 0x7f3d940a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.819+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f3d980d1690 con 0x7f3dac105920 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.819+0000 7f3db0a00700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3d940a3ef0 0x7f3d940a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.820+0000 7f3da9600700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] conn(0x7f3d940ad980 0x7f3d940afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.820+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 --> [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d98003680 con 0x7f3d940ad980 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.820+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f3d980d1a40 con 0x7f3dac105920 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.820+0000 7f3db0a00700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3d940a3ef0 0x7f3d940a63b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009c60 tx=0x7f3da0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.820+0000 7f3db1400700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] conn(0x7f3d940ad980 0x7f3d940afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.821+0000 7f3db1400700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] conn(0x7f3d940ad980 0x7f3d940afe20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:24.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.822+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== osd.8 v2:172.21.15.177:6816/4270133379 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d98003680 con 0x7f3d940ad980 2024-04-24T08:19:24.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.839+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 --> [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3dac063680 con 0x7f3d940ad980 2024-04-24T08:19:24.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.840+0000 7f3da9600700 1 -- 172.21.15.146:0/1180716010 <== osd.8 v2:172.21.15.177:6816/4270133379 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3dac063680 con 0x7f3d940ad980 2024-04-24T08:19:24.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] conn(0x7f3d940ad980 msgr2=0x7f3d940afe20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:24.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] conn(0x7f3d940ad980 0x7f3d940afe20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:24.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3d940a3ef0 msgr2=0x7f3d940a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:24.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3d940a3ef0 0x7f3d940a63b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009c60 tx=0x7f3da0009260 comp rx=0 tx=0).stop 2024-04-24T08:19:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac105920 msgr2=0x7f3dac19ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 --2- 172.21.15.146:0/1180716010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3dac105920 0x7f3dac19ba50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d98005f50 tx=0x7f3d98003a90 comp rx=0 tx=0).stop 2024-04-24T08:19:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db1400700 1 -- 172.21.15.146:0/1180716010 reap_dead start 2024-04-24T08:19:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 shutdown_connections 2024-04-24T08:19:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 >> 172.21.15.146:0/1180716010 conn(0x7f3dac1007c0 msgr2=0x7f3dac102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 shutdown_connections 2024-04-24T08:19:24.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:23.841+0000 7f3db3400700 1 -- 172.21.15.146:0/1180716010 wait complete. 2024-04-24T08:19:25.450 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:25.871 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:25 smithi146 ceph-mon[27805]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:19:25.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:25 smithi153 ceph-mon[32746]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 -- 172.21.15.146:0/2536153865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc640108c50 msgr2=0x7fc640109030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 --2- 172.21.15.146:0/2536153865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc640108c50 0x7fc640109030 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc630009990 tx=0x7fc630009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 -- 172.21.15.146:0/2536153865 shutdown_connections 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 --2- 172.21.15.146:0/2536153865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc640103500 0x7fc64010f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 --2- 172.21.15.146:0/2536153865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc640108c50 0x7fc640109030 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 --2- 172.21.15.146:0/2536153865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640102b40 0x7fc640102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 -- 172.21.15.146:0/2536153865 >> 172.21.15.146:0/2536153865 conn(0x7fc640079ca0 msgr2=0x7fc64007a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:26.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 -- 172.21.15.146:0/2536153865 shutdown_connections 2024-04-24T08:19:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.041+0000 7fc645600700 1 -- 172.21.15.146:0/2536153865 wait complete. 2024-04-24T08:19:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.042+0000 7fc645600700 1 Processor -- start 2024-04-24T08:19:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.042+0000 7fc645600700 1 -- start start 2024-04-24T08:19:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc645600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc640102b40 0x7fc64019b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc645600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640103500 0x7fc64019ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:26.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc645600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc640108c50 0x7fc64019fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:26.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc645600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc640112e40 con 0x7fc640102b40 2024-04-24T08:19:26.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc645600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc640112cc0 con 0x7fc640103500 2024-04-24T08:19:26.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc645600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc640112fc0 con 0x7fc640108c50 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63e000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640103500 0x7fc64019ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63ea00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc640102b40 0x7fc64019b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63e000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640103500 0x7fc64019ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60002/0 (socket says 172.21.15.146:60002) 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63e000700 1 -- 172.21.15.146:0/4268865193 learned_addr learned my addr 172.21.15.146:0/4268865193 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63f400700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc640108c50 0x7fc64019fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63e000700 1 -- 172.21.15.146:0/4268865193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc640108c50 msgr2=0x7fc64019fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63e000700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc640108c50 0x7fc64019fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.043+0000 7fc63e000700 1 -- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc640102b40 msgr2=0x7fc64019b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc63e000700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc640102b40 0x7fc64019b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc63e000700 1 -- 172.21.15.146:0/4268865193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6401a04d0 con 0x7fc640103500 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc63f400700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc640108c50 0x7fc64019fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc63e000700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640103500 0x7fc64019ba00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9a0 tx=0x7fc62c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00cbd0 con 0x7fc640103500 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc62c00cd70 con 0x7fc640103500 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6401a07c0 con 0x7fc640103500 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c018750 con 0x7fc640103500 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.044+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6401a8040 con 0x7fc640103500 2024-04-24T08:19:26.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.045+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc640101e70 con 0x7fc640103500 2024-04-24T08:19:26.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.047+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc62c010430 con 0x7fc640103500 2024-04-24T08:19:26.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.049+0000 7fc637400700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc6240a3d50 0x7fc6240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.049+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc62c0d0ff0 con 0x7fc640103500 2024-04-24T08:19:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.049+0000 7fc63ea00700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc6240a3d50 0x7fc6240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.049+0000 7fc637400700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] conn(0x7fc6240ad910 0x7fc6240afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.049+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 --> [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6240b0480 con 0x7fc6240ad910 2024-04-24T08:19:26.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.049+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc62c0d1400 con 0x7fc640103500 2024-04-24T08:19:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.050+0000 7fc63f400700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] conn(0x7fc6240ad910 0x7fc6240afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.050+0000 7fc63ea00700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc6240a3d50 0x7fc6240a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc63000b2e0 tx=0x7fc630005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:26.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.050+0000 7fc63f400700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] conn(0x7fc6240ad910 0x7fc6240afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:26.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.051+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== osd.6 v2:172.21.15.146:6818/3036960082 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc6240b0480 con 0x7fc6240ad910 2024-04-24T08:19:26.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:25 smithi177 ceph-mon[33046]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:19:26.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.069+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 --> [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc6401041f0 con 0x7fc6240ad910 2024-04-24T08:19:26.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.069+0000 7fc637400700 1 -- 172.21.15.146:0/4268865193 <== osd.6 v2:172.21.15.146:6818/3036960082 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc6401041f0 con 0x7fc6240ad910 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] conn(0x7fc6240ad910 msgr2=0x7fc6240afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] conn(0x7fc6240ad910 0x7fc6240afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc6240a3d50 msgr2=0x7fc6240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc6240a3d50 0x7fc6240a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc63000b2e0 tx=0x7fc630005960 comp rx=0 tx=0).stop 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640103500 msgr2=0x7fc64019ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 --2- 172.21.15.146:0/4268865193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc640103500 0x7fc64019ba00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9a0 tx=0x7fc62c00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc63f400700 1 -- 172.21.15.146:0/4268865193 reap_dead start 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 shutdown_connections 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 >> 172.21.15.146:0/4268865193 conn(0x7fc640079ca0 msgr2=0x7fc640101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 shutdown_connections 2024-04-24T08:19:26.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:26.070+0000 7fc645600700 1 -- 172.21.15.146:0/4268865193 wait complete. 2024-04-24T08:19:26.241 INFO:teuthology.orchestra.run.smithi146.stdout:146028888124 2024-04-24T08:19:26.242 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-24T08:19:26.874 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:26 smithi146 ceph-mon[27805]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-24T08:19:26.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:26 smithi153 ceph-mon[32746]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-24T08:19:27.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:26 smithi177 ceph-mon[33046]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-24T08:19:27.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 -- 172.21.15.146:0/1905733876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 msgr2=0x7f9e2c102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 --2- 172.21.15.146:0/1905733876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 0x7f9e2c102ea0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9e180098f0 tx=0x7f9e18009c00 comp rx=0 tx=0).stop 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 -- 172.21.15.146:0/1905733876 shutdown_connections 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 --2- 172.21.15.146:0/1905733876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e2c1033e0 0x7f9e2c10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 --2- 172.21.15.146:0/1905733876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e2c108a20 0x7f9e2c108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 --2- 172.21.15.146:0/1905733876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 0x7f9e2c102ea0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 -- 172.21.15.146:0/1905733876 >> 172.21.15.146:0/1905733876 conn(0x7f9e2c0fe570 msgr2=0x7f9e2c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 -- 172.21.15.146:0/1905733876 shutdown_connections 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.631+0000 7f9e32400700 1 -- 172.21.15.146:0/1905733876 wait complete. 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.632+0000 7f9e32400700 1 Processor -- start 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.632+0000 7f9e32400700 1 -- start start 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e32400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 0x7f9e2c19b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:27.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e32400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e2c1033e0 0x7f9e2c19b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e32400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e2c108a20 0x7f9e2c19fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e32400700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c112e00 con 0x7f9e2c1033e0 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e32400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c112c80 con 0x7f9e2c108a20 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e32400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c112f80 con 0x7f9e2c102a20 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e31000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 0x7f9e2c19b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e2be00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e2c1033e0 0x7f9e2c19b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e31a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e2c108a20 0x7f9e2c19fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e31000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 0x7f9e2c19b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57988/0 (socket says 172.21.15.146:57988) 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.633+0000 7f9e31000700 1 -- 172.21.15.146:0/2370246950 learned_addr learned my addr 172.21.15.146:0/2370246950 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e31a00700 1 -- 172.21.15.146:0/2370246950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 msgr2=0x7f9e2c19b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e31a00700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 0x7f9e2c19b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e31a00700 1 -- 172.21.15.146:0/2370246950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e2c1033e0 msgr2=0x7f9e2c19b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e31a00700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e2c1033e0 0x7f9e2c19b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:27.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e31a00700 1 -- 172.21.15.146:0/2370246950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2c1a0440 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e31000700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e2c102a20 0x7f9e2c19b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e31a00700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e2c108a20 0x7f9e2c19fd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400efc0 tx=0x7f9e2400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e240092e0 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e24004bb0 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e2c1a06d0 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.634+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24004280 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.635+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e2c1a7fb0 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.635+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9e0c001030 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.639+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f9e2401b030 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.641+0000 7f9e29600700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e140a3cd0 0x7f9e140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.641+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9e240d0910 con 0x7f9e2c108a20 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.641+0000 7f9e31000700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e140a3cd0 0x7f9e140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.641+0000 7f9e29600700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] conn(0x7f9e140ad890 0x7f9e140afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:27.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.641+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 --> [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e240048a0 con 0x7f9e140ad890 2024-04-24T08:19:27.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.641+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f9e240d0cc0 con 0x7f9e2c108a20 2024-04-24T08:19:27.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.642+0000 7f9e2be00700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] conn(0x7f9e140ad890 0x7f9e140afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:27.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.642+0000 7f9e31000700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e140a3cd0 0x7f9e140a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9e20011550 tx=0x7f9e20009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:27.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.642+0000 7f9e2be00700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] conn(0x7f9e140ad890 0x7f9e140afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:27.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.643+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== osd.10 v2:172.21.15.153:6824/679103416 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9e240048a0 con 0x7f9e140ad890 2024-04-24T08:19:27.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.661+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 --> [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9e0c002e60 con 0x7f9e140ad890 2024-04-24T08:19:27.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.662+0000 7f9e29600700 1 -- 172.21.15.146:0/2370246950 <== osd.10 v2:172.21.15.153:6824/679103416 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9e0c002e60 con 0x7f9e140ad890 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] conn(0x7f9e140ad890 msgr2=0x7f9e140afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] conn(0x7f9e140ad890 0x7f9e140afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e140a3cd0 msgr2=0x7f9e140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9e140a3cd0 0x7f9e140a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9e20011550 tx=0x7f9e20009310 comp rx=0 tx=0).stop 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e2c108a20 msgr2=0x7f9e2c19fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 --2- 172.21.15.146:0/2370246950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e2c108a20 0x7f9e2c19fd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400efc0 tx=0x7f9e2400c3a0 comp rx=0 tx=0).stop 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e31a00700 1 -- 172.21.15.146:0/2370246950 reap_dead start 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 shutdown_connections 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 >> 172.21.15.146:0/2370246950 conn(0x7f9e2c0fe570 msgr2=0x7f9e2c10e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 shutdown_connections 2024-04-24T08:19:27.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.663+0000 7f9e32400700 1 -- 172.21.15.146:0/2370246950 wait complete. 2024-04-24T08:19:29.037 INFO:teuthology.orchestra.run.smithi146.stdout:55834574925 2024-04-24T08:19:29.037 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-24T08:19:29.369 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.406+0000 7f5407a00700 1 -- 172.21.15.146:0/3278422917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400108b90 msgr2=0x7f5400108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.406+0000 7f5407a00700 1 --2- 172.21.15.146:0/3278422917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400108b90 0x7f5400108f70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009990 tx=0x7f53f4009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.407+0000 7f5407a00700 1 -- 172.21.15.146:0/3278422917 shutdown_connections 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.407+0000 7f5407a00700 1 --2- 172.21.15.146:0/3278422917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5400103500 0x7f540010f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.407+0000 7f5407a00700 1 --2- 172.21.15.146:0/3278422917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400108b90 0x7f5400108f70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.407+0000 7f5407a00700 1 --2- 172.21.15.146:0/3278422917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5400102b40 0x7f5400102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.407+0000 7f5407a00700 1 -- 172.21.15.146:0/3278422917 >> 172.21.15.146:0/3278422917 conn(0x7f5400079c50 msgr2=0x7f540007a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.407+0000 7f5407a00700 1 -- 172.21.15.146:0/3278422917 shutdown_connections 2024-04-24T08:19:29.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.407+0000 7f5407a00700 1 -- 172.21.15.146:0/3278422917 wait complete. 2024-04-24T08:19:29.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.409+0000 7f5407a00700 1 Processor -- start 2024-04-24T08:19:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.409+0000 7f5407a00700 1 -- start start 2024-04-24T08:19:29.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5407a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5400102b40 0x7f540019b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5407a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400103500 0x7f540019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5407a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5400108b90 0x7f540019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5407a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f54001150a0 con 0x7f5400108b90 2024-04-24T08:19:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5407a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5400114f20 con 0x7f5400103500 2024-04-24T08:19:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5407a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5400115220 con 0x7f5400102b40 2024-04-24T08:19:29.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f53ffe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400103500 0x7f540019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f53ffe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400103500 0x7f540019b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60040/0 (socket says 172.21.15.146:60040) 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f53ffe00700 1 -- 172.21.15.146:0/2581551139 learned_addr learned my addr 172.21.15.146:0/2581551139 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5405000700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5400102b40 0x7f540019b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.410+0000 7f5405a00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5400108b90 0x7f540019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53ffe00700 1 -- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5400102b40 msgr2=0x7f540019b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53ffe00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5400102b40 0x7f540019b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53ffe00700 1 -- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5400108b90 msgr2=0x7f540019fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:29.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53ffe00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5400108b90 0x7f540019fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53ffe00700 1 -- 172.21.15.146:0/2581551139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54001a0480 con 0x7f5400103500 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f5405a00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5400108b90 0x7f540019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53ffe00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400103500 0x7f540019b9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d980 tx=0x7f53ec00dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec016070 con 0x7f5400103500 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec004500 con 0x7f5400103500 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54001a0770 con 0x7f5400103500 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec004d00 con 0x7f5400103500 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.411+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54001a7ff0 con 0x7f5400103500 2024-04-24T08:19:29.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.413+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5400063200 con 0x7f5400103500 2024-04-24T08:19:29.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.415+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f53ec00bbf0 con 0x7f5400103500 2024-04-24T08:19:29.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.416+0000 7f53fd600700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53e80a3cd0 0x7f53e80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:29.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.417+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f53ec0d0ac0 con 0x7f5400103500 2024-04-24T08:19:29.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.417+0000 7f53fd600700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] conn(0x7f53e80ad890 0x7f53e80afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:29.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.417+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 --> [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53ec015ec0 con 0x7f53e80ad890 2024-04-24T08:19:29.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.417+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f53ec0d0e70 con 0x7f5400103500 2024-04-24T08:19:29.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.417+0000 7f5405000700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53e80a3cd0 0x7f53e80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:29.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.418+0000 7f5405a00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] conn(0x7f53e80ad890 0x7f53e80afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:29.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.419+0000 7f5405000700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53e80a3cd0 0x7f53e80a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f53f4006010 tx=0x7f53f400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:29.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.419+0000 7f5405a00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] conn(0x7f53e80ad890 0x7f53e80afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:29.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.420+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== osd.11 v2:172.21.15.177:6824/2574161099 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ec015ec0 con 0x7f53e80ad890 2024-04-24T08:19:29.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.440+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 --> [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5400063680 con 0x7f53e80ad890 2024-04-24T08:19:29.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.440+0000 7f53fd600700 1 -- 172.21.15.146:0/2581551139 <== osd.11 v2:172.21.15.177:6824/2574161099 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5400063680 con 0x7f53e80ad890 2024-04-24T08:19:29.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] conn(0x7f53e80ad890 msgr2=0x7f53e80afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:29.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] conn(0x7f53e80ad890 0x7f53e80afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:29.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53e80a3cd0 msgr2=0x7f53e80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:29.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53e80a3cd0 0x7f53e80a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f53f4006010 tx=0x7f53f400b540 comp rx=0 tx=0).stop 2024-04-24T08:19:29.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400103500 msgr2=0x7f540019b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:29.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 --2- 172.21.15.146:0/2581551139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5400103500 0x7f540019b9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d980 tx=0x7f53ec00dd40 comp rx=0 tx=0).stop 2024-04-24T08:19:29.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5405a00700 1 -- 172.21.15.146:0/2581551139 reap_dead start 2024-04-24T08:19:29.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 shutdown_connections 2024-04-24T08:19:29.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 >> 172.21.15.146:0/2581551139 conn(0x7f5400079c50 msgr2=0x7f5400101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:29.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 shutdown_connections 2024-04-24T08:19:29.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:29.441+0000 7f5407a00700 1 -- 172.21.15.146:0/2581551139 wait complete. 2024-04-24T08:19:29.800 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:29 smithi146 ceph-mon[27805]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:19:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:29 smithi177 ceph-mon[33046]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:19:29.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:29 smithi153 ceph-mon[32746]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:19:30.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.726+0000 7f8bb5000700 1 -- 172.21.15.146:0/178978391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0103500 msgr2=0x7f8bb010f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:30.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.726+0000 7f8bb5000700 1 --2- 172.21.15.146:0/178978391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0103500 0x7f8bb010f960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009f90 tx=0x7f8ba800b330 comp rx=0 tx=0).stop 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.727+0000 7f8bb5000700 1 -- 172.21.15.146:0/178978391 shutdown_connections 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.727+0000 7f8bb5000700 1 --2- 172.21.15.146:0/178978391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0103500 0x7f8bb010f960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.727+0000 7f8bb5000700 1 --2- 172.21.15.146:0/178978391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bb0108b90 0x7f8bb0108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.727+0000 7f8bb5000700 1 --2- 172.21.15.146:0/178978391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bb0102b40 0x7f8bb0102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.727+0000 7f8bb5000700 1 -- 172.21.15.146:0/178978391 >> 172.21.15.146:0/178978391 conn(0x7f8bb0079c50 msgr2=0x7f8bb007a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.727+0000 7f8bb5000700 1 -- 172.21.15.146:0/178978391 shutdown_connections 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.727+0000 7f8bb5000700 1 -- 172.21.15.146:0/178978391 wait complete. 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 Processor -- start 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 -- start start 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0102b40 0x7f8bb010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bb0103500 0x7f8bb010d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bb0108b90 0x7f8bb010d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:30.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb0076b40 con 0x7f8bb0103500 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb00769c0 con 0x7f8bb0102b40 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.728+0000 7f8bb5000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb0076cc0 con 0x7f8bb0108b90 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bade00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0102b40 0x7f8bb010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bae800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bb0108b90 0x7f8bb010d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bade00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0102b40 0x7f8bb010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60028/0 (socket says 172.21.15.146:60028) 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bade00700 1 -- 172.21.15.146:0/2237459595 learned_addr learned my addr 172.21.15.146:0/2237459595 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bad400700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bb0103500 0x7f8bb010d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bae800700 1 -- 172.21.15.146:0/2237459595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0102b40 msgr2=0x7f8bb010ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bae800700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0102b40 0x7f8bb010ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bae800700 1 -- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bb0103500 msgr2=0x7f8bb010d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bae800700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bb0103500 0x7f8bb010d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:30.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bae800700 1 -- 172.21.15.146:0/2237459595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb01abbe0 con 0x7f8bb0108b90 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bade00700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bb0102b40 0x7f8bb010ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.729+0000 7f8bae800700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bb0108b90 0x7f8bb010d750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009f90 tx=0x7f8ba800cfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.730+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8021070 con 0x7f8bb0108b90 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.730+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba8007ac0 con 0x7f8bb0108b90 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.730+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb01abe70 con 0x7f8bb0108b90 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.730+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba80052a0 con 0x7f8bb0108b90 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.730+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb01ac380 con 0x7f8bb0108b90 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.731+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8bb0063200 con 0x7f8bb0108b90 2024-04-24T08:19:30.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.733+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8ba8024070 con 0x7f8bb0108b90 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.734+0000 7f8ba6a00700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8b940a3d50 0x7f8b940a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.734+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8ba80d17b0 con 0x7f8bb0108b90 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.735+0000 7f8bade00700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8b940a3d50 0x7f8b940a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.735+0000 7f8ba6a00700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] conn(0x7f8b940ad910 0x7f8b940afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.735+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 --> [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ba800c850 con 0x7f8b940ad910 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.735+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8ba80d1b60 con 0x7f8bb0108b90 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.735+0000 7f8bade00700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8b940a3d50 0x7f8b940a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0005950 tx=0x7f8ba0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.735+0000 7f8bad400700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] conn(0x7f8b940ad910 0x7f8b940afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.736+0000 7f8bad400700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] conn(0x7f8b940ad910 0x7f8b940afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.737+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== osd.5 v2:172.21.15.177:6808/3773516414 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ba800c850 con 0x7f8b940ad910 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.755+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 --> [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8bb0063680 con 0x7f8b940ad910 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8ba6a00700 1 -- 172.21.15.146:0/2237459595 <== osd.5 v2:172.21.15.177:6808/3773516414 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8bb0063680 con 0x7f8b940ad910 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] conn(0x7f8b940ad910 msgr2=0x7f8b940afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] conn(0x7f8b940ad910 0x7f8b940afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:30.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8b940a3d50 msgr2=0x7f8b940a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8b940a3d50 0x7f8b940a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0005950 tx=0x7f8ba0009310 comp rx=0 tx=0).stop 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bb0108b90 msgr2=0x7f8bb010d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 --2- 172.21.15.146:0/2237459595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8bb0108b90 0x7f8bb010d750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8009f90 tx=0x7f8ba800cfa0 comp rx=0 tx=0).stop 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bae800700 1 -- 172.21.15.146:0/2237459595 reap_dead start 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 shutdown_connections 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 >> 172.21.15.146:0/2237459595 conn(0x7f8bb0079c50 msgr2=0x7f8bb0101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 shutdown_connections 2024-04-24T08:19:30.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:27.756+0000 7f8bb5000700 1 -- 172.21.15.146:0/2237459595 wait complete. 2024-04-24T08:19:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 -- 172.21.15.146:0/1923416494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406b090 msgr2=0x7f972406b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 --2- 172.21.15.146:0/1923416494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406b090 0x7f972406b4f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9710009990 tx=0x7f9710009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 -- 172.21.15.146:0/1923416494 shutdown_connections 2024-04-24T08:19:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 --2- 172.21.15.146:0/1923416494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f972406ba30 0x7f972410f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 --2- 172.21.15.146:0/1923416494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9724108b60 0x7f9724108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 --2- 172.21.15.146:0/1923416494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406b090 0x7f972406b4f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 -- 172.21.15.146:0/1923416494 >> 172.21.15.146:0/1923416494 conn(0x7f9724078960 msgr2=0x7f9724078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 -- 172.21.15.146:0/1923416494 shutdown_connections 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.367+0000 7f972aa00700 1 -- 172.21.15.146:0/1923416494 wait complete. 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.368+0000 7f972aa00700 1 Processor -- start 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.368+0000 7f972aa00700 1 -- start start 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.368+0000 7f972aa00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f972406b090 0x7f97241992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.368+0000 7f972aa00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406ba30 0x7f9724199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f972aa00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9724108b60 0x7f972419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f972aa00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9724112e10 con 0x7f972406ba30 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f972aa00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9724112c90 con 0x7f9724108b60 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f972aa00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9724112f90 con 0x7f972406b090 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9723400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406ba30 0x7f9724199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9728a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9724108b60 0x7f972419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9723400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406ba30 0x7f9724199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51906/0 (socket says 172.21.15.146:51906) 2024-04-24T08:19:31.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9723400700 1 -- 172.21.15.146:0/87919158 learned_addr learned my addr 172.21.15.146:0/87919158 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9723e00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f972406b090 0x7f97241992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9728a00700 1 -- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f972406b090 msgr2=0x7f97241992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9728a00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f972406b090 0x7f97241992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9728a00700 1 -- 172.21.15.146:0/87919158 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406ba30 msgr2=0x7f9724199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9728a00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f972406ba30 0x7f9724199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.369+0000 7f9728a00700 1 -- 172.21.15.146:0/87919158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972419e2d0 con 0x7f9724108b60 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.370+0000 7f9728a00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9724108b60 0x7f972419db90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f971c00efc0 tx=0x7f971c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.370+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c0092e0 con 0x7f9724108b60 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.370+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971c004bb0 con 0x7f9724108b60 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.370+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c004280 con 0x7f9724108b60 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.370+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972419e560 con 0x7f9724108b60 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.370+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97241a5e40 con 0x7f9724108b60 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.371+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9724063240 con 0x7f9724108b60 2024-04-24T08:19:31.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.373+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f971c01b030 con 0x7f9724108b60 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.374+0000 7f9720c00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97080a3d50 0x7f97080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.374+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f971c0d09d0 con 0x7f9724108b60 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.375+0000 7f9723e00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97080a3d50 0x7f97080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.375+0000 7f9720c00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] conn(0x7f97080ad910 0x7f97080afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.375+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 --> [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f971c0048a0 con 0x7f97080ad910 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.375+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f971c0d0d80 con 0x7f9724108b60 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.375+0000 7f9723400700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] conn(0x7f97080ad910 0x7f97080afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:31.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.375+0000 7f9723e00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97080a3d50 0x7f97080a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9714009be0 tx=0x7f9714009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.376+0000 7f9723400700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] conn(0x7f97080ad910 0x7f97080afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.377+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== osd.2 v2:172.21.15.177:6800/96699540 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f971c0048a0 con 0x7f97080ad910 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.396+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 --> [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f97240636c0 con 0x7f97080ad910 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.396+0000 7f9720c00700 1 -- 172.21.15.146:0/87919158 <== osd.2 v2:172.21.15.177:6800/96699540 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f97240636c0 con 0x7f97080ad910 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] conn(0x7f97080ad910 msgr2=0x7f97080afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] conn(0x7f97080ad910 0x7f97080afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97080a3d50 msgr2=0x7f97080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97080a3d50 0x7f97080a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9714009be0 tx=0x7f9714009260 comp rx=0 tx=0).stop 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9724108b60 msgr2=0x7f972419db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 --2- 172.21.15.146:0/87919158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9724108b60 0x7f972419db90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f971c00efc0 tx=0x7f971c00c3a0 comp rx=0 tx=0).stop 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f9728a00700 1 -- 172.21.15.146:0/87919158 reap_dead start 2024-04-24T08:19:31.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 shutdown_connections 2024-04-24T08:19:31.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 >> 172.21.15.146:0/87919158 conn(0x7f9724078960 msgr2=0x7f97240ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:31.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 shutdown_connections 2024-04-24T08:19:31.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:30.397+0000 7f972aa00700 1 -- 172.21.15.146:0/87919158 wait complete. 2024-04-24T08:19:31.646 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:31 smithi146 ceph-mon[27805]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:31 smithi177 ceph-mon[33046]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:31.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:31 smithi153 ceph-mon[32746]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:32.331 INFO:teuthology.orchestra.run.smithi146.stdout:206158430257 2024-04-24T08:19:32.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-24T08:19:32.342 INFO:teuthology.orchestra.run.smithi146.stdout:77309411400 2024-04-24T08:19:32.342 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-24T08:19:32.656 INFO:teuthology.orchestra.run.smithi146.stdout:322122547226 2024-04-24T08:19:32.656 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-24T08:19:33.255 INFO:teuthology.orchestra.run.smithi146.stdout:266287972391 2024-04-24T08:19:33.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-24T08:19:33.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 -- 172.21.15.146:0/3372290773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3630108a10 msgr2=0x7f3630108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:33.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 --2- 172.21.15.146:0/3372290773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3630108a10 0x7f3630108df0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3624009990 tx=0x7f3624009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 -- 172.21.15.146:0/3372290773 shutdown_connections 2024-04-24T08:19:33.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 --2- 172.21.15.146:0/3372290773 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f36301033d0 0x7f363010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 --2- 172.21.15.146:0/3372290773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3630108a10 0x7f3630108df0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 --2- 172.21.15.146:0/3372290773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3630102a10 0x7f3630102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 -- 172.21.15.146:0/3372290773 >> 172.21.15.146:0/3372290773 conn(0x7f36300fe500 msgr2=0x7f3630100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:33.785 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:33 smithi146 ceph-mon[27805]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 -- 172.21.15.146:0/3372290773 shutdown_connections 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.783+0000 7f3636200700 1 -- 172.21.15.146:0/3372290773 wait complete. 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.784+0000 7f3636200700 1 Processor -- start 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.784+0000 7f3636200700 1 -- start start 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3636200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3630102a10 0x7f363019b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3636200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36301033d0 0x7f363019b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3636200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3630108a10 0x7f363019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3636200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3630112d90 con 0x7f36301033d0 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3636200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3630112c10 con 0x7f3630108a10 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3636200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3630112f10 con 0x7f3630102a10 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f362fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36301033d0 0x7f363019b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:33.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3634e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3630102a10 0x7f363019b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:33.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f362fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36301033d0 0x7f363019b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56222/0 (socket says 172.21.15.146:56222) 2024-04-24T08:19:33.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3634e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3630102a10 0x7f363019b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:60704/0 (socket says 172.21.15.146:60704) 2024-04-24T08:19:33.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f362fe00700 1 -- 172.21.15.146:0/1195701747 learned_addr learned my addr 172.21.15.146:0/1195701747 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:33.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.785+0000 7f3635800700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3630108a10 0x7f363019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3635800700 1 -- 172.21.15.146:0/1195701747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3630102a10 msgr2=0x7f363019b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3635800700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3630102a10 0x7f363019b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3635800700 1 -- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36301033d0 msgr2=0x7f363019b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3635800700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36301033d0 0x7f363019b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3635800700 1 -- 172.21.15.146:0/1195701747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36301a03d0 con 0x7f3630108a10 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3634e00700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3630102a10 0x7f363019b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3635800700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3630108a10 0x7f363019fc90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f362800efc0 tx=0x7f362800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36280092e0 con 0x7f3630108a10 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3628004bb0 con 0x7f3630108a10 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36301a0660 con 0x7f3630108a10 2024-04-24T08:19:33.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628004280 con 0x7f3630108a10 2024-04-24T08:19:33.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.786+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36301a7f40 con 0x7f3630108a10 2024-04-24T08:19:33.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.787+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f363010f8c0 con 0x7f3630108a10 2024-04-24T08:19:33.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.790+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f362801b030 con 0x7f3630108a10 2024-04-24T08:19:33.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.791+0000 7f362d600700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36180a3cd0 0x7f36180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:33.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.791+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f36280d09f0 con 0x7f3630108a10 2024-04-24T08:19:33.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.791+0000 7f3634e00700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36180a3cd0 0x7f36180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:33.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.791+0000 7f362d600700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] conn(0x7f36180ad890 0x7f36180afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:33.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.791+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 --> [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36280048a0 con 0x7f36180ad890 2024-04-24T08:19:33.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.791+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f36280d0da0 con 0x7f3630108a10 2024-04-24T08:19:33.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.792+0000 7f362fe00700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] conn(0x7f36180ad890 0x7f36180afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:33.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.792+0000 7f3634e00700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36180a3cd0 0x7f36180a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3624006010 tx=0x7f362400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:33.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.792+0000 7f362fe00700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] conn(0x7f36180ad890 0x7f36180afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:33.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.793+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== osd.0 v2:172.21.15.146:6802/3767707346 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36280048a0 con 0x7f36180ad890 2024-04-24T08:19:33.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.811+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 --> [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3630104150 con 0x7f36180ad890 2024-04-24T08:19:33.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f362d600700 1 -- 172.21.15.146:0/1195701747 <== osd.0 v2:172.21.15.146:6802/3767707346 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3630104150 con 0x7f36180ad890 2024-04-24T08:19:33.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] conn(0x7f36180ad890 msgr2=0x7f36180afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:33.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f3636200700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] conn(0x7f36180ad890 0x7f36180afd30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36180a3cd0 msgr2=0x7f36180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f3636200700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f36180a3cd0 0x7f36180a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3624006010 tx=0x7f362400b260 comp rx=0 tx=0).stop 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3630108a10 msgr2=0x7f363019fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f3636200700 1 --2- 172.21.15.146:0/1195701747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3630108a10 0x7f363019fc90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f362800efc0 tx=0x7f362800c3a0 comp rx=0 tx=0).stop 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.812+0000 7f3635800700 1 -- 172.21.15.146:0/1195701747 reap_dead start 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.813+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 shutdown_connections 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.813+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 >> 172.21.15.146:0/1195701747 conn(0x7f36300fe500 msgr2=0x7f3630107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.813+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 shutdown_connections 2024-04-24T08:19:33.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:33.813+0000 7f3636200700 1 -- 172.21.15.146:0/1195701747 wait complete. 2024-04-24T08:19:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:33 smithi177 ceph-mon[33046]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:33.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:33 smithi153 ceph-mon[32746]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:34.014 INFO:teuthology.orchestra.run.smithi146.stdout:171798691895 2024-04-24T08:19:34.015 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-24T08:19:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:35 smithi177 ceph-mon[33046]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:19:35.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:35 smithi153 ceph-mon[32746]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:19:35.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:35 smithi146 ceph-mon[27805]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:19:36.500 INFO:teuthology.orchestra.run.smithi146.stdout:356482285590 2024-04-24T08:19:36.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-24T08:19:37.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:37 smithi177 ceph-mon[33046]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:19:37.923 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:37 smithi146 ceph-mon[27805]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:19:37.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:37 smithi153 ceph-mon[32746]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:19:38.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 -- 172.21.15.146:0/1464314529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e80102b40 msgr2=0x7f6e80102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 --2- 172.21.15.146:0/1464314529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e80102b40 0x7f6e80102fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009990 tx=0x7f6e6c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 -- 172.21.15.146:0/1464314529 shutdown_connections 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 --2- 172.21.15.146:0/1464314529 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e80103500 0x7f6e8010f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 --2- 172.21.15.146:0/1464314529 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e80108b90 0x7f6e80108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 --2- 172.21.15.146:0/1464314529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e80102b40 0x7f6e80102fc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 -- 172.21.15.146:0/1464314529 >> 172.21.15.146:0/1464314529 conn(0x7f6e80079c50 msgr2=0x7f6e8007a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 -- 172.21.15.146:0/1464314529 shutdown_connections 2024-04-24T08:19:38.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.543+0000 7f6e85200700 1 -- 172.21.15.146:0/1464314529 wait complete. 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.544+0000 7f6e85200700 1 Processor -- start 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e85200700 1 -- start start 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e85200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e80102b40 0x7f6e8019b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e85200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e80103500 0x7f6e8019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e85200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e80108b90 0x7f6e8019fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e85200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6e80112e40 con 0x7f6e80102b40 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e85200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e80112cc0 con 0x7f6e80108b90 2024-04-24T08:19:38.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e85200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e80112fc0 con 0x7f6e80103500 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e7de00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e80102b40 0x7f6e8019b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e7de00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e80102b40 0x7f6e8019b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56248/0 (socket says 172.21.15.146:56248) 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e7de00700 1 -- 172.21.15.146:0/4166143593 learned_addr learned my addr 172.21.15.146:0/4166143593 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.545+0000 7f6e7d400700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e80103500 0x7f6e8019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7e800700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e80108b90 0x7f6e8019fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7de00700 1 -- 172.21.15.146:0/4166143593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e80103500 msgr2=0x7f6e8019ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7de00700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e80103500 0x7f6e8019ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7de00700 1 -- 172.21.15.146:0/4166143593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e80108b90 msgr2=0x7f6e8019fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7de00700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e80108b90 0x7f6e8019fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7de00700 1 -- 172.21.15.146:0/4166143593 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e801a0520 con 0x7f6e80102b40 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7e800700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e80108b90 0x7f6e8019fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e7de00700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e80102b40 0x7f6e8019b510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000e9a0 tx=0x7f6e7000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:38.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7000cbd0 con 0x7f6e80102b40 2024-04-24T08:19:38.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e7000cd70 con 0x7f6e80102b40 2024-04-24T08:19:38.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70018750 con 0x7f6e80102b40 2024-04-24T08:19:38.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.546+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e801a0810 con 0x7f6e80102b40 2024-04-24T08:19:38.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.547+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e801a8090 con 0x7f6e80102b40 2024-04-24T08:19:38.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.547+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e8010f9c0 con 0x7f6e80102b40 2024-04-24T08:19:38.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.550+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f6e70010430 con 0x7f6e80102b40 2024-04-24T08:19:38.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.551+0000 7f6e76a00700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6e640a3cd0 0x7f6e640a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.551+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f6e700d0ff0 con 0x7f6e80102b40 2024-04-24T08:19:38.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.551+0000 7f6e7d400700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6e640a3cd0 0x7f6e640a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.551+0000 7f6e76a00700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] conn(0x7f6e640ad8f0 0x7f6e640afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.551+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 --> [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e640b0460 con 0x7f6e640ad8f0 2024-04-24T08:19:38.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.551+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6e700d13a0 con 0x7f6e80102b40 2024-04-24T08:19:38.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.552+0000 7f6e7e800700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] conn(0x7f6e640ad8f0 0x7f6e640afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.552+0000 7f6e7d400700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6e640a3cd0 0x7f6e640a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6e8007a590 tx=0x7f6e6c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:38.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.552+0000 7f6e7e800700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] conn(0x7f6e640ad8f0 0x7f6e640afd90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:38.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.553+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== osd.3 v2:172.21.15.146:6810/602409571 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6e640b0460 con 0x7f6e640ad8f0 2024-04-24T08:19:38.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.570+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 --> [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6e801041f0 con 0x7f6e640ad8f0 2024-04-24T08:19:38.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.571+0000 7f6e76a00700 1 -- 172.21.15.146:0/4166143593 <== osd.3 v2:172.21.15.146:6810/602409571 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6e801041f0 con 0x7f6e640ad8f0 2024-04-24T08:19:38.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.571+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] conn(0x7f6e640ad8f0 msgr2=0x7f6e640afd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.571+0000 7f6e85200700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] conn(0x7f6e640ad8f0 0x7f6e640afd90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.571+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6e640a3cd0 msgr2=0x7f6e640a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.571+0000 7f6e85200700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f6e640a3cd0 0x7f6e640a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6e8007a590 tx=0x7f6e6c005fd0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.571+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e80102b40 msgr2=0x7f6e8019b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.571+0000 7f6e85200700 1 --2- 172.21.15.146:0/4166143593 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e80102b40 0x7f6e8019b510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6e7000e9a0 tx=0x7f6e7000ecb0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.572+0000 7f6e7e800700 1 -- 172.21.15.146:0/4166143593 reap_dead start 2024-04-24T08:19:38.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.572+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 shutdown_connections 2024-04-24T08:19:38.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.572+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 >> 172.21.15.146:0/4166143593 conn(0x7f6e80079c50 msgr2=0x7f6e8010df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:38.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.572+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 shutdown_connections 2024-04-24T08:19:38.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.572+0000 7f6e85200700 1 -- 172.21.15.146:0/4166143593 wait complete. 2024-04-24T08:19:38.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 -- 172.21.15.146:0/1842650281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818102b40 msgr2=0x7fc818102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 --2- 172.21.15.146:0/1842650281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818102b40 0x7fc818102fc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc800009990 tx=0x7fc800009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 -- 172.21.15.146:0/1842650281 shutdown_connections 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 --2- 172.21.15.146:0/1842650281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc818103500 0x7fc81810f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 --2- 172.21.15.146:0/1842650281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc818108b90 0x7fc818108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 --2- 172.21.15.146:0/1842650281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818102b40 0x7fc818102fc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 -- 172.21.15.146:0/1842650281 >> 172.21.15.146:0/1842650281 conn(0x7fc818079c50 msgr2=0x7fc81807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 -- 172.21.15.146:0/1842650281 shutdown_connections 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.447+0000 7fc81f200700 1 -- 172.21.15.146:0/1842650281 wait complete. 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 Processor -- start 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 -- start start 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc818102b40 0x7fc81819b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818103500 0x7fc81819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc818108b90 0x7fc81819fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc818112e40 con 0x7fc818103500 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc818112cc0 con 0x7fc818102b40 2024-04-24T08:19:38.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.448+0000 7fc81f200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc818112fc0 con 0x7fc818108b90 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818103500 0x7fc81819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818103500 0x7fc81819bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56238/0 (socket says 172.21.15.146:56238) 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 -- 172.21.15.146:0/3627741998 learned_addr learned my addr 172.21.15.146:0/3627741998 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc817e00700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc818102b40 0x7fc81819b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc81d200700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc818108b90 0x7fc81819fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 -- 172.21.15.146:0/3627741998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc818108b90 msgr2=0x7fc81819fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc818108b90 0x7fc81819fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 -- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc818102b40 msgr2=0x7fc81819b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc818102b40 0x7fc81819b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc80fe00700 1 -- 172.21.15.146:0/3627741998 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8181a05a0 con 0x7fc818103500 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc81d200700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc818108b90 0x7fc81819fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.449+0000 7fc817e00700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc818102b40 0x7fc81819b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.450+0000 7fc80fe00700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818103500 0x7fc81819bad0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc800005910 tx=0x7fc80000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.450+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800027070 con 0x7fc818103500 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.450+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc800004650 con 0x7fc818103500 2024-04-24T08:19:38.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.450+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8181a0830 con 0x7fc818103500 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.450+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80000f460 con 0x7fc818103500 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.450+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8181a8110 con 0x7fc818103500 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.451+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc81810f930 con 0x7fc818103500 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.453+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc80002a070 con 0x7fc818103500 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.454+0000 7fc815600700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7fc0a3cd0 0x7fc7fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.455+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc8000d1b60 con 0x7fc818103500 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.455+0000 7fc817e00700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7fc0a3cd0 0x7fc7fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.455+0000 7fc815600700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] conn(0x7fc7fc0ad890 0x7fc7fc0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.455+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 --> [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc800019e80 con 0x7fc7fc0ad890 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.455+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc8000d1f90 con 0x7fc818103500 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.455+0000 7fc81d200700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] conn(0x7fc7fc0ad890 0x7fc7fc0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.456+0000 7fc817e00700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7fc0a3cd0 0x7fc7fc0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc808009fd0 tx=0x7fc808009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.456+0000 7fc81d200700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] conn(0x7fc7fc0ad890 0x7fc7fc0afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.457+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== osd.7 v2:172.21.15.153:6816/2204927697 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc800019e80 con 0x7fc7fc0ad890 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.474+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 --> [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc8181041f0 con 0x7fc7fc0ad890 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.475+0000 7fc815600700 1 -- 172.21.15.146:0/3627741998 <== osd.7 v2:172.21.15.153:6816/2204927697 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc8181041f0 con 0x7fc7fc0ad890 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.475+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] conn(0x7fc7fc0ad890 msgr2=0x7fc7fc0afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.475+0000 7fc81f200700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] conn(0x7fc7fc0ad890 0x7fc7fc0afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7fc0a3cd0 msgr2=0x7fc7fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc7fc0a3cd0 0x7fc7fc0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc808009fd0 tx=0x7fc808009260 comp rx=0 tx=0).stop 2024-04-24T08:19:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818103500 msgr2=0x7fc81819bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 --2- 172.21.15.146:0/3627741998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc818103500 0x7fc81819bad0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc800005910 tx=0x7fc80000bab0 comp rx=0 tx=0).stop 2024-04-24T08:19:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81d200700 1 -- 172.21.15.146:0/3627741998 reap_dead start 2024-04-24T08:19:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 shutdown_connections 2024-04-24T08:19:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 >> 172.21.15.146:0/3627741998 conn(0x7fc818079c50 msgr2=0x7fc81810df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 shutdown_connections 2024-04-24T08:19:38.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:38.476+0000 7fc81f200700 1 -- 172.21.15.146:0/3627741998 wait complete. 2024-04-24T08:19:39.032 INFO:teuthology.orchestra.run.smithi146.stdout:34359738452 2024-04-24T08:19:39.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-24T08:19:39.324 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:39.793 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:39 smithi146 ceph-mon[27805]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:39.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:39 smithi177 ceph-mon[33046]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:39.849 INFO:teuthology.orchestra.run.smithi146.stdout:227633266733 2024-04-24T08:19:39.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-24T08:19:39.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:39 smithi153 ceph-mon[32746]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:19:40.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 -- 172.21.15.146:0/488337788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641c108b90 msgr2=0x7f641c108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:40.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 --2- 172.21.15.146:0/488337788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641c108b90 0x7f641c108f70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f640c009990 tx=0x7f640c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 -- 172.21.15.146:0/488337788 shutdown_connections 2024-04-24T08:19:40.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 --2- 172.21.15.146:0/488337788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f641c103500 0x7f641c10f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 --2- 172.21.15.146:0/488337788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641c108b90 0x7f641c108f70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 --2- 172.21.15.146:0/488337788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f641c102b40 0x7f641c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 -- 172.21.15.146:0/488337788 >> 172.21.15.146:0/488337788 conn(0x7f641c079c50 msgr2=0x7f641c07a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:40.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 -- 172.21.15.146:0/488337788 shutdown_connections 2024-04-24T08:19:40.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.704+0000 7f6422600700 1 -- 172.21.15.146:0/488337788 wait complete. 2024-04-24T08:19:40.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 Processor -- start 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 -- start start 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f641c102b40 0x7f641c199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641c103500 0x7f641c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f641c108b90 0x7f641c19dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f641c112e50 con 0x7f641c108b90 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f641c112cd0 con 0x7f641c102b40 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f6422600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f641c112fd0 con 0x7f641c103500 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f641b400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f641c102b40 0x7f641c199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f641b400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f641c102b40 0x7f641c199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48968/0 (socket says 172.21.15.146:48968) 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.706+0000 7f641b400700 1 -- 172.21.15.146:0/3906016447 learned_addr learned my addr 172.21.15.146:0/3906016447 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641aa00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641c103500 0x7f641c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641be00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f641c108b90 0x7f641c19dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:40.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641b400700 1 -- 172.21.15.146:0/3906016447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641c103500 msgr2=0x7f641c1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:40.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641b400700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641c103500 0x7f641c1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641b400700 1 -- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f641c108b90 msgr2=0x7f641c19dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:40.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641b400700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f641c108b90 0x7f641c19dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641b400700 1 -- 172.21.15.146:0/3906016447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641c19e390 con 0x7f641c102b40 2024-04-24T08:19:40.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641be00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f641c108b90 0x7f641c19dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:40.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.707+0000 7f641b400700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f641c102b40 0x7f641c199380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f640c000c00 tx=0x7f640c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:40.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.708+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c027070 con 0x7f641c102b40 2024-04-24T08:19:40.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.708+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f640c003d70 con 0x7f641c102b40 2024-04-24T08:19:40.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.708+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641c19e620 con 0x7f641c102b40 2024-04-24T08:19:40.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.708+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c00f460 con 0x7f641c102b40 2024-04-24T08:19:40.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.708+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641c1a5f00 con 0x7f641c102b40 2024-04-24T08:19:40.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.709+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f641c063180 con 0x7f641c102b40 2024-04-24T08:19:40.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.713+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f640c02a070 con 0x7f641c102b40 2024-04-24T08:19:40.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.714+0000 7f6413e00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64000a3d50 0x7f64000a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:40.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.714+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f640c0d1db0 con 0x7f641c102b40 2024-04-24T08:19:40.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.714+0000 7f641aa00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64000a3d50 0x7f64000a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:40.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.714+0000 7f6413e00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] conn(0x7f64000ad910 0x7f64000afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:40.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.714+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 --> [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64000b0480 con 0x7f64000ad910 2024-04-24T08:19:40.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.714+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f640c0d21e0 con 0x7f641c102b40 2024-04-24T08:19:40.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.715+0000 7f641be00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] conn(0x7f64000ad910 0x7f64000afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:40.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.715+0000 7f641aa00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64000a3d50 0x7f64000a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f641c07a370 tx=0x7f6408009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:40.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.716+0000 7f641be00700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] conn(0x7f64000ad910 0x7f64000afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:40.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.717+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== osd.9 v2:172.21.15.146:6826/2032154724 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f64000b0480 con 0x7f64000ad910 2024-04-24T08:19:40.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.736+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 --> [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f641c0635b0 con 0x7f64000ad910 2024-04-24T08:19:40.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.736+0000 7f6413e00700 1 -- 172.21.15.146:0/3906016447 <== osd.9 v2:172.21.15.146:6826/2032154724 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f641c0635b0 con 0x7f64000ad910 2024-04-24T08:19:40.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] conn(0x7f64000ad910 msgr2=0x7f64000afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:40.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] conn(0x7f64000ad910 0x7f64000afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64000a3d50 msgr2=0x7f64000a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64000a3d50 0x7f64000a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f641c07a370 tx=0x7f6408009260 comp rx=0 tx=0).stop 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f641c102b40 msgr2=0x7f641c199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 --2- 172.21.15.146:0/3906016447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f641c102b40 0x7f641c199380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f640c000c00 tx=0x7f640c00bab0 comp rx=0 tx=0).stop 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f641be00700 1 -- 172.21.15.146:0/3906016447 reap_dead start 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 shutdown_connections 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 >> 172.21.15.146:0/3906016447 conn(0x7f641c079c50 msgr2=0x7f641c101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 shutdown_connections 2024-04-24T08:19:40.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:40.737+0000 7f6422600700 1 -- 172.21.15.146:0/3906016447 wait complete. 2024-04-24T08:19:41.552 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:41 smithi146 ceph-mon[27805]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-24T08:19:41.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:41 smithi177 ceph-mon[33046]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-24T08:19:41.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:41 smithi153 ceph-mon[32746]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-24T08:19:42.492 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:42.497 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:42.503 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:42.503 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:42.505 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:42.535 INFO:teuthology.orchestra.run.smithi146.stdout:300647710755 2024-04-24T08:19:42.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-24T08:19:42.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 -- 172.21.15.146:0/590503335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c06a800 msgr2=0x7f2e8c06ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:42.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 --2- 172.21.15.146:0/590503335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c06a800 0x7f2e8c06ac60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009990 tx=0x7f2e78009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:42.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 -- 172.21.15.146:0/590503335 shutdown_connections 2024-04-24T08:19:42.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 --2- 172.21.15.146:0/590503335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06b1a0 0x7f2e8c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:42.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 --2- 172.21.15.146:0/590503335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e8c108b70 0x7f2e8c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:42.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 --2- 172.21.15.146:0/590503335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c06a800 0x7f2e8c06ac60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:42.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 -- 172.21.15.146:0/590503335 >> 172.21.15.146:0/590503335 conn(0x7f2e8c0789a0 msgr2=0x7f2e8c078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:42.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 -- 172.21.15.146:0/590503335 shutdown_connections 2024-04-24T08:19:42.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.990+0000 7f2e92e00700 1 -- 172.21.15.146:0/590503335 wait complete. 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.992+0000 7f2e92e00700 1 Processor -- start 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.992+0000 7f2e92e00700 1 -- start start 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.992+0000 7f2e92e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06a800 0x7f2e8c0742c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e92e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e8c06b1a0 0x7f2e8c06d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e92e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c108b70 0x7f2e8c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e92e00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c112e20 con 0x7f2e8c108b70 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e92e00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c112ca0 con 0x7f2e8c06a800 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e92e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8c112fa0 con 0x7f2e8c06b1a0 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e8be00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06a800 0x7f2e8c0742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e8be00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06a800 0x7f2e8c0742c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:36670/0 (socket says 172.21.15.146:36670) 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e8be00700 1 -- 172.21.15.146:0/2456427989 learned_addr learned my addr 172.21.15.146:0/2456427989 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:42.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e90e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c108b70 0x7f2e8c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.993+0000 7f2e8b400700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e8c06b1a0 0x7f2e8c06d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e8be00700 1 -- 172.21.15.146:0/2456427989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e8c06b1a0 msgr2=0x7f2e8c06d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e8be00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e8c06b1a0 0x7f2e8c06d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e8be00700 1 -- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c108b70 msgr2=0x7f2e8c06d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e8be00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c108b70 0x7f2e8c06d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e8be00700 1 -- 172.21.15.146:0/2456427989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c06ded0 con 0x7f2e8c06a800 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e90e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c108b70 0x7f2e8c06d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e8be00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06a800 0x7f2e8c0742c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00e9a0 tx=0x7f2e7c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e88c00700 1 -- 172.21.15.146:0/2456427989 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c00cbd0 con 0x7f2e8c06a800 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e88c00700 1 -- 172.21.15.146:0/2456427989 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e7c00cd70 con 0x7f2e8c06a800 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e8c06e160 con 0x7f2e8c06a800 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e88c00700 1 -- 172.21.15.146:0/2456427989 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0187c0 con 0x7f2e8c06a800 2024-04-24T08:19:42.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.994+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e8c072080 con 0x7f2e8c06a800 2024-04-24T08:19:42.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.996+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e8c1041e0 con 0x7f2e8c06a800 2024-04-24T08:19:42.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:42.998+0000 7f2e88c00700 1 -- 172.21.15.146:0/2456427989 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2e7c010430 con 0x7f2e8c06a800 2024-04-24T08:19:43.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.000+0000 7f2e88c00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e700a3cd0 0x7f2e700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:43.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.000+0000 7f2e88c00700 1 -- 172.21.15.146:0/2456427989 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f2e7c0d08c0 con 0x7f2e8c06a800 2024-04-24T08:19:43.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.001+0000 7f2e8b400700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e700a3cd0 0x7f2e700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:43.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.002+0000 7f2e8b400700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e700a3cd0 0x7f2e700a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006010 tx=0x7f2e78005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:43.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.004+0000 7f2e88c00700 1 -- 172.21.15.146:0/2456427989 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e7c053ff0 con 0x7f2e8c06a800 2024-04-24T08:19:43.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.239+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2e8c072ab0 con 0x7f2e8c06a800 2024-04-24T08:19:43.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.240+0000 7f2e88c00700 1 -- 172.21.15.146:0/2456427989 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2e7c09d100 con 0x7f2e8c06a800 2024-04-24T08:19:43.240 INFO:teuthology.orchestra.run.smithi146.stdout:146028888128 2024-04-24T08:19:43.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.242+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e700a3cd0 msgr2=0x7f2e700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:43.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.242+0000 7f2e92e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e700a3cd0 0x7f2e700a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006010 tx=0x7f2e78005ea0 comp rx=0 tx=0).stop 2024-04-24T08:19:43.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06a800 msgr2=0x7f2e8c0742c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:43.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06a800 0x7f2e8c0742c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00e9a0 tx=0x7f2e7c00ed60 comp rx=0 tx=0).stop 2024-04-24T08:19:43.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 shutdown_connections 2024-04-24T08:19:43.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2e700a3cd0 0x7f2e700a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:43.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e8c108b70 0x7f2e8c06d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:43.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e8c06a800 0x7f2e8c0742c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:43.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 --2- 172.21.15.146:0/2456427989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e8c06b1a0 0x7f2e8c06d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:43.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 >> 172.21.15.146:0/2456427989 conn(0x7f2e8c0789a0 msgr2=0x7f2e8c0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:43.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 shutdown_connections 2024-04-24T08:19:43.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:43.243+0000 7f2e92e00700 1 -- 172.21.15.146:0/2456427989 wait complete. 2024-04-24T08:19:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:43 smithi177 ceph-mon[33046]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:43 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2456427989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T08:19:43.842 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:43 smithi146 ceph-mon[27805]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:43.842 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:43 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2456427989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T08:19:43.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:43 smithi153 ceph-mon[32746]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:43.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:43 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2456427989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T08:19:44.482 INFO:teuthology.orchestra.run.smithi146.stdout:111669149763 2024-04-24T08:19:44.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-24T08:19:45.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:45 smithi146 ceph-mon[27805]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:45 smithi177 ceph-mon[33046]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:45.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:45 smithi153 ceph-mon[32746]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:47 smithi177 ceph-mon[33046]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:19:47.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:47 smithi153 ceph-mon[32746]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:19:47.980 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:47 smithi146 ceph-mon[27805]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:19:48.011 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:48.953 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:48 smithi146 ceph-mon[27805]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:48.954 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:48 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:19:48.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:48 smithi153 ceph-mon[32746]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:48.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:48 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:19:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:49.015 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888124 got 146028888128 for osd.4 2024-04-24T08:19:49.015 DEBUG:teuthology.parallel:result is None 2024-04-24T08:19:49.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:48 smithi177 ceph-mon[33046]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:49.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:48 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:19:49.898 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 -- 172.21.15.146:0/280878158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0628074d20 msgr2=0x7f0628075100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 --2- 172.21.15.146:0/280878158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0628074d20 0x7f0628075100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0618009990 tx=0x7f0618009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 -- 172.21.15.146:0/280878158 shutdown_connections 2024-04-24T08:19:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 --2- 172.21.15.146:0/280878158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f062810f550 0x7f0628111940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 --2- 172.21.15.146:0/280878158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0628074d20 0x7f0628075100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 --2- 172.21.15.146:0/280878158 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628075640 0x7f062810f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 -- 172.21.15.146:0/280878158 >> 172.21.15.146:0/280878158 conn(0x7f0628079c50 msgr2=0x7f062807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 -- 172.21.15.146:0/280878158 shutdown_connections 2024-04-24T08:19:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.520+0000 7f062da00700 1 -- 172.21.15.146:0/280878158 wait complete. 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.521+0000 7f062da00700 1 Processor -- start 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.521+0000 7f062da00700 1 -- start start 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f062da00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628074d20 0x7f062819b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f062da00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0628075640 0x7f062819ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f062da00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f062810f550 0x7f062819fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f062da00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f06281150c0 con 0x7f0628074d20 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f062da00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0628114f40 con 0x7f0628075640 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f062da00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0628115240 con 0x7f062810f550 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f0626a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628074d20 0x7f062819b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f0626a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628074d20 0x7f062819b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35664/0 (socket says 172.21.15.146:35664) 2024-04-24T08:19:50.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f0626a00700 1 -- 172.21.15.146:0/3926324795 learned_addr learned my addr 172.21.15.146:0/3926324795 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:50.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.522+0000 7f0627400700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f062810f550 0x7f062819fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f0627400700 1 -- 172.21.15.146:0/3926324795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0628075640 msgr2=0x7f062819ba10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f0627400700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0628075640 0x7f062819ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f0627400700 1 -- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628074d20 msgr2=0x7f062819b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f0627400700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628074d20 0x7f062819b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f0627400700 1 -- 172.21.15.146:0/3926324795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06281a0490 con 0x7f062810f550 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f0626a00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628074d20 0x7f062819b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f0627400700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f062810f550 0x7f062819fd50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f062000ef90 tx=0x7f062000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f061f400700 1 -- 172.21.15.146:0/3926324795 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06200092e0 con 0x7f062810f550 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f061f400700 1 -- 172.21.15.146:0/3926324795 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0620004bb0 con 0x7f062810f550 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06281a0720 con 0x7f062810f550 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f061f400700 1 -- 172.21.15.146:0/3926324795 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620004280 con 0x7f062810f550 2024-04-24T08:19:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.523+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06281a8000 con 0x7f062810f550 2024-04-24T08:19:50.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.525+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06280697e0 con 0x7f062810f550 2024-04-24T08:19:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.526+0000 7f061f400700 1 -- 172.21.15.146:0/3926324795 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f062001b030 con 0x7f062810f550 2024-04-24T08:19:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.528+0000 7f061f400700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f060c0a3cd0 0x7f060c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.528+0000 7f061f400700 1 -- 172.21.15.146:0/3926324795 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f06200d01a0 con 0x7f062810f550 2024-04-24T08:19:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.528+0000 7f0626a00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f060c0a3cd0 0x7f060c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.528+0000 7f0626a00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f060c0a3cd0 0x7f060c0a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0618006010 tx=0x7f061800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.532+0000 7f061f400700 1 -- 172.21.15.146:0/3926324795 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0620022020 con 0x7f062810f550 2024-04-24T08:19:50.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.770+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f06281a81d0 con 0x7f062810f550 2024-04-24T08:19:50.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.771+0000 7f061f400700 1 -- 172.21.15.146:0/3926324795 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0620010040 con 0x7f062810f550 2024-04-24T08:19:50.772 INFO:teuthology.orchestra.run.smithi146.stdout:171798691899 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f060c0a3cd0 msgr2=0x7f060c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f060c0a3cd0 0x7f060c0a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0618006010 tx=0x7f061800b260 comp rx=0 tx=0).stop 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f062810f550 msgr2=0x7f062819fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f062810f550 0x7f062819fd50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f062000ef90 tx=0x7f062000c3a0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 shutdown_connections 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f062810f550 0x7f062819fd50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f060c0a3cd0 0x7f060c0a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0628074d20 0x7f062819b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 --2- 172.21.15.146:0/3926324795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0628075640 0x7f062819ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 >> 172.21.15.146:0/3926324795 conn(0x7f0628079c50 msgr2=0x7f06280ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 shutdown_connections 2024-04-24T08:19:50.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:50.774+0000 7f062da00700 1 -- 172.21.15.146:0/3926324795 wait complete. 2024-04-24T08:19:50.938 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:51.178 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:51.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:51 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:51 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:51 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:51 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:51 smithi153 ceph-mon[32746]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:51.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:51 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3926324795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T08:19:51.466 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:51 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.466 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:51 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.466 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:51 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.466 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:51 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.466 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:51 smithi146 ceph-mon[27805]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:51.466 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:51 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3926324795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T08:19:51.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:51 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:51 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:51 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:51 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:19:51.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:51 smithi177 ceph-mon[33046]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:51.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:51 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3926324795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T08:19:51.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.464+0000 7fb5fea00700 1 -- 172.21.15.146:0/1809505498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f8075640 msgr2=0x7fb5f810efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.464+0000 7fb5fea00700 1 --2- 172.21.15.146:0/1809505498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f8075640 0x7fb5f810efb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009990 tx=0x7fb5e0009ca0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.465+0000 7fb5fea00700 1 -- 172.21.15.146:0/1809505498 shutdown_connections 2024-04-24T08:19:51.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.465+0000 7fb5fea00700 1 --2- 172.21.15.146:0/1809505498 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5f810f4f0 0x7fb5f81118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.465+0000 7fb5fea00700 1 --2- 172.21.15.146:0/1809505498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 0x7fb5f8075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.465+0000 7fb5fea00700 1 --2- 172.21.15.146:0/1809505498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f8075640 0x7fb5f810efb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.465+0000 7fb5fea00700 1 -- 172.21.15.146:0/1809505498 >> 172.21.15.146:0/1809505498 conn(0x7fb5f8079c50 msgr2=0x7fb5f807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.465+0000 7fb5fea00700 1 -- 172.21.15.146:0/1809505498 shutdown_connections 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.465+0000 7fb5fea00700 1 -- 172.21.15.146:0/1809505498 wait complete. 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.466+0000 7fb5fea00700 1 Processor -- start 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.466+0000 7fb5fea00700 1 -- start start 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.466+0000 7fb5fea00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 0x7fb5f819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.466+0000 7fb5fea00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5f8075640 0x7fb5f819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.466+0000 7fb5fea00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f810f4f0 0x7fb5f819fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.466+0000 7fb5fea00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f8115020 con 0x7fb5f8075640 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.466+0000 7fb5fea00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f8114ea0 con 0x7fb5f810f4f0 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5fea00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f81151a0 con 0x7fb5f8074d20 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5efe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5f8075640 0x7fb5f819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5f7e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 0x7fb5f819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5fca00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f810f4f0 0x7fb5f819fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5efe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5f8075640 0x7fb5f819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46350/0 (socket says 172.21.15.146:46350) 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5fca00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f810f4f0 0x7fb5f819fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:56236/0 (socket says 172.21.15.146:56236) 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5fca00700 1 -- 172.21.15.146:0/2687574808 learned_addr learned my addr 172.21.15.146:0/2687574808 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5f7e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 0x7fb5f819b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39534/0 (socket says 172.21.15.146:39534) 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5f7e00700 1 -- 172.21.15.146:0/2687574808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f810f4f0 msgr2=0x7fb5f819fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5f7e00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f810f4f0 0x7fb5f819fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5f7e00700 1 -- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5f8075640 msgr2=0x7fb5f819ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5f7e00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5f8075640 0x7fb5f819ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.467+0000 7fb5f7e00700 1 -- 172.21.15.146:0/2687574808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f81a0500 con 0x7fb5f8074d20 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.468+0000 7fb5f7e00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 0x7fb5f819b4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800e9f0 tx=0x7fb5e800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.468+0000 7fb5f5600700 1 -- 172.21.15.146:0/2687574808 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e800c9c0 con 0x7fb5f8074d20 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.468+0000 7fb5f5600700 1 -- 172.21.15.146:0/2687574808 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5e800cb60 con 0x7fb5f8074d20 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.468+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f81a07f0 con 0x7fb5f8074d20 2024-04-24T08:19:51.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.468+0000 7fb5f5600700 1 -- 172.21.15.146:0/2687574808 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8018680 con 0x7fb5f8074d20 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.468+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f81a8070 con 0x7fb5f8074d20 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.469+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f80697e0 con 0x7fb5f8074d20 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.471+0000 7fb5f5600700 1 -- 172.21.15.146:0/2687574808 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb5e8010430 con 0x7fb5f8074d20 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.473+0000 7fb5f5600700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb5dc0a3cd0 0x7fb5dc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.473+0000 7fb5f5600700 1 -- 172.21.15.146:0/2687574808 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb5e80d06e0 con 0x7fb5f8074d20 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.473+0000 7fb5efe00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb5dc0a3cd0 0x7fb5dc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.474+0000 7fb5efe00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb5dc0a3cd0 0x7fb5dc0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8111210 tx=0x7fb5e000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.476+0000 7fb5f5600700 1 -- 172.21.15.146:0/2687574808 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5e809cf20 con 0x7fb5f8074d20 2024-04-24T08:19:51.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.712+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb5f802dfe0 con 0x7fb5f8074d20 2024-04-24T08:19:51.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.712+0000 7fb5f5600700 1 -- 172.21.15.146:0/2687574808 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb5e809c820 con 0x7fb5f8074d20 2024-04-24T08:19:51.713 INFO:teuthology.orchestra.run.smithi146.stdout:55834574930 2024-04-24T08:19:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb5dc0a3cd0 msgr2=0x7fb5dc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb5dc0a3cd0 0x7fb5dc0a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8111210 tx=0x7fb5e000b260 comp rx=0 tx=0).stop 2024-04-24T08:19:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 msgr2=0x7fb5f819b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 0x7fb5f819b4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800e9f0 tx=0x7fb5e800ed00 comp rx=0 tx=0).stop 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 shutdown_connections 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5f810f4f0 0x7fb5f819fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5f8074d20 0x7fb5f819b4f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb5dc0a3cd0 0x7fb5dc0a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 --2- 172.21.15.146:0/2687574808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5f8075640 0x7fb5f819ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.715+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 >> 172.21.15.146:0/2687574808 conn(0x7fb5f8079c50 msgr2=0x7fb5f80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.716+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 shutdown_connections 2024-04-24T08:19:51.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:51.716+0000 7fb5fea00700 1 -- 172.21.15.146:0/2687574808 wait complete. 2024-04-24T08:19:51.848 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:19:52.641 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:52 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2687574808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T08:19:52.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:52 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2687574808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T08:19:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:52 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2687574808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T08:19:53.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:53 smithi146 ceph-mon[27805]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:19:53.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:53 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1406263752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T08:19:53.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:53 smithi153 ceph-mon[32746]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:19:53.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:53 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1406263752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T08:19:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:53 smithi177 ceph-mon[33046]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:19:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:53 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1406263752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T08:19:54.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 -- 172.21.15.146:0/2191586145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae5810f4f0 msgr2=0x7fae581118e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 --2- 172.21.15.146:0/2191586145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae5810f4f0 0x7fae581118e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fae50009f90 tx=0x7fae5000b330 comp rx=0 tx=0).stop 2024-04-24T08:19:54.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 -- 172.21.15.146:0/2191586145 shutdown_connections 2024-04-24T08:19:54.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 --2- 172.21.15.146:0/2191586145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae5810f4f0 0x7fae581118e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 --2- 172.21.15.146:0/2191586145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae58074d20 0x7fae58075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 --2- 172.21.15.146:0/2191586145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae58075640 0x7fae5810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 -- 172.21.15.146:0/2191586145 >> 172.21.15.146:0/2191586145 conn(0x7fae58079c50 msgr2=0x7fae5807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.806+0000 7fae5ee00700 1 -- 172.21.15.146:0/2191586145 shutdown_connections 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.807+0000 7fae5ee00700 1 -- 172.21.15.146:0/2191586145 wait complete. 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 Processor -- start 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 -- start start 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58074d20 0x7fae5819b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae58075640 0x7fae5819b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5810f4f0 0x7fae5819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fae58114fd0 con 0x7fae58075640 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae58114e50 con 0x7fae5810f4f0 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.808+0000 7fae5ee00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae58115150 con 0x7fae58074d20 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58074d20 0x7fae5819b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae58075640 0x7fae5819b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58074d20 0x7fae5819b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39548/0 (socket says 172.21.15.146:39548) 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 -- 172.21.15.146:0/1406263752 learned_addr learned my addr 172.21.15.146:0/1406263752 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae5ce00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5810f4f0 0x7fae5819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 -- 172.21.15.146:0/1406263752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5810f4f0 msgr2=0x7fae5819fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5810f4f0 0x7fae5819fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 -- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae58075640 msgr2=0x7fae5819b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae58075640 0x7fae5819b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 -- 172.21.15.146:0/1406263752 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae581a0450 con 0x7fae58074d20 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.809+0000 7fae57e00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58074d20 0x7fae5819b440 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae4800e9a0 tx=0x7fae4800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.810+0000 7fae54c00700 1 -- 172.21.15.146:0/1406263752 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4800ca10 con 0x7fae58074d20 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.810+0000 7fae54c00700 1 -- 172.21.15.146:0/1406263752 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae4800cbb0 con 0x7fae58074d20 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.810+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae581a0740 con 0x7fae58074d20 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.810+0000 7fae54c00700 1 -- 172.21.15.146:0/1406263752 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48018680 con 0x7fae58074d20 2024-04-24T08:19:54.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.810+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae581a7fc0 con 0x7fae58074d20 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.811+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae580697e0 con 0x7fae58074d20 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.813+0000 7fae54c00700 1 -- 172.21.15.146:0/1406263752 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fae48010430 con 0x7fae58074d20 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.814+0000 7fae54c00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fae3c0a3d50 0x7fae3c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.815+0000 7fae54c00700 1 -- 172.21.15.146:0/1406263752 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fae480d0710 con 0x7fae58074d20 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.815+0000 7fae57400700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fae3c0a3d50 0x7fae3c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.815+0000 7fae57400700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fae3c0a3d50 0x7fae3c0a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae44009750 tx=0x7fae44009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:52.817+0000 7fae54c00700 1 -- 172.21.15.146:0/1406263752 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae4809cf50 con 0x7fae58074d20 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.051+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fae581a8230 con 0x7fae58074d20 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.052+0000 7fae54c00700 1 -- 172.21.15.146:0/1406263752 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fae4809c850 con 0x7fae58074d20 2024-04-24T08:19:54.698 INFO:teuthology.orchestra.run.smithi146.stdout:356482285594 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.054+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fae3c0a3d50 msgr2=0x7fae3c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.054+0000 7fae5ee00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fae3c0a3d50 0x7fae3c0a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fae44009750 tx=0x7fae44009310 comp rx=0 tx=0).stop 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58074d20 msgr2=0x7fae5819b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58074d20 0x7fae5819b440 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fae4800e9a0 tx=0x7fae4800ecb0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 shutdown_connections 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5810f4f0 0x7fae5819fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58074d20 0x7fae5819b440 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fae3c0a3d50 0x7fae3c0a6210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 --2- 172.21.15.146:0/1406263752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae58075640 0x7fae5819b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 >> 172.21.15.146:0/1406263752 conn(0x7fae58079c50 msgr2=0x7fae580ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 shutdown_connections 2024-04-24T08:19:54.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:53.055+0000 7fae5ee00700 1 -- 172.21.15.146:0/1406263752 wait complete. 2024-04-24T08:19:55.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:55 smithi146 ceph-mon[27805]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:55.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:55 smithi177 ceph-mon[33046]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:55 smithi153 ceph-mon[32746]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:56.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 -- 172.21.15.146:0/2460385056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc81058f0 msgr2=0x7ffbc8109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 --2- 172.21.15.146:0/2460385056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc81058f0 0x7ffbc8109940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009920 tx=0x7ffbb4009c30 comp rx=0 tx=0).stop 2024-04-24T08:19:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 -- 172.21.15.146:0/2460385056 shutdown_connections 2024-04-24T08:19:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 --2- 172.21.15.146:0/2460385056 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc810a020 0x7ffbc8111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 --2- 172.21.15.146:0/2460385056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbc8104f40 0x7ffbc8105320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 --2- 172.21.15.146:0/2460385056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc81058f0 0x7ffbc8109940 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 -- 172.21.15.146:0/2460385056 >> 172.21.15.146:0/2460385056 conn(0x7ffbc81007d0 msgr2=0x7ffbc8102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 -- 172.21.15.146:0/2460385056 shutdown_connections 2024-04-24T08:19:56.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.227+0000 7ffbcce00700 1 -- 172.21.15.146:0/2460385056 wait complete. 2024-04-24T08:19:56.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.229+0000 7ffbcce00700 1 Processor -- start 2024-04-24T08:19:56.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbcce00700 1 -- start start 2024-04-24T08:19:56.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbcce00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbc8104f40 0x7ffbc819b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbcce00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc81058f0 0x7ffbc819bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbcce00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc810a020 0x7ffbc819fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbcce00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc8115050 con 0x7ffbc8104f40 2024-04-24T08:19:56.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbcce00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc8114ed0 con 0x7ffbc81058f0 2024-04-24T08:19:56.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbcce00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc81151d0 con 0x7ffbc810a020 2024-04-24T08:19:56.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbc5400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc81058f0 0x7ffbc819bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbc5400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc81058f0 0x7ffbc819bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:56280/0 (socket says 172.21.15.146:56280) 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbc5400700 1 -- 172.21.15.146:0/3276919609 learned_addr learned my addr 172.21.15.146:0/3276919609 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.230+0000 7ffbc5e00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbc8104f40 0x7ffbc819b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc6800700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc810a020 0x7ffbc819fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc5400700 1 -- 172.21.15.146:0/3276919609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc810a020 msgr2=0x7ffbc819fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc5400700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc810a020 0x7ffbc819fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc5400700 1 -- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbc8104f40 msgr2=0x7ffbc819b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc5400700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbc8104f40 0x7ffbc819b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc5400700 1 -- 172.21.15.146:0/3276919609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc81a05a0 con 0x7ffbc81058f0 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc6800700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc810a020 0x7ffbc819fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.231+0000 7ffbc5400700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc81058f0 0x7ffbc819bb20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400a010 tx=0x7ffbb400fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.232+0000 7ffbbea00700 1 -- 172.21.15.146:0/3276919609 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40052b0 con 0x7ffbc81058f0 2024-04-24T08:19:56.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.232+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc81a0830 con 0x7ffbc81058f0 2024-04-24T08:19:56.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.232+0000 7ffbbea00700 1 -- 172.21.15.146:0/3276919609 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbb400ba40 con 0x7ffbc81058f0 2024-04-24T08:19:56.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.232+0000 7ffbbea00700 1 -- 172.21.15.146:0/3276919609 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4003b60 con 0x7ffbc81058f0 2024-04-24T08:19:56.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.232+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc81a8110 con 0x7ffbc81058f0 2024-04-24T08:19:56.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.233+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc8106420 con 0x7ffbc81058f0 2024-04-24T08:19:56.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.235+0000 7ffbbea00700 1 -- 172.21.15.146:0/3276919609 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ffbb4005450 con 0x7ffbc81058f0 2024-04-24T08:19:56.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.236+0000 7ffbbea00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffbac0a3cd0 0x7ffbac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.236+0000 7ffbbea00700 1 -- 172.21.15.146:0/3276919609 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ffbb40d1800 con 0x7ffbc81058f0 2024-04-24T08:19:56.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.236+0000 7ffbc5e00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffbac0a3cd0 0x7ffbac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.237+0000 7ffbc5e00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffbac0a3cd0 0x7ffbac0a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009750 tx=0x7ffbb8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.239+0000 7ffbbea00700 1 -- 172.21.15.146:0/3276919609 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffbb4013030 con 0x7ffbc81058f0 2024-04-24T08:19:56.350 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691895 got 171798691899 for osd.5 2024-04-24T08:19:56.350 DEBUG:teuthology.parallel:result is None 2024-04-24T08:19:56.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.510+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ffbc810d290 con 0x7ffbc81058f0 2024-04-24T08:19:56.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.510+0000 7ffbbea00700 1 -- 172.21.15.146:0/3276919609 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ffbb409e0d0 con 0x7ffbc81058f0 2024-04-24T08:19:56.514 INFO:teuthology.orchestra.run.smithi146.stdout:322122547231 2024-04-24T08:19:56.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffbac0a3cd0 msgr2=0x7ffbac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffbac0a3cd0 0x7ffbac0a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009750 tx=0x7ffbb8009310 comp rx=0 tx=0).stop 2024-04-24T08:19:56.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc81058f0 msgr2=0x7ffbc819bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc81058f0 0x7ffbc819bb20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400a010 tx=0x7ffbb400fb20 comp rx=0 tx=0).stop 2024-04-24T08:19:56.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 shutdown_connections 2024-04-24T08:19:56.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffbc810a020 0x7ffbc819fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ffbac0a3cd0 0x7ffbac0a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbc8104f40 0x7ffbc819b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 --2- 172.21.15.146:0/3276919609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffbc81058f0 0x7ffbc819bb20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 >> 172.21.15.146:0/3276919609 conn(0x7ffbc81007d0 msgr2=0x7ffbc8101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.513+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 shutdown_connections 2024-04-24T08:19:56.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:56.514+0000 7ffbcce00700 1 -- 172.21.15.146:0/3276919609 wait complete. 2024-04-24T08:19:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3276919609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T08:19:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3276919609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T08:19:56.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3276919609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T08:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 -- 172.21.15.146:0/4030139881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 msgr2=0x7f89f810dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 --2- 172.21.15.146:0/4030139881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f810dd80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89e40099c0 tx=0x7f89e4009cd0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 -- 172.21.15.146:0/4030139881 shutdown_connections 2024-04-24T08:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 --2- 172.21.15.146:0/4030139881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89f810e460 0x7f89f8116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 --2- 172.21.15.146:0/4030139881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89f8109380 0x7f89f8109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 --2- 172.21.15.146:0/4030139881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f810dd80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 -- 172.21.15.146:0/4030139881 >> 172.21.15.146:0/4030139881 conn(0x7f89f80762c0 msgr2=0x7f89f8107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 -- 172.21.15.146:0/4030139881 shutdown_connections 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.695+0000 7f8a00400700 1 -- 172.21.15.146:0/4030139881 wait complete. 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.696+0000 7f8a00400700 1 Processor -- start 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.696+0000 7f8a00400700 1 -- start start 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f8a00400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89f8109380 0x7f89f81a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f8a00400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f81a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f8a00400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89f810e460 0x7f89f819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f8a00400700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f89f8119490 con 0x7f89f8109380 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f8a00400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f89f8119310 con 0x7f89f810e460 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f8a00400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f89f8119610 con 0x7f89f8109d30 2024-04-24T08:19:57.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f89fd000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f81a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f89fd000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f81a5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:39592/0 (socket says 172.21.15.146:39592) 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f89fda00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89f8109380 0x7f89f81a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f89fd000700 1 -- 172.21.15.146:0/1122791450 learned_addr learned my addr 172.21.15.146:0/1122791450 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.697+0000 7f89fe400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89f810e460 0x7f89f819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fd000700 1 -- 172.21.15.146:0/1122791450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89f810e460 msgr2=0x7f89f819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fd000700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89f810e460 0x7f89f819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fd000700 1 -- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89f8109380 msgr2=0x7f89f81a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fd000700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89f8109380 0x7f89f81a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fd000700 1 -- 172.21.15.146:0/1122791450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f81a0050 con 0x7f89f8109d30 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fe400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89f810e460 0x7f89f819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fda00700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89f8109380 0x7f89f81a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89fd000700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f81a5be0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f89e400f690 tx=0x7f89e4005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:57.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89f6000700 1 -- 172.21.15.146:0/1122791450 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4027070 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89f6000700 1 -- 172.21.15.146:0/1122791450 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e4003680 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f89f6000700 1 -- 172.21.15.146:0/1122791450 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e40173f0 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f81a02e0 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.698+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f81ac450 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.700+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89f810a860 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.701+0000 7f89f6000700 1 -- 172.21.15.146:0/1122791450 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f89e402a070 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.703+0000 7f89f6000700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89e00a3c40 0x7f89e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.703+0000 7f89f6000700 1 -- 172.21.15.146:0/1122791450 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f89e40d0fe0 con 0x7f89f8109d30 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.703+0000 7f89fda00700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89e00a3c40 0x7f89e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.704+0000 7f89fda00700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89e00a3c40 0x7f89e00a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f89ec011490 tx=0x7f89ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:57.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.707+0000 7f89f6000700 1 -- 172.21.15.146:0/1122791450 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89e405eff0 con 0x7f89f8109d30 2024-04-24T08:19:57.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:57 smithi177 ceph-mon[33046]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:19:57.940 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:57 smithi146 ceph-mon[27805]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:19:57.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.938+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f89f802dfe0 con 0x7f89f8109d30 2024-04-24T08:19:57.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.939+0000 7f89f6000700 1 -- 172.21.15.146:0/1122791450 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f89e4013070 con 0x7f89f8109d30 2024-04-24T08:19:57.941 INFO:teuthology.orchestra.run.smithi146.stdout:77309411405 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89e00a3c40 msgr2=0x7f89e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89e00a3c40 0x7f89e00a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f89ec011490 tx=0x7f89ec009260 comp rx=0 tx=0).stop 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 msgr2=0x7f89f81a5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f81a5be0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f89e400f690 tx=0x7f89e4005f10 comp rx=0 tx=0).stop 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 shutdown_connections 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f89f810e460 0x7f89f819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f89e00a3c40 0x7f89e00a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89f8109380 0x7f89f81a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 --2- 172.21.15.146:0/1122791450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89f8109d30 0x7f89f81a5be0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 >> 172.21.15.146:0/1122791450 conn(0x7f89f80762c0 msgr2=0x7f89f8076840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 shutdown_connections 2024-04-24T08:19:57.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:19:57.942+0000 7f8a00400700 1 -- 172.21.15.146:0/1122791450 wait complete. 2024-04-24T08:19:57.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:57 smithi153 ceph-mon[32746]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:19:58.521 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574930 for osd.1 2024-04-24T08:19:58.521 DEBUG:teuthology.parallel:result is None 2024-04-24T08:19:58.694 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:58 smithi146 ceph-mon[27805]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T08:19:58.694 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:58 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1122791450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T08:19:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:58 smithi177 ceph-mon[33046]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T08:19:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:58 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1122791450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T08:19:58.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:58 smithi153 ceph-mon[32746]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T08:19:58.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:58 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1122791450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T08:19:59.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411405 for osd.2 2024-04-24T08:19:59.441 DEBUG:teuthology.parallel:result is None 2024-04-24T08:19:59.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:19:59 smithi177 ceph-mon[33046]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-24T08:19:59.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:19:59 smithi153 ceph-mon[32746]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-24T08:19:59.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:19:59 smithi146 ceph-mon[27805]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-24T08:20:00.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285590 got 356482285594 for osd.11 2024-04-24T08:20:00.587 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:00.951 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:00 smithi146 ceph-mon[27805]: overall HEALTH_OK 2024-04-24T08:20:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:00 smithi153 ceph-mon[32746]: overall HEALTH_OK 2024-04-24T08:20:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:00 smithi177 ceph-mon[33046]: overall HEALTH_OK 2024-04-24T08:20:01.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:01 smithi146 ceph-mon[27805]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:01.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:01 smithi153 ceph-mon[32746]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:02.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:01 smithi177 ceph-mon[33046]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:02.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547226 got 322122547231 for osd.10 2024-04-24T08:20:02.482 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:03.944 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:03 smithi146 ceph-mon[27805]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-24T08:20:03.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:03 smithi153 ceph-mon[32746]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-24T08:20:04.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:03 smithi177 ceph-mon[33046]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 -- 172.21.15.146:0/3026303290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06a800 msgr2=0x7f97bc06ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 --2- 172.21.15.146:0/3026303290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06a800 0x7f97bc06ac60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009990 tx=0x7f97a8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 -- 172.21.15.146:0/3026303290 shutdown_connections 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 --2- 172.21.15.146:0/3026303290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc06b1a0 0x7f97bc10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 --2- 172.21.15.146:0/3026303290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc108b70 0x7f97bc108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 --2- 172.21.15.146:0/3026303290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06a800 0x7f97bc06ac60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 -- 172.21.15.146:0/3026303290 >> 172.21.15.146:0/3026303290 conn(0x7f97bc0789a0 msgr2=0x7f97bc078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 -- 172.21.15.146:0/3026303290 shutdown_connections 2024-04-24T08:20:04.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.113+0000 7f97c2a00700 1 -- 172.21.15.146:0/3026303290 wait complete. 2024-04-24T08:20:04.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 Processor -- start 2024-04-24T08:20:04.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 -- start start 2024-04-24T08:20:04.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc06a800 0x7f97bc1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06b1a0 0x7f97bc1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc108b70 0x7f97bc19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc112e60 con 0x7f97bc06a800 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc112ce0 con 0x7f97bc06b1a0 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.115+0000 7f97c2a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc112fe0 con 0x7f97bc108b70 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc06a800 0x7f97bc1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc06a800 0x7f97bc1992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47208/0 (socket says 172.21.15.146:47208) 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 -- 172.21.15.146:0/501487074 learned_addr learned my addr 172.21.15.146:0/501487074 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bb400700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06b1a0 0x7f97bc1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97c0a00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc108b70 0x7f97bc19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 -- 172.21.15.146:0/501487074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc108b70 msgr2=0x7f97bc19db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc108b70 0x7f97bc19db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 -- 172.21.15.146:0/501487074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06b1a0 msgr2=0x7f97bc1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06b1a0 0x7f97bc1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 -- 172.21.15.146:0/501487074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97bc19e2c0 con 0x7f97bc06a800 2024-04-24T08:20:04.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bb400700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06b1a0 0x7f97bc1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.116+0000 7f97bbe00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc06a800 0x7f97bc1992b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e9a0 tx=0x7f97ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.117+0000 7f97b8c00700 1 -- 172.21.15.146:0/501487074 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00ca10 con 0x7f97bc06a800 2024-04-24T08:20:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.117+0000 7f97b8c00700 1 -- 172.21.15.146:0/501487074 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97ac00cbb0 con 0x7f97bc06a800 2024-04-24T08:20:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.117+0000 7f97b8c00700 1 -- 172.21.15.146:0/501487074 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac018680 con 0x7f97bc06a800 2024-04-24T08:20:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.117+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97bc19e5b0 con 0x7f97bc06a800 2024-04-24T08:20:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.117+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97bc1a5e30 con 0x7f97bc06a800 2024-04-24T08:20:04.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.118+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97bc0697e0 con 0x7f97bc06a800 2024-04-24T08:20:04.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.120+0000 7f97b8c00700 1 -- 172.21.15.146:0/501487074 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f97ac010430 con 0x7f97bc06a800 2024-04-24T08:20:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.122+0000 7f97b8c00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97a00a3c40 0x7f97a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.123+0000 7f97b8c00700 1 -- 172.21.15.146:0/501487074 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f97ac0d05f0 con 0x7f97bc06a800 2024-04-24T08:20:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.123+0000 7f97bb400700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97a00a3c40 0x7f97a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.123+0000 7f97bb400700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97a00a3c40 0x7f97a00a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f97bc078f10 tx=0x7f97a800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.127+0000 7f97b8c00700 1 -- 172.21.15.146:0/501487074 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97ac09cf90 con 0x7f97bc06a800 2024-04-24T08:20:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.364+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f97bc02dfe0 con 0x7f97bc06a800 2024-04-24T08:20:04.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.364+0000 7f97b8c00700 1 -- 172.21.15.146:0/501487074 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f97ac085e20 con 0x7f97bc06a800 2024-04-24T08:20:04.366 INFO:teuthology.orchestra.run.smithi146.stdout:300647710760 2024-04-24T08:20:04.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97a00a3c40 msgr2=0x7f97a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97a00a3c40 0x7f97a00a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f97bc078f10 tx=0x7f97a800b260 comp rx=0 tx=0).stop 2024-04-24T08:20:04.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc06a800 msgr2=0x7f97bc1992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc06a800 0x7f97bc1992b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e9a0 tx=0x7f97ac00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 shutdown_connections 2024-04-24T08:20:04.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc108b70 0x7f97bc19db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97a00a3c40 0x7f97a00a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97bc06a800 0x7f97bc1992b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 --2- 172.21.15.146:0/501487074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97bc06b1a0 0x7f97bc1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 >> 172.21.15.146:0/501487074 conn(0x7f97bc0789a0 msgr2=0x7f97bc0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 shutdown_connections 2024-04-24T08:20:04.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.368+0000 7f97c2a00700 1 -- 172.21.15.146:0/501487074 wait complete. 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.351+0000 7fe6ab200700 1 -- 172.21.15.146:0/443966909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 msgr2=0x7fe6ac10f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.351+0000 7fe6ab200700 1 --2- 172.21.15.146:0/443966909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 0x7fe6ac10f520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009f90 tx=0x7fe6a400b330 comp rx=0 tx=0).stop 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.352+0000 7fe6ab200700 1 -- 172.21.15.146:0/443966909 shutdown_connections 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.352+0000 7fe6ab200700 1 --2- 172.21.15.146:0/443966909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 0x7fe6ac10f520 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.352+0000 7fe6ab200700 1 --2- 172.21.15.146:0/443966909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac103c30 0x7fe6ac104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.352+0000 7fe6ab200700 1 --2- 172.21.15.146:0/443966909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6ac104550 0x7fe6ac10ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.352+0000 7fe6ab200700 1 -- 172.21.15.146:0/443966909 >> 172.21.15.146:0/443966909 conn(0x7fe6ac0fd480 msgr2=0x7fe6ac0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.352+0000 7fe6ab200700 1 -- 172.21.15.146:0/443966909 shutdown_connections 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.352+0000 7fe6ab200700 1 -- 172.21.15.146:0/443966909 wait complete. 2024-04-24T08:20:04.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.354+0000 7fe6ab200700 1 Processor -- start 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.354+0000 7fe6ab200700 1 -- start start 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6ab200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6ac103c30 0x7fe6ac1991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6ab200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac104550 0x7fe6ac1996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6ab200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 0x7fe6ac19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6ab200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac112da0 con 0x7fe6ac104550 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6ab200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac112c20 con 0x7fe6ac103c30 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6ab200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac112f20 con 0x7fe6ac10d060 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6a9400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac104550 0x7fe6ac1996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6a9e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6ac103c30 0x7fe6ac1991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6a9400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac104550 0x7fe6ac1996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47232/0 (socket says 172.21.15.146:47232) 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6a9400700 1 -- 172.21.15.146:0/3212654458 learned_addr learned my addr 172.21.15.146:0/3212654458 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6aa800700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 0x7fe6ac19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.355+0000 7fe6aa800700 1 -- 172.21.15.146:0/3212654458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6ac103c30 msgr2=0x7fe6ac1991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.356+0000 7fe6aa800700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6ac103c30 0x7fe6ac1991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.356+0000 7fe6aa800700 1 -- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac104550 msgr2=0x7fe6ac1996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.356+0000 7fe6aa800700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac104550 0x7fe6ac1996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.356+0000 7fe6aa800700 1 -- 172.21.15.146:0/3212654458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac19e1c0 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.356+0000 7fe6a9400700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac104550 0x7fe6ac1996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.357+0000 7fe6aa800700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 0x7fe6ac19da80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4015000 tx=0x7fe6a4004f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.357+0000 7fe6a2a00700 1 -- 172.21.15.146:0/3212654458 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4021070 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.357+0000 7fe6a2a00700 1 -- 172.21.15.146:0/3212654458 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a4007ac0 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.357+0000 7fe6a2a00700 1 -- 172.21.15.146:0/3212654458 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4005040 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.357+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ac19e450 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.357+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ac1017c0 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.359+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6ac06b940 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.362+0000 7fe6a2a00700 1 -- 172.21.15.146:0/3212654458 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fe6a4024070 con 0x7fe6ac10d060 2024-04-24T08:20:04.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.363+0000 7fe6a2a00700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe6900a3cc0 0x7fe6900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.363+0000 7fe6a2a00700 1 -- 172.21.15.146:0/3212654458 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fe6a400f020 con 0x7fe6ac10d060 2024-04-24T08:20:04.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.364+0000 7fe6a9e00700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe6900a3cc0 0x7fe6900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.365+0000 7fe6a9e00700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe6900a3cc0 0x7fe6900a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe69c005fd0 tx=0x7fe69c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.367+0000 7fe6a2a00700 1 -- 172.21.15.146:0/3212654458 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6a402b020 con 0x7fe6ac10d060 2024-04-24T08:20:04.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.608+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe6ac19a510 con 0x7fe6ac10d060 2024-04-24T08:20:04.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.609+0000 7fe6a2a00700 1 -- 172.21.15.146:0/3212654458 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe6a409e0f0 con 0x7fe6ac10d060 2024-04-24T08:20:04.611 INFO:teuthology.orchestra.run.smithi146.stdout:34359738458 2024-04-24T08:20:04.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe6900a3cc0 msgr2=0x7fe6900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe6900a3cc0 0x7fe6900a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe69c005fd0 tx=0x7fe69c009310 comp rx=0 tx=0).stop 2024-04-24T08:20:04.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 msgr2=0x7fe6ac19da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 0x7fe6ac19da80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4015000 tx=0x7fe6a4004f00 comp rx=0 tx=0).stop 2024-04-24T08:20:04.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 shutdown_connections 2024-04-24T08:20:04.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6ac10d060 0x7fe6ac19da80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6ac103c30 0x7fe6ac1991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fe6900a3cc0 0x7fe6900a6180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 --2- 172.21.15.146:0/3212654458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6ac104550 0x7fe6ac1996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 >> 172.21.15.146:0/3212654458 conn(0x7fe6ac0fd480 msgr2=0x7fe6ac0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 shutdown_connections 2024-04-24T08:20:04.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.612+0000 7fe6ab200700 1 -- 172.21.15.146:0/3212654458 wait complete. 2024-04-24T08:20:04.768 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:04 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/501487074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T08:20:04.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:04 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/501487074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T08:20:05.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:04 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/501487074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T08:20:05.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 -- 172.21.15.146:0/53849656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658104f70 msgr2=0x7fa658105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 --2- 172.21.15.146:0/53849656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658104f70 0x7fa658105350 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009990 tx=0x7fa64c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 -- 172.21.15.146:0/53849656 shutdown_connections 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 --2- 172.21.15.146:0/53849656 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa65810a050 0x7fa658111c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 --2- 172.21.15.146:0/53849656 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658104f70 0x7fa658105350 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 --2- 172.21.15.146:0/53849656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658105920 0x7fa658109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 -- 172.21.15.146:0/53849656 >> 172.21.15.146:0/53849656 conn(0x7fa658100800 msgr2=0x7fa658102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 -- 172.21.15.146:0/53849656 shutdown_connections 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.766+0000 7fa660400700 1 -- 172.21.15.146:0/53849656 wait complete. 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 Processor -- start 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 -- start start 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658104f70 0x7fa65819b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658105920 0x7fa65819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa65810a050 0x7fa65819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581150a0 con 0x7fa65810a050 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa658114f20 con 0x7fa658105920 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.768+0000 7fa660400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa658115220 con 0x7fa658104f70 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65e400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa65810a050 0x7fa65819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658104f70 0x7fa65819b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658104f70 0x7fa65819b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46040/0 (socket says 172.21.15.146:46040) 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 -- 172.21.15.146:0/1871583133 learned_addr learned my addr 172.21.15.146:0/1871583133 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65d000700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658105920 0x7fa65819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 -- 172.21.15.146:0/1871583133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658105920 msgr2=0x7fa65819ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658105920 0x7fa65819ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 -- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa65810a050 msgr2=0x7fa65819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa65810a050 0x7fa65819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 -- 172.21.15.146:0/1871583133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6581a0510 con 0x7fa658104f70 2024-04-24T08:20:05.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.769+0000 7fa65da00700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658104f70 0x7fa65819b550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.770+0000 7fa656000700 1 -- 172.21.15.146:0/1871583133 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c027070 con 0x7fa658104f70 2024-04-24T08:20:05.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.770+0000 7fa656000700 1 -- 172.21.15.146:0/1871583133 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa64c00fda0 con 0x7fa658104f70 2024-04-24T08:20:05.077 INFO:teuthology.orchestra.run.smithi146.stdout:206158430264 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.770+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6581a07a0 con 0x7fa658104f70 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.770+0000 7fa656000700 1 -- 172.21.15.146:0/1871583133 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0173f0 con 0x7fa658104f70 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.770+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581a8080 con 0x7fa658104f70 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.771+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa658106450 con 0x7fa658104f70 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.773+0000 7fa656000700 1 -- 172.21.15.146:0/1871583133 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fa64c02a070 con 0x7fa658104f70 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.774+0000 7fa656000700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6400a3c40 0x7fa6400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.775+0000 7fa656000700 1 -- 172.21.15.146:0/1871583133 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fa64c0d1220 con 0x7fa658104f70 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.775+0000 7fa65d000700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6400a3c40 0x7fa6400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.775+0000 7fa65d000700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6400a3c40 0x7fa6400a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa65810b630 tx=0x7fa644009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.777+0000 7fa656000700 1 -- 172.21.15.146:0/1871583133 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa64c013070 con 0x7fa658104f70 2024-04-24T08:20:05.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.995+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa65802dfe0 con 0x7fa658104f70 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.996+0000 7fa656000700 1 -- 172.21.15.146:0/1871583133 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa64c09dc50 con 0x7fa658104f70 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.998+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6400a3c40 msgr2=0x7fa6400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.998+0000 7fa660400700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6400a3c40 0x7fa6400a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa65810b630 tx=0x7fa644009260 comp rx=0 tx=0).stop 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658104f70 msgr2=0x7fa65819b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658104f70 0x7fa65819b550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000c00 tx=0x7fa64c005590 comp rx=0 tx=0).stop 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 shutdown_connections 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa6400a3c40 0x7fa6400a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa65810a050 0x7fa65819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa658104f70 0x7fa65819b550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 --2- 172.21.15.146:0/1871583133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa658105920 0x7fa65819ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 >> 172.21.15.146:0/1871583133 conn(0x7fa658100800 msgr2=0x7fa658101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 shutdown_connections 2024-04-24T08:20:05.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:04.999+0000 7fa660400700 1 -- 172.21.15.146:0/1871583133 wait complete. 2024-04-24T08:20:05.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:05 smithi146 ceph-mon[27805]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:05.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:05 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3212654458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T08:20:05.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:05 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1871583133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T08:20:05.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:05 smithi153 ceph-mon[32746]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:05.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:05 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3212654458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T08:20:05.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:05 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1871583133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T08:20:06.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:05 smithi177 ceph-mon[33046]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:06.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:05 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3212654458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T08:20:06.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:05 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1871583133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 -- 172.21.15.146:0/57738915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec108a20 msgr2=0x7f45ec108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 --2- 172.21.15.146:0/57738915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec108a20 0x7f45ec108e00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009990 tx=0x7f45dc009ca0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 -- 172.21.15.146:0/57738915 shutdown_connections 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 --2- 172.21.15.146:0/57738915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45ec1033e0 0x7f45ec10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 --2- 172.21.15.146:0/57738915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec108a20 0x7f45ec108e00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 --2- 172.21.15.146:0/57738915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ec102a20 0x7f45ec102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 -- 172.21.15.146:0/57738915 >> 172.21.15.146:0/57738915 conn(0x7f45ec0fe570 msgr2=0x7f45ec100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 -- 172.21.15.146:0/57738915 shutdown_connections 2024-04-24T08:20:06.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.022+0000 7f45f0c00700 1 -- 172.21.15.146:0/57738915 wait complete. 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 Processor -- start 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 -- start start 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec102a20 0x7f45ec19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ec1033e0 0x7f45ec19b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45ec108a20 0x7f45ec19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f45ec112d50 con 0x7f45ec102a20 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45ec112bd0 con 0x7f45ec1033e0 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45f0c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f45ec112ed0 con 0x7f45ec108a20 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45eb400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec102a20 0x7f45ec19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45eb400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec102a20 0x7f45ec19b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47264/0 (socket says 172.21.15.146:47264) 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.024+0000 7f45eb400700 1 -- 172.21.15.146:0/3540980312 learned_addr learned my addr 172.21.15.146:0/3540980312 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45eaa00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ec1033e0 0x7f45ec19b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45ebe00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45ec108a20 0x7f45ec19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45eb400700 1 -- 172.21.15.146:0/3540980312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45ec108a20 msgr2=0x7f45ec19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45eb400700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45ec108a20 0x7f45ec19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45eb400700 1 -- 172.21.15.146:0/3540980312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ec1033e0 msgr2=0x7f45ec19b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45eb400700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ec1033e0 0x7f45ec19b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45eb400700 1 -- 172.21.15.146:0/3540980312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ec1a03b0 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45eb400700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec102a20 0x7f45ec19b3a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00bae0 tx=0x7f45dc00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.025+0000 7f45e3e00700 1 -- 172.21.15.146:0/3540980312 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45dc026070 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.026+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ec1a0640 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.026+0000 7f45e3e00700 1 -- 172.21.15.146:0/3540980312 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45dc003d70 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.026+0000 7f45e3e00700 1 -- 172.21.15.146:0/3540980312 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45dc021410 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.026+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ec1a7f20 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.027+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ec0697e0 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.029+0000 7f45e3e00700 1 -- 172.21.15.146:0/3540980312 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f45dc029070 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.030+0000 7f45e3e00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f45d00a3d40 0x7f45d00a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.030+0000 7f45e3e00700 1 -- 172.21.15.146:0/3540980312 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f45dc0d10f0 con 0x7f45ec102a20 2024-04-24T08:20:06.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.031+0000 7f45eaa00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f45d00a3d40 0x7f45d00a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.031+0000 7f45eaa00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f45d00a3d40 0x7f45d00a6200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f45ec10fc70 tx=0x7f45d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.035+0000 7f45e3e00700 1 -- 172.21.15.146:0/3540980312 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45dc05dff0 con 0x7f45ec102a20 2024-04-24T08:20:06.251 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710755 got 300647710760 for osd.9 2024-04-24T08:20:06.252 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:06.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.269+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f45ec1a80f0 con 0x7f45ec102a20 2024-04-24T08:20:06.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.270+0000 7f45e3e00700 1 -- 172.21.15.146:0/3540980312 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f45dc013070 con 0x7f45ec102a20 2024-04-24T08:20:06.272 INFO:teuthology.orchestra.run.smithi146.stdout:266287972399 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f45d00a3d40 msgr2=0x7f45d00a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f45d00a3d40 0x7f45d00a6200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f45ec10fc70 tx=0x7f45d8009260 comp rx=0 tx=0).stop 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec102a20 msgr2=0x7f45ec19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec102a20 0x7f45ec19b3a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00bae0 tx=0x7f45dc00bb10 comp rx=0 tx=0).stop 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 shutdown_connections 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f45ec108a20 0x7f45ec19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f45d00a3d40 0x7f45d00a6200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45ec102a20 0x7f45ec19b3a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 --2- 172.21.15.146:0/3540980312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45ec1033e0 0x7f45ec19b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 >> 172.21.15.146:0/3540980312 conn(0x7f45ec0fe570 msgr2=0x7f45ec10e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 shutdown_connections 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.273+0000 7f45f0c00700 1 -- 172.21.15.146:0/3540980312 wait complete. 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.468+0000 7fd824a00700 1 -- 172.21.15.146:0/305041516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 msgr2=0x7fd8201118e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.468+0000 7fd824a00700 1 --2- 172.21.15.146:0/305041516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 0x7fd8201118e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd81800b0a0 tx=0x7fd81800b3b0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.469+0000 7fd824a00700 1 -- 172.21.15.146:0/305041516 shutdown_connections 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.469+0000 7fd824a00700 1 --2- 172.21.15.146:0/305041516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 0x7fd8201118e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.469+0000 7fd824a00700 1 --2- 172.21.15.146:0/305041516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd820074d20 0x7fd820075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.469+0000 7fd824a00700 1 --2- 172.21.15.146:0/305041516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820075640 0x7fd82010efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.469+0000 7fd824a00700 1 -- 172.21.15.146:0/305041516 >> 172.21.15.146:0/305041516 conn(0x7fd820079c50 msgr2=0x7fd82007a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.469+0000 7fd824a00700 1 -- 172.21.15.146:0/305041516 shutdown_connections 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.469+0000 7fd824a00700 1 -- 172.21.15.146:0/305041516 wait complete. 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.470+0000 7fd824a00700 1 Processor -- start 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.470+0000 7fd824a00700 1 -- start start 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd824a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820074d20 0x7fd82019b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd824a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd820075640 0x7fd82019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd824a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 0x7fd82019fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd824a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd820115010 con 0x7fd820074d20 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd824a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd820114e90 con 0x7fd82010f4f0 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd824a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd820115190 con 0x7fd820075640 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820074d20 0x7fd82019b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820074d20 0x7fd82019b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47278/0 (socket says 172.21.15.146:47278) 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 -- 172.21.15.146:0/862358307 learned_addr learned my addr 172.21.15.146:0/862358307 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:06.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81e800700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 0x7fd82019fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81d400700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd820075640 0x7fd82019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 -- 172.21.15.146:0/862358307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd820075640 msgr2=0x7fd82019b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd820075640 0x7fd82019b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 -- 172.21.15.146:0/862358307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 msgr2=0x7fd82019fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 0x7fd82019fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81de00700 1 -- 172.21.15.146:0/862358307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8201a0470 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.471+0000 7fd81e800700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 0x7fd82019fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.472+0000 7fd81de00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820074d20 0x7fd82019b460 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd81000e9a0 tx=0x7fd81000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.472+0000 7fd816a00700 1 -- 172.21.15.146:0/862358307 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81000ca10 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.472+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8201a0760 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.472+0000 7fd816a00700 1 -- 172.21.15.146:0/862358307 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81000cbb0 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.472+0000 7fd816a00700 1 -- 172.21.15.146:0/862358307 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd810018680 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.472+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8201a7fe0 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.473+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8200697e0 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.475+0000 7fd816a00700 1 -- 172.21.15.146:0/862358307 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fd810010430 con 0x7fd820074d20 2024-04-24T08:20:06.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.477+0000 7fd816a00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd8040a3c40 0x7fd8040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.477+0000 7fd816a00700 1 -- 172.21.15.146:0/862358307 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd8100d0aa0 con 0x7fd820074d20 2024-04-24T08:20:06.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.477+0000 7fd81d400700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd8040a3c40 0x7fd8040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.478+0000 7fd81d400700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd8040a3c40 0x7fd8040a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009680 tx=0x7fd80c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.479+0000 7fd816a00700 1 -- 172.21.15.146:0/862358307 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd81009d440 con 0x7fd820074d20 2024-04-24T08:20:06.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.711+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd82019c7c0 con 0x7fd820074d20 2024-04-24T08:20:06.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.712+0000 7fd816a00700 1 -- 172.21.15.146:0/862358307 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd81009cd40 con 0x7fd820074d20 2024-04-24T08:20:06.714 INFO:teuthology.orchestra.run.smithi146.stdout:111669149769 2024-04-24T08:20:06.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd8040a3c40 msgr2=0x7fd8040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd8040a3c40 0x7fd8040a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009680 tx=0x7fd80c009260 comp rx=0 tx=0).stop 2024-04-24T08:20:06.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820074d20 msgr2=0x7fd82019b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820074d20 0x7fd82019b460 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd81000e9a0 tx=0x7fd81000ecb0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 shutdown_connections 2024-04-24T08:20:06.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd82010f4f0 0x7fd82019fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd8040a3c40 0x7fd8040a6100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd820074d20 0x7fd82019b460 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 --2- 172.21.15.146:0/862358307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd820075640 0x7fd82019b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 >> 172.21.15.146:0/862358307 conn(0x7fd820079c50 msgr2=0x7fd8200ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 shutdown_connections 2024-04-24T08:20:06.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:06.715+0000 7fd824a00700 1 -- 172.21.15.146:0/862358307 wait complete. 2024-04-24T08:20:06.735 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:06 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3540980312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T08:20:06.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:06 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3540980312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T08:20:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:06 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3540980312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T08:20:07.607 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430257 got 206158430264 for osd.6 2024-04-24T08:20:07.607 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:07.833 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:07 smithi146 ceph-mon[27805]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:07.833 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:07 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/862358307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T08:20:07.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:07 smithi153 ceph-mon[32746]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:07.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:07 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/862358307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T08:20:08.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:07 smithi177 ceph-mon[33046]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:08.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:07 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/862358307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T08:20:08.393 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149763 got 111669149769 for osd.3 2024-04-24T08:20:08.393 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:08.540 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738452 got 34359738458 for osd.0 2024-04-24T08:20:08.540 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:08.856 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:08 smithi146 ceph-mon[27805]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:08.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:08 smithi153 ceph-mon[32746]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:09.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:08 smithi177 ceph-mon[33046]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-24T08:20:09.422 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972391 got 266287972399 for osd.8 2024-04-24T08:20:09.422 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:09.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 -- 172.21.15.146:0/2890193990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff674102a20 msgr2=0x7ff674102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 --2- 172.21.15.146:0/2890193990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff674102a20 0x7ff674102ea0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff660009990 tx=0x7ff660009ca0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 -- 172.21.15.146:0/2890193990 shutdown_connections 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 --2- 172.21.15.146:0/2890193990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6741033e0 0x7ff67410f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 --2- 172.21.15.146:0/2890193990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff674108a20 0x7ff674108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 --2- 172.21.15.146:0/2890193990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff674102a20 0x7ff674102ea0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 -- 172.21.15.146:0/2890193990 >> 172.21.15.146:0/2890193990 conn(0x7ff6740fe570 msgr2=0x7ff674100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 -- 172.21.15.146:0/2890193990 shutdown_connections 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.186+0000 7ff673200700 1 -- 172.21.15.146:0/2890193990 wait complete. 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 Processor -- start 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 -- start start 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff674102a20 0x7ff6741991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6741033e0 0x7ff674199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff674108a20 0x7ff67419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff674112e00 con 0x7ff6741033e0 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff674112c80 con 0x7ff674102a20 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.187+0000 7ff673200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff674112f80 con 0x7ff674108a20 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6741033e0 0x7ff674199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff674102a20 0x7ff6741991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff674102a20 0x7ff6741991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:44824/0 (socket says 172.21.15.146:44824) 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 -- 172.21.15.146:0/1473214185 learned_addr learned my addr 172.21.15.146:0/1473214185 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff672800700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff674108a20 0x7ff67419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 -- 172.21.15.146:0/1473214185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff674108a20 msgr2=0x7ff67419dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff674108a20 0x7ff67419dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 -- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6741033e0 msgr2=0x7ff674199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6741033e0 0x7ff674199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 -- 172.21.15.146:0/1473214185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67419e200 con 0x7ff674102a20 2024-04-24T08:20:09.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.188+0000 7ff671e00700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff674102a20 0x7ff6741991f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff66400e9c0 tx=0x7ff66400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.189+0000 7ff66aa00700 1 -- 172.21.15.146:0/1473214185 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66400cbd0 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.189+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67419e4f0 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.189+0000 7ff66aa00700 1 -- 172.21.15.146:0/1473214185 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff66400ee60 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.189+0000 7ff66aa00700 1 -- 172.21.15.146:0/1473214185 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff664005630 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.189+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6741a5d70 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.190+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff654005330 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.193+0000 7ff66aa00700 1 -- 172.21.15.146:0/1473214185 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7ff664004750 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.194+0000 7ff66aa00700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6580a3c40 0x7ff6580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.195+0000 7ff66aa00700 1 -- 172.21.15.146:0/1473214185 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff6640d0520 con 0x7ff674102a20 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.195+0000 7ff671400700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6580a3c40 0x7ff6580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.195+0000 7ff671400700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6580a3c40 0x7ff6580a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff674100b70 tx=0x7ff66000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:09.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.199+0000 7ff66aa00700 1 -- 172.21.15.146:0/1473214185 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff66409cf00 con 0x7ff674102a20 2024-04-24T08:20:09.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.421+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff654005620 con 0x7ff674102a20 2024-04-24T08:20:09.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.422+0000 7ff66aa00700 1 -- 172.21.15.146:0/1473214185 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff66409c800 con 0x7ff674102a20 2024-04-24T08:20:09.498 INFO:teuthology.orchestra.run.smithi146.stdout:227633266739 2024-04-24T08:20:09.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.424+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6580a3c40 msgr2=0x7ff6580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.424+0000 7ff673200700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6580a3c40 0x7ff6580a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff674100b70 tx=0x7ff66000b260 comp rx=0 tx=0).stop 2024-04-24T08:20:09.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.424+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff674102a20 msgr2=0x7ff6741991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.424+0000 7ff673200700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff674102a20 0x7ff6741991f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff66400e9c0 tx=0x7ff66400ecd0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 shutdown_connections 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff674108a20 0x7ff67419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff674102a20 0x7ff6741991f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff6580a3c40 0x7ff6580a6100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 --2- 172.21.15.146:0/1473214185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6741033e0 0x7ff674199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 >> 172.21.15.146:0/1473214185 conn(0x7ff6740fe570 msgr2=0x7ff6740fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 shutdown_connections 2024-04-24T08:20:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:09.425+0000 7ff673200700 1 -- 172.21.15.146:0/1473214185 wait complete. 2024-04-24T08:20:09.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:09 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1473214185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T08:20:09.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:09 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1473214185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T08:20:10.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:09 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1473214185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T08:20:10.922 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:10 smithi146 ceph-mon[27805]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:10.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:10 smithi153 ceph-mon[32746]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:10.980 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266733 got 227633266739 for osd.7 2024-04-24T08:20:10.980 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:10.981 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-24T08:20:10.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T08:20:11.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:10 smithi177 ceph-mon[33046]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:11.428 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:13.147 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:13.147 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:13 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:13.147 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:13 smithi146 ceph-mon[27805]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-04-24T08:20:13.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:13.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:13 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:13.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:13 smithi177 ceph-mon[33046]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-04-24T08:20:13.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:13.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:13 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:13.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:13 smithi153 ceph-mon[32746]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2024-04-24T08:20:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 -- 172.21.15.146:0/998327345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac10a070 msgr2=0x7f2dac111c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 --2- 172.21.15.146:0/998327345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac10a070 0x7f2dac111c40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2da400b0d0 tx=0x7f2da400b3e0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 -- 172.21.15.146:0/998327345 shutdown_connections 2024-04-24T08:20:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 --2- 172.21.15.146:0/998327345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac10a070 0x7f2dac111c40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 --2- 172.21.15.146:0/998327345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dac104f90 0x7f2dac105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 --2- 172.21.15.146:0/998327345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac105940 0x7f2dac109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 -- 172.21.15.146:0/998327345 >> 172.21.15.146:0/998327345 conn(0x7f2dac100820 msgr2=0x7f2dac102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:13.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 -- 172.21.15.146:0/998327345 shutdown_connections 2024-04-24T08:20:13.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.678+0000 7f2db4000700 1 -- 172.21.15.146:0/998327345 wait complete. 2024-04-24T08:20:13.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.680+0000 7f2db4000700 1 Processor -- start 2024-04-24T08:20:13.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.680+0000 7f2db4000700 1 -- start start 2024-04-24T08:20:13.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db4000700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dac104f90 0x7f2dac19b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db4000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac105940 0x7f2dac19b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db4000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac10a070 0x7f2dac19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db4000700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac115060 con 0x7f2dac104f90 2024-04-24T08:20:13.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db4000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac114ee0 con 0x7f2dac10a070 2024-04-24T08:20:13.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db4000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac1151e0 con 0x7f2dac105940 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db1600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dac104f90 0x7f2dac19b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db0c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac105940 0x7f2dac19b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db0c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac105940 0x7f2dac19b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54232/0 (socket says 172.21.15.146:54232) 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db0c00700 1 -- 172.21.15.146:0/1949960537 learned_addr learned my addr 172.21.15.146:0/1949960537 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.681+0000 7f2db2000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac10a070 0x7f2dac19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2db0c00700 1 -- 172.21.15.146:0/1949960537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac10a070 msgr2=0x7f2dac19fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2db0c00700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac10a070 0x7f2dac19fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2db0c00700 1 -- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dac104f90 msgr2=0x7f2dac19b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2db0c00700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dac104f90 0x7f2dac19b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2db0c00700 1 -- 172.21.15.146:0/1949960537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac1a0420 con 0x7f2dac105940 2024-04-24T08:20:13.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2db0c00700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac105940 0x7f2dac19b950 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800e970 tx=0x7f2d9800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2daa000700 1 -- 172.21.15.146:0/1949960537 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9800cbd0 con 0x7f2dac105940 2024-04-24T08:20:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2daa000700 1 -- 172.21.15.146:0/1949960537 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d9800cd70 con 0x7f2dac105940 2024-04-24T08:20:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac1a0710 con 0x7f2dac105940 2024-04-24T08:20:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.682+0000 7f2daa000700 1 -- 172.21.15.146:0/1949960537 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980056b0 con 0x7f2dac105940 2024-04-24T08:20:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.683+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac1a7f90 con 0x7f2dac105940 2024-04-24T08:20:13.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.684+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac0697e0 con 0x7f2dac105940 2024-04-24T08:20:13.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.686+0000 7f2daa000700 1 -- 172.21.15.146:0/1949960537 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f2d98004750 con 0x7f2dac105940 2024-04-24T08:20:13.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.687+0000 7f2daa000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d940a3c40 0x7f2d940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.687+0000 7f2daa000700 1 -- 172.21.15.146:0/1949960537 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f2d980d0820 con 0x7f2dac105940 2024-04-24T08:20:13.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.687+0000 7f2db1600700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d940a3c40 0x7f2d940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.688+0000 7f2db1600700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d940a3c40 0x7f2d940a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2da0005950 tx=0x7f2da000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.691+0000 7f2daa000700 1 -- 172.21.15.146:0/1949960537 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d9809d1c0 con 0x7f2dac105940 2024-04-24T08:20:13.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.914+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2dac101930 con 0x7f2d940a3c40 2024-04-24T08:20:13.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.934+0000 7f2daa000700 1 -- 172.21.15.146:0/1949960537 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258825 (secure 0 0 0) 0x7f2dac101930 con 0x7f2d940a3c40 2024-04-24T08:20:13.942 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:13.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.941+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d940a3c40 msgr2=0x7f2d940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d940a3c40 0x7f2d940a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2da0005950 tx=0x7f2da000a300 comp rx=0 tx=0).stop 2024-04-24T08:20:13.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac105940 msgr2=0x7f2dac19b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac105940 0x7f2dac19b950 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800e970 tx=0x7f2d9800ec80 comp rx=0 tx=0).stop 2024-04-24T08:20:13.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 shutdown_connections 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dac10a070 0x7f2dac19fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f2d940a3c40 0x7f2d940a6100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dac104f90 0x7f2dac19b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 --2- 172.21.15.146:0/1949960537 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dac105940 0x7f2dac19b950 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 >> 172.21.15.146:0/1949960537 conn(0x7f2dac100820 msgr2=0x7f2dac101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 shutdown_connections 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:13.942+0000 7f2db4000700 1 -- 172.21.15.146:0/1949960537 wait complete. 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-24T08:20:14.240 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:20:14.240 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:20:14.241 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:14 smithi146 ceph-mon[27805]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:14.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:20:14.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:20:14.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:14 smithi177 ceph-mon[33046]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:14.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:20:14.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:20:14.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:14 smithi153 ceph-mon[32746]: from='mgr.14152 172.21.15.146:0/1999147475' entity='mgr.x' 2024-04-24T08:20:14.780 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-04-24T08:20:12.478274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656039,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":634,"num_read_kb":439,"num_write":350,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375276,"kb_used_data":13644,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566548,"statfs":{"total":1151940427776,"available":1151556145152,"internally_reserved":0,"allocated":13971456,"data_stored":7596657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":46,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004596"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450345+0000","last_change":"2024-04-24T08:18:47.309666+0000","last_active":"2024-04-24T08:19:56.450345+0000","last_peered":"2024-04-24T08:19:56.450345+0000","last_clean":"2024-04-24T08:19:56.450345+0000","last_became_active":"2024-04-24T08:18:47.309413+0000","last_became_peered":"2024-04-24T08:18:47.309413+0000","last_unstale":"2024-04-24T08:19:56.450345+0000","last_undegraded":"2024-04-24T08:19:56.450345+0000","last_fullsized":"2024-04-24T08:19:56.450345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:29.662505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.484474+0000","last_change":"2024-04-24T08:18:47.310321+0000","last_active":"2024-04-24T08:19:56.484474+0000","last_peered":"2024-04-24T08:19:56.484474+0000","last_clean":"2024-04-24T08:19:56.484474+0000","last_became_active":"2024-04-24T08:18:47.309956+0000","last_became_peered":"2024-04-24T08:18:47.309956+0000","last_unstale":"2024-04-24T08:19:56.484474+0000","last_undegraded":"2024-04-24T08:19:56.484474+0000","last_fullsized":"2024-04-24T08:19:56.484474+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:51:48.432725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.484477+0000","last_change":"2024-04-24T08:18:47.312153+0000","last_active":"2024-04-24T08:19:56.484477+0000","last_peered":"2024-04-24T08:19:56.484477+0000","last_clean":"2024-04-24T08:19:56.484477+0000","last_became_active":"2024-04-24T08:18:47.309465+0000","last_became_peered":"2024-04-24T08:18:47.309465+0000","last_unstale":"2024-04-24T08:19:56.484477+0000","last_undegraded":"2024-04-24T08:19:56.484477+0000","last_fullsized":"2024-04-24T08:19:56.484477+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:55.829143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445592+0000","last_change":"2024-04-24T08:18:47.297017+0000","last_active":"2024-04-24T08:19:56.445592+0000","last_peered":"2024-04-24T08:19:56.445592+0000","last_clean":"2024-04-24T08:19:56.445592+0000","last_became_active":"2024-04-24T08:18:47.296693+0000","last_became_peered":"2024-04-24T08:18:47.296693+0000","last_unstale":"2024-04-24T08:19:56.445592+0000","last_undegraded":"2024-04-24T08:19:56.445592+0000","last_fullsized":"2024-04-24T08:19:56.445592+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:23.400403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.465746+0000","last_change":"2024-04-24T08:18:47.315376+0000","last_active":"2024-04-24T08:19:56.465746+0000","last_peered":"2024-04-24T08:19:56.465746+0000","last_clean":"2024-04-24T08:19:56.465746+0000","last_became_active":"2024-04-24T08:18:47.315027+0000","last_became_peered":"2024-04-24T08:18:47.315027+0000","last_unstale":"2024-04-24T08:19:56.465746+0000","last_undegraded":"2024-04-24T08:19:56.465746+0000","last_fullsized":"2024-04-24T08:19:56.465746+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:12.504515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.444839+0000","last_change":"2024-04-24T08:18:47.306388+0000","last_active":"2024-04-24T08:19:56.444839+0000","last_peered":"2024-04-24T08:19:56.444839+0000","last_clean":"2024-04-24T08:19:56.444839+0000","last_became_active":"2024-04-24T08:18:47.306082+0000","last_became_peered":"2024-04-24T08:18:47.306082+0000","last_unstale":"2024-04-24T08:19:56.444839+0000","last_undegraded":"2024-04-24T08:19:56.444839+0000","last_fullsized":"2024-04-24T08:19:56.444839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:34:52.612799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450343+0000","last_change":"2024-04-24T08:18:47.303336+0000","last_active":"2024-04-24T08:19:56.450343+0000","last_peered":"2024-04-24T08:19:56.450343+0000","last_clean":"2024-04-24T08:19:56.450343+0000","last_became_active":"2024-04-24T08:18:47.303008+0000","last_became_peered":"2024-04-24T08:18:47.303008+0000","last_unstale":"2024-04-24T08:19:56.450343+0000","last_undegraded":"2024-04-24T08:19:56.450343+0000","last_fullsized":"2024-04-24T08:19:56.450343+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:57.404693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.475683+0000","last_change":"2024-04-24T08:18:47.316672+0000","last_active":"2024-04-24T08:19:56.475683+0000","last_peered":"2024-04-24T08:19:56.475683+0000","last_clean":"2024-04-24T08:19:56.475683+0000","last_became_active":"2024-04-24T08:18:47.315741+0000","last_became_peered":"2024-04-24T08:18:47.315741+0000","last_unstale":"2024-04-24T08:19:56.475683+0000","last_undegraded":"2024-04-24T08:19:56.475683+0000","last_fullsized":"2024-04-24T08:19:56.475683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:29.156710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445660+0000","last_change":"2024-04-24T08:18:47.291733+0000","last_active":"2024-04-24T08:19:56.445660+0000","last_peered":"2024-04-24T08:19:56.445660+0000","last_clean":"2024-04-24T08:19:56.445660+0000","last_became_active":"2024-04-24T08:18:47.291432+0000","last_became_peered":"2024-04-24T08:18:47.291432+0000","last_unstale":"2024-04-24T08:19:56.445660+0000","last_undegraded":"2024-04-24T08:19:56.445660+0000","last_fullsized":"2024-04-24T08:19:56.445660+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:38:03.601540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.437487+0000","last_change":"2024-04-24T08:18:47.294544+0000","last_active":"2024-04-24T08:19:56.437487+0000","last_peered":"2024-04-24T08:19:56.437487+0000","last_clean":"2024-04-24T08:19:56.437487+0000","last_became_active":"2024-04-24T08:18:47.294183+0000","last_became_peered":"2024-04-24T08:18:47.294183+0000","last_unstale":"2024-04-24T08:19:56.437487+0000","last_undegraded":"2024-04-24T08:19:56.437487+0000","last_fullsized":"2024-04-24T08:19:56.437487+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:53.082644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.456339+0000","last_change":"2024-04-24T08:18:47.292615+0000","last_active":"2024-04-24T08:19:56.456339+0000","last_peered":"2024-04-24T08:19:56.456339+0000","last_clean":"2024-04-24T08:19:56.456339+0000","last_became_active":"2024-04-24T08:18:47.292300+0000","last_became_peered":"2024-04-24T08:18:47.292300+0000","last_unstale":"2024-04-24T08:19:56.456339+0000","last_undegraded":"2024-04-24T08:19:56.456339+0000","last_fullsized":"2024-04-24T08:19:56.456339+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:58.293253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.446752+0000","last_change":"2024-04-24T08:18:47.298388+0000","last_active":"2024-04-24T08:19:56.446752+0000","last_peered":"2024-04-24T08:19:56.446752+0000","last_clean":"2024-04-24T08:19:56.446752+0000","last_became_active":"2024-04-24T08:18:47.298008+0000","last_became_peered":"2024-04-24T08:18:47.298008+0000","last_unstale":"2024-04-24T08:19:56.446752+0000","last_undegraded":"2024-04-24T08:19:56.446752+0000","last_fullsized":"2024-04-24T08:19:56.446752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:10:00.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.463572+0000","last_change":"2024-04-24T08:18:47.298809+0000","last_active":"2024-04-24T08:19:56.463572+0000","last_peered":"2024-04-24T08:19:56.463572+0000","last_clean":"2024-04-24T08:19:56.463572+0000","last_became_active":"2024-04-24T08:18:47.298103+0000","last_became_peered":"2024-04-24T08:18:47.298103+0000","last_unstale":"2024-04-24T08:19:56.463572+0000","last_undegraded":"2024-04-24T08:19:56.463572+0000","last_fullsized":"2024-04-24T08:19:56.463572+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:38:53.625721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.480789+0000","last_change":"2024-04-24T08:18:47.314572+0000","last_active":"2024-04-24T08:19:56.480789+0000","last_peered":"2024-04-24T08:19:56.480789+0000","last_clean":"2024-04-24T08:19:56.480789+0000","last_became_active":"2024-04-24T08:18:47.313372+0000","last_became_peered":"2024-04-24T08:18:47.313372+0000","last_unstale":"2024-04-24T08:19:56.480789+0000","last_undegraded":"2024-04-24T08:19:56.480789+0000","last_fullsized":"2024-04-24T08:19:56.480789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:56:01.954909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.480908+0000","last_change":"2024-04-24T08:18:47.314822+0000","last_active":"2024-04-24T08:19:56.480908+0000","last_peered":"2024-04-24T08:19:56.480908+0000","last_clean":"2024-04-24T08:19:56.480908+0000","last_became_active":"2024-04-24T08:18:47.310774+0000","last_became_peered":"2024-04-24T08:18:47.310774+0000","last_unstale":"2024-04-24T08:19:56.480908+0000","last_undegraded":"2024-04-24T08:19:56.480908+0000","last_fullsized":"2024-04-24T08:19:56.480908+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:24.934100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.455176+0000","last_change":"2024-04-24T08:18:47.307208+0000","last_active":"2024-04-24T08:19:56.455176+0000","last_peered":"2024-04-24T08:19:56.455176+0000","last_clean":"2024-04-24T08:19:56.455176+0000","last_became_active":"2024-04-24T08:18:47.306889+0000","last_became_peered":"2024-04-24T08:18:47.306889+0000","last_unstale":"2024-04-24T08:19:56.455176+0000","last_undegraded":"2024-04-24T08:19:56.455176+0000","last_fullsized":"2024-04-24T08:19:56.455176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:20:01.438388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450146+0000","last_change":"2024-04-24T08:18:47.322929+0000","last_active":"2024-04-24T08:19:56.450146+0000","last_peered":"2024-04-24T08:19:56.450146+0000","last_clean":"2024-04-24T08:19:56.450146+0000","last_became_active":"2024-04-24T08:18:47.322593+0000","last_became_peered":"2024-04-24T08:18:47.322593+0000","last_unstale":"2024-04-24T08:19:56.450146+0000","last_undegraded":"2024-04-24T08:19:56.450146+0000","last_fullsized":"2024-04-24T08:19:56.450146+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:09.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.453148+0000","last_change":"2024-04-24T08:18:47.304886+0000","last_active":"2024-04-24T08:19:56.453148+0000","last_peered":"2024-04-24T08:19:56.453148+0000","last_clean":"2024-04-24T08:19:56.453148+0000","last_became_active":"2024-04-24T08:18:47.302631+0000","last_became_peered":"2024-04-24T08:18:47.302631+0000","last_unstale":"2024-04-24T08:19:56.453148+0000","last_undegraded":"2024-04-24T08:19:56.453148+0000","last_fullsized":"2024-04-24T08:19:56.453148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:13.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.473510+0000","last_change":"2024-04-24T08:18:47.316828+0000","last_active":"2024-04-24T08:19:56.473510+0000","last_peered":"2024-04-24T08:19:56.473510+0000","last_clean":"2024-04-24T08:19:56.473510+0000","last_became_active":"2024-04-24T08:18:47.316540+0000","last_became_peered":"2024-04-24T08:18:47.316540+0000","last_unstale":"2024-04-24T08:19:56.473510+0000","last_undegraded":"2024-04-24T08:19:56.473510+0000","last_fullsized":"2024-04-24T08:19:56.473510+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:41:42.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.586850+0000","last_change":"2024-04-24T08:18:46.444744+0000","last_active":"2024-04-24T08:18:57.586850+0000","last_peered":"2024-04-24T08:18:57.586850+0000","last_clean":"2024-04-24T08:18:57.586850+0000","last_became_active":"2024-04-24T08:18:46.442495+0000","last_became_peered":"2024-04-24T08:18:46.442495+0000","last_unstale":"2024-04-24T08:18:57.586850+0000","last_undegraded":"2024-04-24T08:18:57.586850+0000","last_fullsized":"2024-04-24T08:18:57.586850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:33:32.270984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.468158+0000","last_change":"2024-04-24T08:18:47.320462+0000","last_active":"2024-04-24T08:19:56.468158+0000","last_peered":"2024-04-24T08:19:56.468158+0000","last_clean":"2024-04-24T08:19:56.468158+0000","last_became_active":"2024-04-24T08:18:47.319267+0000","last_became_peered":"2024-04-24T08:18:47.319267+0000","last_unstale":"2024-04-24T08:19:56.468158+0000","last_undegraded":"2024-04-24T08:19:56.468158+0000","last_fullsized":"2024-04-24T08:19:56.468158+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:50:15.882077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.594498+0000","last_change":"2024-04-24T08:18:46.453278+0000","last_active":"2024-04-24T08:18:57.594498+0000","last_peered":"2024-04-24T08:18:57.594498+0000","last_clean":"2024-04-24T08:18:57.594498+0000","last_became_active":"2024-04-24T08:18:46.452913+0000","last_became_peered":"2024-04-24T08:18:46.452913+0000","last_unstale":"2024-04-24T08:18:57.594498+0000","last_undegraded":"2024-04-24T08:18:57.594498+0000","last_fullsized":"2024-04-24T08:18:57.594498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:48.272190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.483647+0000","last_change":"2024-04-24T08:18:47.323610+0000","last_active":"2024-04-24T08:19:56.483647+0000","last_peered":"2024-04-24T08:19:56.483647+0000","last_clean":"2024-04-24T08:19:56.483647+0000","last_became_active":"2024-04-24T08:18:47.322795+0000","last_became_peered":"2024-04-24T08:18:47.322795+0000","last_unstale":"2024-04-24T08:19:56.483647+0000","last_undegraded":"2024-04-24T08:19:56.483647+0000","last_fullsized":"2024-04-24T08:19:56.483647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:32.951564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.730353+0000","last_change":"2024-04-24T08:18:57.730353+0000","last_active":"2024-04-24T08:18:57.730353+0000","last_peered":"2024-04-24T08:18:57.730353+0000","last_clean":"2024-04-24T08:18:57.730353+0000","last_became_active":"2024-04-24T08:18:57.725936+0000","last_became_peered":"2024-04-24T08:18:57.725936+0000","last_unstale":"2024-04-24T08:18:57.730353+0000","last_undegraded":"2024-04-24T08:18:57.730353+0000","last_fullsized":"2024-04-24T08:18:57.730353+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:00:27.764723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.460965+0000","last_change":"2024-04-24T08:18:47.304884+0000","last_active":"2024-04-24T08:19:56.460965+0000","last_peered":"2024-04-24T08:19:56.460965+0000","last_clean":"2024-04-24T08:19:56.460965+0000","last_became_active":"2024-04-24T08:18:47.304520+0000","last_became_peered":"2024-04-24T08:18:47.304520+0000","last_unstale":"2024-04-24T08:19:56.460965+0000","last_undegraded":"2024-04-24T08:19:56.460965+0000","last_fullsized":"2024-04-24T08:19:56.460965+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:42.325010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.580076+0000","last_change":"2024-04-24T08:18:46.441073+0000","last_active":"2024-04-24T08:18:57.580076+0000","last_peered":"2024-04-24T08:18:57.580076+0000","last_clean":"2024-04-24T08:18:57.580076+0000","last_became_active":"2024-04-24T08:18:46.440721+0000","last_became_peered":"2024-04-24T08:18:46.440721+0000","last_unstale":"2024-04-24T08:18:57.580076+0000","last_undegraded":"2024-04-24T08:18:57.580076+0000","last_fullsized":"2024-04-24T08:18:57.580076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:51:39.516458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.447372+0000","last_change":"2024-04-24T08:18:47.293947+0000","last_active":"2024-04-24T08:19:56.447372+0000","last_peered":"2024-04-24T08:19:56.447372+0000","last_clean":"2024-04-24T08:19:56.447372+0000","last_became_active":"2024-04-24T08:18:47.293623+0000","last_became_peered":"2024-04-24T08:18:47.293623+0000","last_unstale":"2024-04-24T08:19:56.447372+0000","last_undegraded":"2024-04-24T08:19:56.447372+0000","last_fullsized":"2024-04-24T08:19:56.447372+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:23.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":107,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:07.642202+0000","last_change":"2024-04-24T08:18:46.439909+0000","last_active":"2024-04-24T08:20:07.642202+0000","last_peered":"2024-04-24T08:20:07.642202+0000","last_clean":"2024-04-24T08:20:07.642202+0000","last_became_active":"2024-04-24T08:18:46.439597+0000","last_became_peered":"2024-04-24T08:18:46.439597+0000","last_unstale":"2024-04-24T08:20:07.642202+0000","last_undegraded":"2024-04-24T08:20:07.642202+0000","last_fullsized":"2024-04-24T08:20:07.642202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:49:11.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.443437+0000","last_change":"2024-04-24T08:18:47.306480+0000","last_active":"2024-04-24T08:19:56.443437+0000","last_peered":"2024-04-24T08:19:56.443437+0000","last_clean":"2024-04-24T08:19:56.443437+0000","last_became_active":"2024-04-24T08:18:47.306104+0000","last_became_peered":"2024-04-24T08:18:47.306104+0000","last_unstale":"2024-04-24T08:19:56.443437+0000","last_undegraded":"2024-04-24T08:19:56.443437+0000","last_fullsized":"2024-04-24T08:19:56.443437+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:56.340836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.723501+0000","last_change":"2024-04-24T08:18:46.439381+0000","last_active":"2024-04-24T08:18:57.723501+0000","last_peered":"2024-04-24T08:18:57.723501+0000","last_clean":"2024-04-24T08:18:57.723501+0000","last_became_active":"2024-04-24T08:18:46.439033+0000","last_became_peered":"2024-04-24T08:18:46.439033+0000","last_unstale":"2024-04-24T08:18:57.723501+0000","last_undegraded":"2024-04-24T08:18:57.723501+0000","last_fullsized":"2024-04-24T08:18:57.723501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:48:28.833066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.465908+0000","last_change":"2024-04-24T08:18:57.595549+0000","last_active":"2024-04-24T08:19:56.465908+0000","last_peered":"2024-04-24T08:19:56.465908+0000","last_clean":"2024-04-24T08:19:56.465908+0000","last_became_active":"2024-04-24T08:18:57.595098+0000","last_became_peered":"2024-04-24T08:18:57.595098+0000","last_unstale":"2024-04-24T08:19:56.465908+0000","last_undegraded":"2024-04-24T08:19:56.465908+0000","last_fullsized":"2024-04-24T08:19:56.465908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:09:22.722996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577343+0000","last_change":"2024-04-24T08:18:46.448151+0000","last_active":"2024-04-24T08:18:57.577343+0000","last_peered":"2024-04-24T08:18:57.577343+0000","last_clean":"2024-04-24T08:18:57.577343+0000","last_became_active":"2024-04-24T08:18:46.447077+0000","last_became_peered":"2024-04-24T08:18:46.447077+0000","last_unstale":"2024-04-24T08:18:57.577343+0000","last_undegraded":"2024-04-24T08:18:57.577343+0000","last_fullsized":"2024-04-24T08:18:57.577343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:00.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.474596+0000","last_change":"2024-04-24T08:18:47.319423+0000","last_active":"2024-04-24T08:19:56.474596+0000","last_peered":"2024-04-24T08:19:56.474596+0000","last_clean":"2024-04-24T08:19:56.474596+0000","last_became_active":"2024-04-24T08:18:47.316284+0000","last_became_peered":"2024-04-24T08:18:47.316284+0000","last_unstale":"2024-04-24T08:19:56.474596+0000","last_undegraded":"2024-04-24T08:19:56.474596+0000","last_fullsized":"2024-04-24T08:19:56.474596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:55.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584609+0000","last_change":"2024-04-24T08:18:46.436152+0000","last_active":"2024-04-24T08:18:57.584609+0000","last_peered":"2024-04-24T08:18:57.584609+0000","last_clean":"2024-04-24T08:18:57.584609+0000","last_became_active":"2024-04-24T08:18:46.435835+0000","last_became_peered":"2024-04-24T08:18:46.435835+0000","last_unstale":"2024-04-24T08:18:57.584609+0000","last_undegraded":"2024-04-24T08:18:57.584609+0000","last_fullsized":"2024-04-24T08:18:57.584609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:12:30.536455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.457938+0000","last_change":"2024-04-24T08:18:57.725728+0000","last_active":"2024-04-24T08:19:56.457938+0000","last_peered":"2024-04-24T08:19:56.457938+0000","last_clean":"2024-04-24T08:19:56.457938+0000","last_became_active":"2024-04-24T08:18:57.724704+0000","last_became_peered":"2024-04-24T08:18:57.724704+0000","last_unstale":"2024-04-24T08:19:56.457938+0000","last_undegraded":"2024-04-24T08:19:56.457938+0000","last_fullsized":"2024-04-24T08:19:56.457938+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:27.501006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":102,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:09.644932+0000","last_change":"2024-04-24T08:18:46.446652+0000","last_active":"2024-04-24T08:20:09.644932+0000","last_peered":"2024-04-24T08:20:09.644932+0000","last_clean":"2024-04-24T08:20:09.644932+0000","last_became_active":"2024-04-24T08:18:46.446348+0000","last_became_peered":"2024-04-24T08:18:46.446348+0000","last_unstale":"2024-04-24T08:20:09.644932+0000","last_undegraded":"2024-04-24T08:20:09.644932+0000","last_fullsized":"2024-04-24T08:20:09.644932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:25.791359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445344+0000","last_change":"2024-04-24T08:18:47.309072+0000","last_active":"2024-04-24T08:19:56.445344+0000","last_peered":"2024-04-24T08:19:56.445344+0000","last_clean":"2024-04-24T08:19:56.445344+0000","last_became_active":"2024-04-24T08:18:47.308729+0000","last_became_peered":"2024-04-24T08:18:47.308729+0000","last_unstale":"2024-04-24T08:19:56.445344+0000","last_undegraded":"2024-04-24T08:19:56.445344+0000","last_fullsized":"2024-04-24T08:19:56.445344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:58.256876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.582322+0000","last_change":"2024-04-24T08:18:46.439190+0000","last_active":"2024-04-24T08:18:57.582322+0000","last_peered":"2024-04-24T08:18:57.582322+0000","last_clean":"2024-04-24T08:18:57.582322+0000","last_became_active":"2024-04-24T08:18:46.438442+0000","last_became_peered":"2024-04-24T08:18:46.438442+0000","last_unstale":"2024-04-24T08:18:57.582322+0000","last_undegraded":"2024-04-24T08:18:57.582322+0000","last_fullsized":"2024-04-24T08:18:57.582322+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:42.614563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.484174+0000","last_change":"2024-04-24T08:18:47.311447+0000","last_active":"2024-04-24T08:19:56.484174+0000","last_peered":"2024-04-24T08:19:56.484174+0000","last_clean":"2024-04-24T08:19:56.484174+0000","last_became_active":"2024-04-24T08:18:47.310939+0000","last_became_peered":"2024-04-24T08:18:47.310939+0000","last_unstale":"2024-04-24T08:19:56.484174+0000","last_undegraded":"2024-04-24T08:19:56.484174+0000","last_fullsized":"2024-04-24T08:19:56.484174+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:42.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584998+0000","last_change":"2024-04-24T08:18:46.450596+0000","last_active":"2024-04-24T08:18:57.584998+0000","last_peered":"2024-04-24T08:18:57.584998+0000","last_clean":"2024-04-24T08:18:57.584998+0000","last_became_active":"2024-04-24T08:18:46.450226+0000","last_became_peered":"2024-04-24T08:18:46.450226+0000","last_unstale":"2024-04-24T08:18:57.584998+0000","last_undegraded":"2024-04-24T08:18:57.584998+0000","last_fullsized":"2024-04-24T08:18:57.584998+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:05.722507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.447318+0000","last_change":"2024-04-24T08:18:47.295266+0000","last_active":"2024-04-24T08:19:56.447318+0000","last_peered":"2024-04-24T08:19:56.447318+0000","last_clean":"2024-04-24T08:19:56.447318+0000","last_became_active":"2024-04-24T08:18:47.294963+0000","last_became_peered":"2024-04-24T08:18:47.294963+0000","last_unstale":"2024-04-24T08:19:56.447318+0000","last_undegraded":"2024-04-24T08:19:56.447318+0000","last_fullsized":"2024-04-24T08:19:56.447318+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:07.638921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'31","reported_seq":217,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.652961+0000","last_change":"2024-04-24T08:18:46.435524+0000","last_active":"2024-04-24T08:20:10.652961+0000","last_peered":"2024-04-24T08:20:10.652961+0000","last_clean":"2024-04-24T08:20:10.652961+0000","last_became_active":"2024-04-24T08:18:46.435212+0000","last_became_peered":"2024-04-24T08:18:46.435212+0000","last_unstale":"2024-04-24T08:20:10.652961+0000","last_undegraded":"2024-04-24T08:20:10.652961+0000","last_fullsized":"2024-04-24T08:20:10.652961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:45.575905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":167,"num_read_kb":143,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.454437+0000","last_change":"2024-04-24T08:18:47.293648+0000","last_active":"2024-04-24T08:19:56.454437+0000","last_peered":"2024-04-24T08:19:56.454437+0000","last_clean":"2024-04-24T08:19:56.454437+0000","last_became_active":"2024-04-24T08:18:47.293260+0000","last_became_peered":"2024-04-24T08:18:47.293260+0000","last_unstale":"2024-04-24T08:19:56.454437+0000","last_undegraded":"2024-04-24T08:19:56.454437+0000","last_fullsized":"2024-04-24T08:19:56.454437+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:36.462606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577836+0000","last_change":"2024-04-24T08:18:46.441401+0000","last_active":"2024-04-24T08:18:57.577836+0000","last_peered":"2024-04-24T08:18:57.577836+0000","last_clean":"2024-04-24T08:18:57.577836+0000","last_became_active":"2024-04-24T08:18:46.440985+0000","last_became_peered":"2024-04-24T08:18:46.440985+0000","last_unstale":"2024-04-24T08:18:57.577836+0000","last_undegraded":"2024-04-24T08:18:57.577836+0000","last_fullsized":"2024-04-24T08:18:57.577836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:45.201518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.476644+0000","last_change":"2024-04-24T08:18:47.320928+0000","last_active":"2024-04-24T08:19:56.476644+0000","last_peered":"2024-04-24T08:19:56.476644+0000","last_clean":"2024-04-24T08:19:56.476644+0000","last_became_active":"2024-04-24T08:18:47.320515+0000","last_became_peered":"2024-04-24T08:18:47.320515+0000","last_unstale":"2024-04-24T08:19:56.476644+0000","last_undegraded":"2024-04-24T08:19:56.476644+0000","last_fullsized":"2024-04-24T08:19:56.476644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:55:42.899438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584699+0000","last_change":"2024-04-24T08:18:46.437677+0000","last_active":"2024-04-24T08:18:57.584699+0000","last_peered":"2024-04-24T08:18:57.584699+0000","last_clean":"2024-04-24T08:18:57.584699+0000","last_became_active":"2024-04-24T08:18:46.437364+0000","last_became_peered":"2024-04-24T08:18:46.437364+0000","last_unstale":"2024-04-24T08:18:57.584699+0000","last_undegraded":"2024-04-24T08:18:57.584699+0000","last_fullsized":"2024-04-24T08:18:57.584699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:25:55.408887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.466029+0000","last_change":"2024-04-24T08:18:47.298396+0000","last_active":"2024-04-24T08:19:56.466029+0000","last_peered":"2024-04-24T08:19:56.466029+0000","last_clean":"2024-04-24T08:19:56.466029+0000","last_became_active":"2024-04-24T08:18:47.298021+0000","last_became_peered":"2024-04-24T08:18:47.298021+0000","last_unstale":"2024-04-24T08:19:56.466029+0000","last_undegraded":"2024-04-24T08:19:56.466029+0000","last_fullsized":"2024-04-24T08:19:56.466029+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:24.015897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:07.694665+0000","last_change":"2024-04-24T08:18:46.436612+0000","last_active":"2024-04-24T08:20:07.694665+0000","last_peered":"2024-04-24T08:20:07.694665+0000","last_clean":"2024-04-24T08:20:07.694665+0000","last_became_active":"2024-04-24T08:18:46.436211+0000","last_became_peered":"2024-04-24T08:18:46.436211+0000","last_unstale":"2024-04-24T08:20:07.694665+0000","last_undegraded":"2024-04-24T08:20:07.694665+0000","last_fullsized":"2024-04-24T08:20:07.694665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:49:05.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.453480+0000","last_change":"2024-04-24T08:18:47.290849+0000","last_active":"2024-04-24T08:19:56.453480+0000","last_peered":"2024-04-24T08:19:56.453480+0000","last_clean":"2024-04-24T08:19:56.453480+0000","last_became_active":"2024-04-24T08:18:47.290514+0000","last_became_peered":"2024-04-24T08:18:47.290514+0000","last_unstale":"2024-04-24T08:19:56.453480+0000","last_undegraded":"2024-04-24T08:19:56.453480+0000","last_fullsized":"2024-04-24T08:19:56.453480+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:39:12.527477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.579682+0000","last_change":"2024-04-24T08:18:46.451815+0000","last_active":"2024-04-24T08:18:57.579682+0000","last_peered":"2024-04-24T08:18:57.579682+0000","last_clean":"2024-04-24T08:18:57.579682+0000","last_became_active":"2024-04-24T08:18:46.451478+0000","last_became_peered":"2024-04-24T08:18:46.451478+0000","last_unstale":"2024-04-24T08:18:57.579682+0000","last_undegraded":"2024-04-24T08:18:57.579682+0000","last_fullsized":"2024-04-24T08:18:57.579682+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:54:22.907143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.446838+0000","last_change":"2024-04-24T08:18:47.295181+0000","last_active":"2024-04-24T08:19:56.446838+0000","last_peered":"2024-04-24T08:19:56.446838+0000","last_clean":"2024-04-24T08:19:56.446838+0000","last_became_active":"2024-04-24T08:18:47.294842+0000","last_became_peered":"2024-04-24T08:18:47.294842+0000","last_unstale":"2024-04-24T08:19:56.446838+0000","last_undegraded":"2024-04-24T08:19:56.446838+0000","last_fullsized":"2024-04-24T08:19:56.446838+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:37.058025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.580128+0000","last_change":"2024-04-24T08:18:46.437967+0000","last_active":"2024-04-24T08:18:57.580128+0000","last_peered":"2024-04-24T08:18:57.580128+0000","last_clean":"2024-04-24T08:18:57.580128+0000","last_became_active":"2024-04-24T08:18:46.437617+0000","last_became_peered":"2024-04-24T08:18:46.437617+0000","last_unstale":"2024-04-24T08:18:57.580128+0000","last_undegraded":"2024-04-24T08:18:57.580128+0000","last_fullsized":"2024-04-24T08:18:57.580128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:07.303827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.443717+0000","last_change":"2024-04-24T08:18:57.726323+0000","last_active":"2024-04-24T08:19:56.443717+0000","last_peered":"2024-04-24T08:19:56.443717+0000","last_clean":"2024-04-24T08:19:56.443717+0000","last_became_active":"2024-04-24T08:18:57.725924+0000","last_became_peered":"2024-04-24T08:18:57.725924+0000","last_unstale":"2024-04-24T08:19:56.443717+0000","last_undegraded":"2024-04-24T08:19:56.443717+0000","last_fullsized":"2024-04-24T08:19:56.443717+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:15:33.756804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.591169+0000","last_change":"2024-04-24T08:18:57.591169+0000","last_active":"2024-04-24T08:18:57.591169+0000","last_peered":"2024-04-24T08:18:57.591169+0000","last_clean":"2024-04-24T08:18:57.591169+0000","last_became_active":"2024-04-24T08:18:57.590300+0000","last_became_peered":"2024-04-24T08:18:57.590300+0000","last_unstale":"2024-04-24T08:18:57.591169+0000","last_undegraded":"2024-04-24T08:18:57.591169+0000","last_fullsized":"2024-04-24T08:18:57.591169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:13.632780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.473340+0000","last_change":"2024-04-24T08:18:47.316791+0000","last_active":"2024-04-24T08:19:56.473340+0000","last_peered":"2024-04-24T08:19:56.473340+0000","last_clean":"2024-04-24T08:19:56.473340+0000","last_became_active":"2024-04-24T08:18:47.316486+0000","last_became_peered":"2024-04-24T08:18:47.316486+0000","last_unstale":"2024-04-24T08:19:56.473340+0000","last_undegraded":"2024-04-24T08:19:56.473340+0000","last_fullsized":"2024-04-24T08:19:56.473340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:36:37.190732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.582062+0000","last_change":"2024-04-24T08:18:46.432831+0000","last_active":"2024-04-24T08:18:57.582062+0000","last_peered":"2024-04-24T08:18:57.582062+0000","last_clean":"2024-04-24T08:18:57.582062+0000","last_became_active":"2024-04-24T08:18:46.432483+0000","last_became_peered":"2024-04-24T08:18:46.432483+0000","last_unstale":"2024-04-24T08:18:57.582062+0000","last_undegraded":"2024-04-24T08:18:57.582062+0000","last_fullsized":"2024-04-24T08:18:57.582062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:43:24.802099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.451323+0000","last_change":"2024-04-24T08:18:47.309408+0000","last_active":"2024-04-24T08:19:56.451323+0000","last_peered":"2024-04-24T08:19:56.451323+0000","last_clean":"2024-04-24T08:19:56.451323+0000","last_became_active":"2024-04-24T08:18:47.309083+0000","last_became_peered":"2024-04-24T08:18:47.309083+0000","last_unstale":"2024-04-24T08:19:56.451323+0000","last_undegraded":"2024-04-24T08:19:56.451323+0000","last_fullsized":"2024-04-24T08:19:56.451323+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:15:44.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.436736+0000","last_change":"2024-04-24T08:18:47.291060+0000","last_active":"2024-04-24T08:19:56.436736+0000","last_peered":"2024-04-24T08:19:56.436736+0000","last_clean":"2024-04-24T08:19:56.436736+0000","last_became_active":"2024-04-24T08:18:47.290635+0000","last_became_peered":"2024-04-24T08:18:47.290635+0000","last_unstale":"2024-04-24T08:19:56.436736+0000","last_undegraded":"2024-04-24T08:19:56.436736+0000","last_fullsized":"2024-04-24T08:19:56.436736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:20.372853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.695834+0000","last_change":"2024-04-24T08:18:46.449850+0000","last_active":"2024-04-24T08:20:08.695834+0000","last_peered":"2024-04-24T08:20:08.695834+0000","last_clean":"2024-04-24T08:20:08.695834+0000","last_became_active":"2024-04-24T08:18:46.448486+0000","last_became_peered":"2024-04-24T08:18:46.448486+0000","last_unstale":"2024-04-24T08:20:08.695834+0000","last_undegraded":"2024-04-24T08:20:08.695834+0000","last_fullsized":"2024-04-24T08:20:08.695834+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:54.611867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.466651+0000","last_change":"2024-04-24T08:18:47.328051+0000","last_active":"2024-04-24T08:19:56.466651+0000","last_peered":"2024-04-24T08:19:56.466651+0000","last_clean":"2024-04-24T08:19:56.466651+0000","last_became_active":"2024-04-24T08:18:47.327474+0000","last_became_peered":"2024-04-24T08:18:47.327474+0000","last_unstale":"2024-04-24T08:19:56.466651+0000","last_undegraded":"2024-04-24T08:19:56.466651+0000","last_fullsized":"2024-04-24T08:19:56.466651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:07.577741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.724791+0000","last_change":"2024-04-24T08:18:46.444375+0000","last_active":"2024-04-24T08:18:57.724791+0000","last_peered":"2024-04-24T08:18:57.724791+0000","last_clean":"2024-04-24T08:18:57.724791+0000","last_became_active":"2024-04-24T08:18:46.443963+0000","last_became_peered":"2024-04-24T08:18:46.443963+0000","last_unstale":"2024-04-24T08:18:57.724791+0000","last_undegraded":"2024-04-24T08:18:57.724791+0000","last_fullsized":"2024-04-24T08:18:57.724791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:22.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.581987+0000","last_change":"2024-04-24T08:17:29.632219+0000","last_active":"2024-04-24T08:18:57.581987+0000","last_peered":"2024-04-24T08:18:57.581987+0000","last_clean":"2024-04-24T08:18:57.581987+0000","last_became_active":"2024-04-24T08:17:29.631841+0000","last_became_peered":"2024-04-24T08:17:29.631841+0000","last_unstale":"2024-04-24T08:18:57.581987+0000","last_undegraded":"2024-04-24T08:18:57.581987+0000","last_fullsized":"2024-04-24T08:18:57.581987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_clean_scrub_stamp":"2024-04-24T08:13:41.228801+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:02:55.415959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.460885+0000","last_change":"2024-04-24T08:18:47.301876+0000","last_active":"2024-04-24T08:19:56.460885+0000","last_peered":"2024-04-24T08:19:56.460885+0000","last_clean":"2024-04-24T08:19:56.460885+0000","last_became_active":"2024-04-24T08:18:47.301545+0000","last_became_peered":"2024-04-24T08:18:47.301545+0000","last_unstale":"2024-04-24T08:19:56.460885+0000","last_undegraded":"2024-04-24T08:19:56.460885+0000","last_fullsized":"2024-04-24T08:19:56.460885+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:34.650014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.581866+0000","last_change":"2024-04-24T08:18:46.437915+0000","last_active":"2024-04-24T08:18:57.581866+0000","last_peered":"2024-04-24T08:18:57.581866+0000","last_clean":"2024-04-24T08:18:57.581866+0000","last_became_active":"2024-04-24T08:18:46.437567+0000","last_became_peered":"2024-04-24T08:18:46.437567+0000","last_unstale":"2024-04-24T08:18:57.581866+0000","last_undegraded":"2024-04-24T08:18:57.581866+0000","last_fullsized":"2024-04-24T08:18:57.581866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:36.801535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.580177+0000","last_change":"2024-04-24T08:18:46.451834+0000","last_active":"2024-04-24T08:18:57.580177+0000","last_peered":"2024-04-24T08:18:57.580177+0000","last_clean":"2024-04-24T08:18:57.580177+0000","last_became_active":"2024-04-24T08:18:46.451505+0000","last_became_peered":"2024-04-24T08:18:46.451505+0000","last_unstale":"2024-04-24T08:18:57.580177+0000","last_undegraded":"2024-04-24T08:18:57.580177+0000","last_fullsized":"2024-04-24T08:18:57.580177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:26.468176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.462457+0000","last_change":"2024-04-24T08:18:47.310490+0000","last_active":"2024-04-24T08:19:56.462457+0000","last_peered":"2024-04-24T08:19:56.462457+0000","last_clean":"2024-04-24T08:19:56.462457+0000","last_became_active":"2024-04-24T08:18:47.309615+0000","last_became_peered":"2024-04-24T08:18:47.309615+0000","last_unstale":"2024-04-24T08:19:56.462457+0000","last_undegraded":"2024-04-24T08:19:56.462457+0000","last_fullsized":"2024-04-24T08:19:56.462457+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:22:55.063485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'11","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:01.461007+0000","last_change":"2024-04-24T08:18:57.602098+0000","last_active":"2024-04-24T08:19:01.461007+0000","last_peered":"2024-04-24T08:19:01.461007+0000","last_clean":"2024-04-24T08:19:01.461007+0000","last_became_active":"2024-04-24T08:18:57.595899+0000","last_became_peered":"2024-04-24T08:18:57.595899+0000","last_unstale":"2024-04-24T08:19:01.461007+0000","last_undegraded":"2024-04-24T08:19:01.461007+0000","last_fullsized":"2024-04-24T08:19:01.461007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:31.951803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.469016+0000","last_change":"2024-04-24T08:18:47.319022+0000","last_active":"2024-04-24T08:19:56.469016+0000","last_peered":"2024-04-24T08:19:56.469016+0000","last_clean":"2024-04-24T08:19:56.469016+0000","last_became_active":"2024-04-24T08:18:47.318167+0000","last_became_peered":"2024-04-24T08:18:47.318167+0000","last_unstale":"2024-04-24T08:19:56.469016+0000","last_undegraded":"2024-04-24T08:19:56.469016+0000","last_fullsized":"2024-04-24T08:19:56.469016+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:18.980856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.725014+0000","last_change":"2024-04-24T08:18:46.433018+0000","last_active":"2024-04-24T08:18:57.725014+0000","last_peered":"2024-04-24T08:18:57.725014+0000","last_clean":"2024-04-24T08:18:57.725014+0000","last_became_active":"2024-04-24T08:18:46.432681+0000","last_became_peered":"2024-04-24T08:18:46.432681+0000","last_unstale":"2024-04-24T08:18:57.725014+0000","last_undegraded":"2024-04-24T08:18:57.725014+0000","last_fullsized":"2024-04-24T08:18:57.725014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:53.691684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.485040+0000","last_change":"2024-04-24T08:18:47.326055+0000","last_active":"2024-04-24T08:19:56.485040+0000","last_peered":"2024-04-24T08:19:56.485040+0000","last_clean":"2024-04-24T08:19:56.485040+0000","last_became_active":"2024-04-24T08:18:47.324956+0000","last_became_peered":"2024-04-24T08:18:47.324956+0000","last_unstale":"2024-04-24T08:19:56.485040+0000","last_undegraded":"2024-04-24T08:19:56.485040+0000","last_fullsized":"2024-04-24T08:19:56.485040+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:44:37.431264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.575383+0000","last_change":"2024-04-24T08:18:46.450710+0000","last_active":"2024-04-24T08:18:57.575383+0000","last_peered":"2024-04-24T08:18:57.575383+0000","last_clean":"2024-04-24T08:18:57.575383+0000","last_became_active":"2024-04-24T08:18:46.450391+0000","last_became_peered":"2024-04-24T08:18:46.450391+0000","last_unstale":"2024-04-24T08:18:57.575383+0000","last_undegraded":"2024-04-24T08:18:57.575383+0000","last_fullsized":"2024-04-24T08:18:57.575383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:19.905524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.414318+0000","last_change":"2024-04-24T08:18:47.292114+0000","last_active":"2024-04-24T08:19:56.414318+0000","last_peered":"2024-04-24T08:19:56.414318+0000","last_clean":"2024-04-24T08:19:56.414318+0000","last_became_active":"2024-04-24T08:18:47.291787+0000","last_became_peered":"2024-04-24T08:18:47.291787+0000","last_unstale":"2024-04-24T08:19:56.414318+0000","last_undegraded":"2024-04-24T08:19:56.414318+0000","last_fullsized":"2024-04-24T08:19:56.414318+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:37.020368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.593098+0000","last_change":"2024-04-24T08:18:46.446027+0000","last_active":"2024-04-24T08:18:57.593098+0000","last_peered":"2024-04-24T08:18:57.593098+0000","last_clean":"2024-04-24T08:18:57.593098+0000","last_became_active":"2024-04-24T08:18:46.445634+0000","last_became_peered":"2024-04-24T08:18:46.445634+0000","last_unstale":"2024-04-24T08:18:57.593098+0000","last_undegraded":"2024-04-24T08:18:57.593098+0000","last_fullsized":"2024-04-24T08:18:57.593098+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:46.208560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450275+0000","last_change":"2024-04-24T08:18:47.307800+0000","last_active":"2024-04-24T08:19:56.450275+0000","last_peered":"2024-04-24T08:19:56.450275+0000","last_clean":"2024-04-24T08:19:56.450275+0000","last_became_active":"2024-04-24T08:18:47.307501+0000","last_became_peered":"2024-04-24T08:18:47.307501+0000","last_unstale":"2024-04-24T08:19:56.450275+0000","last_undegraded":"2024-04-24T08:19:56.450275+0000","last_fullsized":"2024-04-24T08:19:56.450275+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:04:05.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577702+0000","last_change":"2024-04-24T08:18:46.442811+0000","last_active":"2024-04-24T08:18:57.577702+0000","last_peered":"2024-04-24T08:18:57.577702+0000","last_clean":"2024-04-24T08:18:57.577702+0000","last_became_active":"2024-04-24T08:18:46.442479+0000","last_became_peered":"2024-04-24T08:18:46.442479+0000","last_unstale":"2024-04-24T08:18:57.577702+0000","last_undegraded":"2024-04-24T08:18:57.577702+0000","last_fullsized":"2024-04-24T08:18:57.577702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:24:07.236529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.478705+0000","last_change":"2024-04-24T08:18:47.325046+0000","last_active":"2024-04-24T08:19:56.478705+0000","last_peered":"2024-04-24T08:19:56.478705+0000","last_clean":"2024-04-24T08:19:56.478705+0000","last_became_active":"2024-04-24T08:18:47.324720+0000","last_became_peered":"2024-04-24T08:18:47.324720+0000","last_unstale":"2024-04-24T08:19:56.478705+0000","last_undegraded":"2024-04-24T08:19:56.478705+0000","last_fullsized":"2024-04-24T08:19:56.478705+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:11.240388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.579744+0000","last_change":"2024-04-24T08:18:46.440572+0000","last_active":"2024-04-24T08:18:57.579744+0000","last_peered":"2024-04-24T08:18:57.579744+0000","last_clean":"2024-04-24T08:18:57.579744+0000","last_became_active":"2024-04-24T08:18:46.440216+0000","last_became_peered":"2024-04-24T08:18:46.440216+0000","last_unstale":"2024-04-24T08:18:57.579744+0000","last_undegraded":"2024-04-24T08:18:57.579744+0000","last_fullsized":"2024-04-24T08:18:57.579744+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:51.352874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.463196+0000","last_change":"2024-04-24T08:18:47.293465+0000","last_active":"2024-04-24T08:19:56.463196+0000","last_peered":"2024-04-24T08:19:56.463196+0000","last_clean":"2024-04-24T08:19:56.463196+0000","last_became_active":"2024-04-24T08:18:47.293062+0000","last_became_peered":"2024-04-24T08:18:47.293062+0000","last_unstale":"2024-04-24T08:19:56.463196+0000","last_undegraded":"2024-04-24T08:19:56.463196+0000","last_fullsized":"2024-04-24T08:19:56.463196+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:36.029511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.578157+0000","last_change":"2024-04-24T08:18:46.451007+0000","last_active":"2024-04-24T08:18:57.578157+0000","last_peered":"2024-04-24T08:18:57.578157+0000","last_clean":"2024-04-24T08:18:57.578157+0000","last_became_active":"2024-04-24T08:18:46.450676+0000","last_became_peered":"2024-04-24T08:18:46.450676+0000","last_unstale":"2024-04-24T08:18:57.578157+0000","last_undegraded":"2024-04-24T08:18:57.578157+0000","last_fullsized":"2024-04-24T08:18:57.578157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:40:00.329311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.456457+0000","last_change":"2024-04-24T08:18:47.293859+0000","last_active":"2024-04-24T08:19:56.456457+0000","last_peered":"2024-04-24T08:19:56.456457+0000","last_clean":"2024-04-24T08:19:56.456457+0000","last_became_active":"2024-04-24T08:18:47.293509+0000","last_became_peered":"2024-04-24T08:18:47.293509+0000","last_unstale":"2024-04-24T08:19:56.456457+0000","last_undegraded":"2024-04-24T08:19:56.456457+0000","last_fullsized":"2024-04-24T08:19:56.456457+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:41.224037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584722+0000","last_change":"2024-04-24T08:18:46.436661+0000","last_active":"2024-04-24T08:18:57.584722+0000","last_peered":"2024-04-24T08:18:57.584722+0000","last_clean":"2024-04-24T08:18:57.584722+0000","last_became_active":"2024-04-24T08:18:46.436353+0000","last_became_peered":"2024-04-24T08:18:46.436353+0000","last_unstale":"2024-04-24T08:18:57.584722+0000","last_undegraded":"2024-04-24T08:18:57.584722+0000","last_fullsized":"2024-04-24T08:18:57.584722+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:57:25.559706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.451593+0000","last_change":"2024-04-24T08:18:57.597302+0000","last_active":"2024-04-24T08:19:56.451593+0000","last_peered":"2024-04-24T08:19:56.451593+0000","last_clean":"2024-04-24T08:19:56.451593+0000","last_became_active":"2024-04-24T08:18:57.591420+0000","last_became_peered":"2024-04-24T08:18:57.591420+0000","last_unstale":"2024-04-24T08:19:56.451593+0000","last_undegraded":"2024-04-24T08:19:56.451593+0000","last_fullsized":"2024-04-24T08:19:56.451593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:03:34.286436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577353+0000","last_change":"2024-04-24T08:18:46.432499+0000","last_active":"2024-04-24T08:18:57.577353+0000","last_peered":"2024-04-24T08:18:57.577353+0000","last_clean":"2024-04-24T08:18:57.577353+0000","last_became_active":"2024-04-24T08:18:46.432085+0000","last_became_peered":"2024-04-24T08:18:46.432085+0000","last_unstale":"2024-04-24T08:18:57.577353+0000","last_undegraded":"2024-04-24T08:18:57.577353+0000","last_fullsized":"2024-04-24T08:18:57.577353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:03.794091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.455423+0000","last_change":"2024-04-24T08:18:47.291193+0000","last_active":"2024-04-24T08:19:56.455423+0000","last_peered":"2024-04-24T08:19:56.455423+0000","last_clean":"2024-04-24T08:19:56.455423+0000","last_became_active":"2024-04-24T08:18:47.289903+0000","last_became_peered":"2024-04-24T08:18:47.289903+0000","last_unstale":"2024-04-24T08:19:56.455423+0000","last_undegraded":"2024-04-24T08:19:56.455423+0000","last_fullsized":"2024-04-24T08:19:56.455423+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:35:18.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.439071+0000","last_change":"2024-04-24T08:18:47.292628+0000","last_active":"2024-04-24T08:19:56.439071+0000","last_peered":"2024-04-24T08:19:56.439071+0000","last_clean":"2024-04-24T08:19:56.439071+0000","last_became_active":"2024-04-24T08:18:47.292060+0000","last_became_peered":"2024-04-24T08:18:47.292060+0000","last_unstale":"2024-04-24T08:19:56.439071+0000","last_undegraded":"2024-04-24T08:19:56.439071+0000","last_fullsized":"2024-04-24T08:19:56.439071+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:53.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445363+0000","last_change":"2024-04-24T08:18:47.297121+0000","last_active":"2024-04-24T08:19:56.445363+0000","last_peered":"2024-04-24T08:19:56.445363+0000","last_clean":"2024-04-24T08:19:56.445363+0000","last_became_active":"2024-04-24T08:18:47.296787+0000","last_became_peered":"2024-04-24T08:18:47.296787+0000","last_unstale":"2024-04-24T08:19:56.445363+0000","last_undegraded":"2024-04-24T08:19:56.445363+0000","last_fullsized":"2024-04-24T08:19:56.445363+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:44:10.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.473806+0000","last_change":"2024-04-24T08:18:47.320669+0000","last_active":"2024-04-24T08:19:56.473806+0000","last_peered":"2024-04-24T08:19:56.473806+0000","last_clean":"2024-04-24T08:19:56.473806+0000","last_became_active":"2024-04-24T08:18:47.320287+0000","last_became_peered":"2024-04-24T08:18:47.320287+0000","last_unstale":"2024-04-24T08:19:56.473806+0000","last_undegraded":"2024-04-24T08:19:56.473806+0000","last_fullsized":"2024-04-24T08:19:56.473806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:39:03.275189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.481107+0000","last_change":"2024-04-24T08:18:57.585568+0000","last_active":"2024-04-24T08:19:56.481107+0000","last_peered":"2024-04-24T08:19:56.481107+0000","last_clean":"2024-04-24T08:19:56.481107+0000","last_became_active":"2024-04-24T08:18:57.584799+0000","last_became_peered":"2024-04-24T08:18:57.584799+0000","last_unstale":"2024-04-24T08:19:56.481107+0000","last_undegraded":"2024-04-24T08:19:56.481107+0000","last_fullsized":"2024-04-24T08:19:56.481107+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:47.202344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.461915+0000","last_change":"2024-04-24T08:18:47.316068+0000","last_active":"2024-04-24T08:19:56.461915+0000","last_peered":"2024-04-24T08:19:56.461915+0000","last_clean":"2024-04-24T08:19:56.461915+0000","last_became_active":"2024-04-24T08:18:47.315647+0000","last_became_peered":"2024-04-24T08:18:47.315647+0000","last_unstale":"2024-04-24T08:19:56.461915+0000","last_undegraded":"2024-04-24T08:19:56.461915+0000","last_fullsized":"2024-04-24T08:19:56.461915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:33.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.436673+0000","last_change":"2024-04-24T08:18:47.294966+0000","last_active":"2024-04-24T08:19:56.436673+0000","last_peered":"2024-04-24T08:19:56.436673+0000","last_clean":"2024-04-24T08:19:56.436673+0000","last_became_active":"2024-04-24T08:18:47.294672+0000","last_became_peered":"2024-04-24T08:18:47.294672+0000","last_unstale":"2024-04-24T08:19:56.436673+0000","last_undegraded":"2024-04-24T08:19:56.436673+0000","last_fullsized":"2024-04-24T08:19:56.436673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:37.359114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.443180+0000","last_change":"2024-04-24T08:18:47.294548+0000","last_active":"2024-04-24T08:19:56.443180+0000","last_peered":"2024-04-24T08:19:56.443180+0000","last_clean":"2024-04-24T08:19:56.443180+0000","last_became_active":"2024-04-24T08:18:47.293589+0000","last_became_peered":"2024-04-24T08:18:47.293589+0000","last_unstale":"2024-04-24T08:19:56.443180+0000","last_undegraded":"2024-04-24T08:19:56.443180+0000","last_fullsized":"2024-04-24T08:19:56.443180+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:34:56.208453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.469129+0000","last_change":"2024-04-24T08:18:47.313156+0000","last_active":"2024-04-24T08:19:56.469129+0000","last_peered":"2024-04-24T08:19:56.469129+0000","last_clean":"2024-04-24T08:19:56.469129+0000","last_became_active":"2024-04-24T08:18:47.312829+0000","last_became_peered":"2024-04-24T08:18:47.312829+0000","last_unstale":"2024-04-24T08:19:56.469129+0000","last_undegraded":"2024-04-24T08:19:56.469129+0000","last_fullsized":"2024-04-24T08:19:56.469129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:28:36.638514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.467098+0000","last_change":"2024-04-24T08:18:47.320022+0000","last_active":"2024-04-24T08:19:56.467098+0000","last_peered":"2024-04-24T08:19:56.467098+0000","last_clean":"2024-04-24T08:19:56.467098+0000","last_became_active":"2024-04-24T08:18:47.319707+0000","last_became_peered":"2024-04-24T08:18:47.319707+0000","last_unstale":"2024-04-24T08:19:56.467098+0000","last_undegraded":"2024-04-24T08:19:56.467098+0000","last_fullsized":"2024-04-24T08:19:56.467098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:51.330874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.470982+0000","last_change":"2024-04-24T08:18:47.323187+0000","last_active":"2024-04-24T08:19:56.470982+0000","last_peered":"2024-04-24T08:19:56.470982+0000","last_clean":"2024-04-24T08:19:56.470982+0000","last_became_active":"2024-04-24T08:18:47.320258+0000","last_became_peered":"2024-04-24T08:18:47.320258+0000","last_unstale":"2024-04-24T08:19:56.470982+0000","last_undegraded":"2024-04-24T08:19:56.470982+0000","last_fullsized":"2024-04-24T08:19:56.470982+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:42:32.770950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.456892+0000","last_change":"2024-04-24T08:18:47.321926+0000","last_active":"2024-04-24T08:19:56.456892+0000","last_peered":"2024-04-24T08:19:56.456892+0000","last_clean":"2024-04-24T08:19:56.456892+0000","last_became_active":"2024-04-24T08:18:47.321584+0000","last_became_peered":"2024-04-24T08:18:47.321584+0000","last_unstale":"2024-04-24T08:19:56.456892+0000","last_undegraded":"2024-04-24T08:19:56.456892+0000","last_fullsized":"2024-04-24T08:19:56.456892+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:24.357157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.463681+0000","last_change":"2024-04-24T08:18:47.313310+0000","last_active":"2024-04-24T08:19:56.463681+0000","last_peered":"2024-04-24T08:19:56.463681+0000","last_clean":"2024-04-24T08:19:56.463681+0000","last_became_active":"2024-04-24T08:18:47.312968+0000","last_became_peered":"2024-04-24T08:18:47.312968+0000","last_unstale":"2024-04-24T08:19:56.463681+0000","last_undegraded":"2024-04-24T08:19:56.463681+0000","last_fullsized":"2024-04-24T08:19:56.463681+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:08.967577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.444201+0000","last_change":"2024-04-24T08:18:57.591543+0000","last_active":"2024-04-24T08:19:56.444201+0000","last_peered":"2024-04-24T08:19:56.444201+0000","last_clean":"2024-04-24T08:19:56.444201+0000","last_became_active":"2024-04-24T08:18:57.591043+0000","last_became_peered":"2024-04-24T08:18:57.591043+0000","last_unstale":"2024-04-24T08:19:56.444201+0000","last_undegraded":"2024-04-24T08:19:56.444201+0000","last_fullsized":"2024-04-24T08:19:56.444201+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:52:38.669469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":498,"num_read_kb":320,"num_write":167,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":196311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285598,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":469627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547235,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1679360,"data_stored":1122403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":469980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972401,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":469783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266740,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1593344,"data_stored":1061854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430266,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":469252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":530340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":470523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":532222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":469446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":469689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691904,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1605632,"data_stored":1061538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:20:14.788 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T08:20:15.284 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:15.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:15 smithi177 ceph-mon[33046]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:20:15.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:15 smithi177 ceph-mon[33046]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:15.437 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:15 smithi146 ceph-mon[27805]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:20:15.438 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:15 smithi146 ceph-mon[27805]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:15.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:15 smithi153 ceph-mon[32746]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:20:15.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:15 smithi153 ceph-mon[32746]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 -- 172.21.15.146:0/1421306507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08406ad40 msgr2=0x7fa08406b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 --2- 172.21.15.146:0/1421306507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08406ad40 0x7fa08406b1a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa074009990 tx=0x7fa074009ca0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 -- 172.21.15.146:0/1421306507 shutdown_connections 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 --2- 172.21.15.146:0/1421306507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa084105530 0x7fa0841059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 --2- 172.21.15.146:0/1421306507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa084107a20 0x7fa08406a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 --2- 172.21.15.146:0/1421306507 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa08406ad40 0x7fa08406b1a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 -- 172.21.15.146:0/1421306507 >> 172.21.15.146:0/1421306507 conn(0x7fa0840789a0 msgr2=0x7fa084078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 -- 172.21.15.146:0/1421306507 shutdown_connections 2024-04-24T08:20:17.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.583+0000 7fa08ba00700 1 -- 172.21.15.146:0/1421306507 wait complete. 2024-04-24T08:20:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.584+0000 7fa08ba00700 1 Processor -- start 2024-04-24T08:20:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.584+0000 7fa08ba00700 1 -- start start 2024-04-24T08:20:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa08ba00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08406ad40 0x7fa0841a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:17.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa08ba00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa084105530 0x7fa0841a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:17.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa08ba00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa084107a20 0x7fa08419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:17.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa08ba00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa0841193c0 con 0x7fa084105530 2024-04-24T08:20:17.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa08ba00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa084119240 con 0x7fa084107a20 2024-04-24T08:20:17.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa08ba00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa084119540 con 0x7fa08406ad40 2024-04-24T08:20:17.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa089000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08406ad40 0x7fa0841a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:17.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa07be00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa084105530 0x7fa0841a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa089000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08406ad40 0x7fa0841a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54238/0 (socket says 172.21.15.146:54238) 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa089000700 1 -- 172.21.15.146:0/4016642665 learned_addr learned my addr 172.21.15.146:0/4016642665 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa089a00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa084107a20 0x7fa08419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa07be00700 1 -- 172.21.15.146:0/4016642665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08406ad40 msgr2=0x7fa0841a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa07be00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08406ad40 0x7fa0841a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa07be00700 1 -- 172.21.15.146:0/4016642665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa084107a20 msgr2=0x7fa08419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.585+0000 7fa07be00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa084107a20 0x7fa08419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.586+0000 7fa07be00700 1 -- 172.21.15.146:0/4016642665 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08419ffc0 con 0x7fa084105530 2024-04-24T08:20:17.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.586+0000 7fa07be00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa084105530 0x7fa0841a5b00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa0740059d0 tx=0x7fa07400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:17.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.586+0000 7fa082000700 1 -- 172.21.15.146:0/4016642665 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074027070 con 0x7fa084105530 2024-04-24T08:20:17.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.586+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0841a0250 con 0x7fa084105530 2024-04-24T08:20:17.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.587+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0841ac540 con 0x7fa084105530 2024-04-24T08:20:17.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.587+0000 7fa082000700 1 -- 172.21.15.146:0/4016642665 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa074003d70 con 0x7fa084105530 2024-04-24T08:20:17.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.587+0000 7fa082000700 1 -- 172.21.15.146:0/4016642665 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa074031060 con 0x7fa084105530 2024-04-24T08:20:17.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.588+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa060005330 con 0x7fa084105530 2024-04-24T08:20:17.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.590+0000 7fa082000700 1 -- 172.21.15.146:0/4016642665 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fa07402a070 con 0x7fa084105530 2024-04-24T08:20:17.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.591+0000 7fa082000700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0640a3cc0 0x7fa0640a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:17.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.591+0000 7fa082000700 1 -- 172.21.15.146:0/4016642665 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fa0740d1050 con 0x7fa084105530 2024-04-24T08:20:17.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.591+0000 7fa089000700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0640a3cc0 0x7fa0640a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:17.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.592+0000 7fa089000700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0640a3cc0 0x7fa0640a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009ca0 tx=0x7fa06c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:17.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.596+0000 7fa082000700 1 -- 172.21.15.146:0/4016642665 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa074013070 con 0x7fa084105530 2024-04-24T08:20:17.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:17 smithi146 ceph-mon[27805]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:17.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:17 smithi177 ceph-mon[33046]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:17.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.825+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa060000bc0 con 0x7fa0640a3cc0 2024-04-24T08:20:17.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.847+0000 7fa082000700 1 -- 172.21.15.146:0/4016642665 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258825 (secure 0 0 0) 0x7fa060000bc0 con 0x7fa0640a3cc0 2024-04-24T08:20:17.855 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:17.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.854+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0640a3cc0 msgr2=0x7fa0640a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:17.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.854+0000 7fa08ba00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0640a3cc0 0x7fa0640a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009ca0 tx=0x7fa06c009260 comp rx=0 tx=0).stop 2024-04-24T08:20:17.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa084105530 msgr2=0x7fa0841a5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:17.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa084105530 0x7fa0841a5b00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa0740059d0 tx=0x7fa07400bab0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 shutdown_connections 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa084107a20 0x7fa08419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa08406ad40 0x7fa0841a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0640a3cc0 0x7fa0640a6180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 --2- 172.21.15.146:0/4016642665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa084105530 0x7fa0841a5b00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 >> 172.21.15.146:0/4016642665 conn(0x7fa0840789a0 msgr2=0x7fa084104510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 shutdown_connections 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:17.855+0000 7fa08ba00700 1 -- 172.21.15.146:0/4016642665 wait complete. 2024-04-24T08:20:17.858 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-24T08:20:17.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:17 smithi153 ceph-mon[32746]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:18.829 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-24T08:20:16.480247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656039,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":649,"num_read_kb":444,"num_write":350,"num_write_kb":3054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375276,"kb_used_data":13644,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566548,"statfs":{"total":1151940427776,"available":1151556145152,"internally_reserved":0,"allocated":13971456,"data_stored":7596657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004996"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450345+0000","last_change":"2024-04-24T08:18:47.309666+0000","last_active":"2024-04-24T08:19:56.450345+0000","last_peered":"2024-04-24T08:19:56.450345+0000","last_clean":"2024-04-24T08:19:56.450345+0000","last_became_active":"2024-04-24T08:18:47.309413+0000","last_became_peered":"2024-04-24T08:18:47.309413+0000","last_unstale":"2024-04-24T08:19:56.450345+0000","last_undegraded":"2024-04-24T08:19:56.450345+0000","last_fullsized":"2024-04-24T08:19:56.450345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:29.662505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.484474+0000","last_change":"2024-04-24T08:18:47.310321+0000","last_active":"2024-04-24T08:19:56.484474+0000","last_peered":"2024-04-24T08:19:56.484474+0000","last_clean":"2024-04-24T08:19:56.484474+0000","last_became_active":"2024-04-24T08:18:47.309956+0000","last_became_peered":"2024-04-24T08:18:47.309956+0000","last_unstale":"2024-04-24T08:19:56.484474+0000","last_undegraded":"2024-04-24T08:19:56.484474+0000","last_fullsized":"2024-04-24T08:19:56.484474+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:51:48.432725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.484477+0000","last_change":"2024-04-24T08:18:47.312153+0000","last_active":"2024-04-24T08:19:56.484477+0000","last_peered":"2024-04-24T08:19:56.484477+0000","last_clean":"2024-04-24T08:19:56.484477+0000","last_became_active":"2024-04-24T08:18:47.309465+0000","last_became_peered":"2024-04-24T08:18:47.309465+0000","last_unstale":"2024-04-24T08:19:56.484477+0000","last_undegraded":"2024-04-24T08:19:56.484477+0000","last_fullsized":"2024-04-24T08:19:56.484477+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:55.829143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445592+0000","last_change":"2024-04-24T08:18:47.297017+0000","last_active":"2024-04-24T08:19:56.445592+0000","last_peered":"2024-04-24T08:19:56.445592+0000","last_clean":"2024-04-24T08:19:56.445592+0000","last_became_active":"2024-04-24T08:18:47.296693+0000","last_became_peered":"2024-04-24T08:18:47.296693+0000","last_unstale":"2024-04-24T08:19:56.445592+0000","last_undegraded":"2024-04-24T08:19:56.445592+0000","last_fullsized":"2024-04-24T08:19:56.445592+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:23.400403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.465746+0000","last_change":"2024-04-24T08:18:47.315376+0000","last_active":"2024-04-24T08:19:56.465746+0000","last_peered":"2024-04-24T08:19:56.465746+0000","last_clean":"2024-04-24T08:19:56.465746+0000","last_became_active":"2024-04-24T08:18:47.315027+0000","last_became_peered":"2024-04-24T08:18:47.315027+0000","last_unstale":"2024-04-24T08:19:56.465746+0000","last_undegraded":"2024-04-24T08:19:56.465746+0000","last_fullsized":"2024-04-24T08:19:56.465746+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:12.504515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.444839+0000","last_change":"2024-04-24T08:18:47.306388+0000","last_active":"2024-04-24T08:19:56.444839+0000","last_peered":"2024-04-24T08:19:56.444839+0000","last_clean":"2024-04-24T08:19:56.444839+0000","last_became_active":"2024-04-24T08:18:47.306082+0000","last_became_peered":"2024-04-24T08:18:47.306082+0000","last_unstale":"2024-04-24T08:19:56.444839+0000","last_undegraded":"2024-04-24T08:19:56.444839+0000","last_fullsized":"2024-04-24T08:19:56.444839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:34:52.612799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450343+0000","last_change":"2024-04-24T08:18:47.303336+0000","last_active":"2024-04-24T08:19:56.450343+0000","last_peered":"2024-04-24T08:19:56.450343+0000","last_clean":"2024-04-24T08:19:56.450343+0000","last_became_active":"2024-04-24T08:18:47.303008+0000","last_became_peered":"2024-04-24T08:18:47.303008+0000","last_unstale":"2024-04-24T08:19:56.450343+0000","last_undegraded":"2024-04-24T08:19:56.450343+0000","last_fullsized":"2024-04-24T08:19:56.450343+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:57.404693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.475683+0000","last_change":"2024-04-24T08:18:47.316672+0000","last_active":"2024-04-24T08:19:56.475683+0000","last_peered":"2024-04-24T08:19:56.475683+0000","last_clean":"2024-04-24T08:19:56.475683+0000","last_became_active":"2024-04-24T08:18:47.315741+0000","last_became_peered":"2024-04-24T08:18:47.315741+0000","last_unstale":"2024-04-24T08:19:56.475683+0000","last_undegraded":"2024-04-24T08:19:56.475683+0000","last_fullsized":"2024-04-24T08:19:56.475683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:29.156710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445660+0000","last_change":"2024-04-24T08:18:47.291733+0000","last_active":"2024-04-24T08:19:56.445660+0000","last_peered":"2024-04-24T08:19:56.445660+0000","last_clean":"2024-04-24T08:19:56.445660+0000","last_became_active":"2024-04-24T08:18:47.291432+0000","last_became_peered":"2024-04-24T08:18:47.291432+0000","last_unstale":"2024-04-24T08:19:56.445660+0000","last_undegraded":"2024-04-24T08:19:56.445660+0000","last_fullsized":"2024-04-24T08:19:56.445660+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:38:03.601540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.437487+0000","last_change":"2024-04-24T08:18:47.294544+0000","last_active":"2024-04-24T08:19:56.437487+0000","last_peered":"2024-04-24T08:19:56.437487+0000","last_clean":"2024-04-24T08:19:56.437487+0000","last_became_active":"2024-04-24T08:18:47.294183+0000","last_became_peered":"2024-04-24T08:18:47.294183+0000","last_unstale":"2024-04-24T08:19:56.437487+0000","last_undegraded":"2024-04-24T08:19:56.437487+0000","last_fullsized":"2024-04-24T08:19:56.437487+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:53.082644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.456339+0000","last_change":"2024-04-24T08:18:47.292615+0000","last_active":"2024-04-24T08:19:56.456339+0000","last_peered":"2024-04-24T08:19:56.456339+0000","last_clean":"2024-04-24T08:19:56.456339+0000","last_became_active":"2024-04-24T08:18:47.292300+0000","last_became_peered":"2024-04-24T08:18:47.292300+0000","last_unstale":"2024-04-24T08:19:56.456339+0000","last_undegraded":"2024-04-24T08:19:56.456339+0000","last_fullsized":"2024-04-24T08:19:56.456339+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:58.293253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.446752+0000","last_change":"2024-04-24T08:18:47.298388+0000","last_active":"2024-04-24T08:19:56.446752+0000","last_peered":"2024-04-24T08:19:56.446752+0000","last_clean":"2024-04-24T08:19:56.446752+0000","last_became_active":"2024-04-24T08:18:47.298008+0000","last_became_peered":"2024-04-24T08:18:47.298008+0000","last_unstale":"2024-04-24T08:19:56.446752+0000","last_undegraded":"2024-04-24T08:19:56.446752+0000","last_fullsized":"2024-04-24T08:19:56.446752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:10:00.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.463572+0000","last_change":"2024-04-24T08:18:47.298809+0000","last_active":"2024-04-24T08:19:56.463572+0000","last_peered":"2024-04-24T08:19:56.463572+0000","last_clean":"2024-04-24T08:19:56.463572+0000","last_became_active":"2024-04-24T08:18:47.298103+0000","last_became_peered":"2024-04-24T08:18:47.298103+0000","last_unstale":"2024-04-24T08:19:56.463572+0000","last_undegraded":"2024-04-24T08:19:56.463572+0000","last_fullsized":"2024-04-24T08:19:56.463572+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:38:53.625721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.480789+0000","last_change":"2024-04-24T08:18:47.314572+0000","last_active":"2024-04-24T08:19:56.480789+0000","last_peered":"2024-04-24T08:19:56.480789+0000","last_clean":"2024-04-24T08:19:56.480789+0000","last_became_active":"2024-04-24T08:18:47.313372+0000","last_became_peered":"2024-04-24T08:18:47.313372+0000","last_unstale":"2024-04-24T08:19:56.480789+0000","last_undegraded":"2024-04-24T08:19:56.480789+0000","last_fullsized":"2024-04-24T08:19:56.480789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:56:01.954909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.480908+0000","last_change":"2024-04-24T08:18:47.314822+0000","last_active":"2024-04-24T08:19:56.480908+0000","last_peered":"2024-04-24T08:19:56.480908+0000","last_clean":"2024-04-24T08:19:56.480908+0000","last_became_active":"2024-04-24T08:18:47.310774+0000","last_became_peered":"2024-04-24T08:18:47.310774+0000","last_unstale":"2024-04-24T08:19:56.480908+0000","last_undegraded":"2024-04-24T08:19:56.480908+0000","last_fullsized":"2024-04-24T08:19:56.480908+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:24.934100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.455176+0000","last_change":"2024-04-24T08:18:47.307208+0000","last_active":"2024-04-24T08:19:56.455176+0000","last_peered":"2024-04-24T08:19:56.455176+0000","last_clean":"2024-04-24T08:19:56.455176+0000","last_became_active":"2024-04-24T08:18:47.306889+0000","last_became_peered":"2024-04-24T08:18:47.306889+0000","last_unstale":"2024-04-24T08:19:56.455176+0000","last_undegraded":"2024-04-24T08:19:56.455176+0000","last_fullsized":"2024-04-24T08:19:56.455176+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:20:01.438388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450146+0000","last_change":"2024-04-24T08:18:47.322929+0000","last_active":"2024-04-24T08:19:56.450146+0000","last_peered":"2024-04-24T08:19:56.450146+0000","last_clean":"2024-04-24T08:19:56.450146+0000","last_became_active":"2024-04-24T08:18:47.322593+0000","last_became_peered":"2024-04-24T08:18:47.322593+0000","last_unstale":"2024-04-24T08:19:56.450146+0000","last_undegraded":"2024-04-24T08:19:56.450146+0000","last_fullsized":"2024-04-24T08:19:56.450146+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:09.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.453148+0000","last_change":"2024-04-24T08:18:47.304886+0000","last_active":"2024-04-24T08:19:56.453148+0000","last_peered":"2024-04-24T08:19:56.453148+0000","last_clean":"2024-04-24T08:19:56.453148+0000","last_became_active":"2024-04-24T08:18:47.302631+0000","last_became_peered":"2024-04-24T08:18:47.302631+0000","last_unstale":"2024-04-24T08:19:56.453148+0000","last_undegraded":"2024-04-24T08:19:56.453148+0000","last_fullsized":"2024-04-24T08:19:56.453148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:13.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.473510+0000","last_change":"2024-04-24T08:18:47.316828+0000","last_active":"2024-04-24T08:19:56.473510+0000","last_peered":"2024-04-24T08:19:56.473510+0000","last_clean":"2024-04-24T08:19:56.473510+0000","last_became_active":"2024-04-24T08:18:47.316540+0000","last_became_peered":"2024-04-24T08:18:47.316540+0000","last_unstale":"2024-04-24T08:19:56.473510+0000","last_undegraded":"2024-04-24T08:19:56.473510+0000","last_fullsized":"2024-04-24T08:19:56.473510+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:41:42.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.586850+0000","last_change":"2024-04-24T08:18:46.444744+0000","last_active":"2024-04-24T08:18:57.586850+0000","last_peered":"2024-04-24T08:18:57.586850+0000","last_clean":"2024-04-24T08:18:57.586850+0000","last_became_active":"2024-04-24T08:18:46.442495+0000","last_became_peered":"2024-04-24T08:18:46.442495+0000","last_unstale":"2024-04-24T08:18:57.586850+0000","last_undegraded":"2024-04-24T08:18:57.586850+0000","last_fullsized":"2024-04-24T08:18:57.586850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:33:32.270984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.468158+0000","last_change":"2024-04-24T08:18:47.320462+0000","last_active":"2024-04-24T08:19:56.468158+0000","last_peered":"2024-04-24T08:19:56.468158+0000","last_clean":"2024-04-24T08:19:56.468158+0000","last_became_active":"2024-04-24T08:18:47.319267+0000","last_became_peered":"2024-04-24T08:18:47.319267+0000","last_unstale":"2024-04-24T08:19:56.468158+0000","last_undegraded":"2024-04-24T08:19:56.468158+0000","last_fullsized":"2024-04-24T08:19:56.468158+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:50:15.882077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.594498+0000","last_change":"2024-04-24T08:18:46.453278+0000","last_active":"2024-04-24T08:18:57.594498+0000","last_peered":"2024-04-24T08:18:57.594498+0000","last_clean":"2024-04-24T08:18:57.594498+0000","last_became_active":"2024-04-24T08:18:46.452913+0000","last_became_peered":"2024-04-24T08:18:46.452913+0000","last_unstale":"2024-04-24T08:18:57.594498+0000","last_undegraded":"2024-04-24T08:18:57.594498+0000","last_fullsized":"2024-04-24T08:18:57.594498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:48.272190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.483647+0000","last_change":"2024-04-24T08:18:47.323610+0000","last_active":"2024-04-24T08:19:56.483647+0000","last_peered":"2024-04-24T08:19:56.483647+0000","last_clean":"2024-04-24T08:19:56.483647+0000","last_became_active":"2024-04-24T08:18:47.322795+0000","last_became_peered":"2024-04-24T08:18:47.322795+0000","last_unstale":"2024-04-24T08:19:56.483647+0000","last_undegraded":"2024-04-24T08:19:56.483647+0000","last_fullsized":"2024-04-24T08:19:56.483647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:32.951564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.730353+0000","last_change":"2024-04-24T08:18:57.730353+0000","last_active":"2024-04-24T08:18:57.730353+0000","last_peered":"2024-04-24T08:18:57.730353+0000","last_clean":"2024-04-24T08:18:57.730353+0000","last_became_active":"2024-04-24T08:18:57.725936+0000","last_became_peered":"2024-04-24T08:18:57.725936+0000","last_unstale":"2024-04-24T08:18:57.730353+0000","last_undegraded":"2024-04-24T08:18:57.730353+0000","last_fullsized":"2024-04-24T08:18:57.730353+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:00:27.764723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.460965+0000","last_change":"2024-04-24T08:18:47.304884+0000","last_active":"2024-04-24T08:19:56.460965+0000","last_peered":"2024-04-24T08:19:56.460965+0000","last_clean":"2024-04-24T08:19:56.460965+0000","last_became_active":"2024-04-24T08:18:47.304520+0000","last_became_peered":"2024-04-24T08:18:47.304520+0000","last_unstale":"2024-04-24T08:19:56.460965+0000","last_undegraded":"2024-04-24T08:19:56.460965+0000","last_fullsized":"2024-04-24T08:19:56.460965+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:42.325010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.580076+0000","last_change":"2024-04-24T08:18:46.441073+0000","last_active":"2024-04-24T08:18:57.580076+0000","last_peered":"2024-04-24T08:18:57.580076+0000","last_clean":"2024-04-24T08:18:57.580076+0000","last_became_active":"2024-04-24T08:18:46.440721+0000","last_became_peered":"2024-04-24T08:18:46.440721+0000","last_unstale":"2024-04-24T08:18:57.580076+0000","last_undegraded":"2024-04-24T08:18:57.580076+0000","last_fullsized":"2024-04-24T08:18:57.580076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:51:39.516458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.447372+0000","last_change":"2024-04-24T08:18:47.293947+0000","last_active":"2024-04-24T08:19:56.447372+0000","last_peered":"2024-04-24T08:19:56.447372+0000","last_clean":"2024-04-24T08:19:56.447372+0000","last_became_active":"2024-04-24T08:18:47.293623+0000","last_became_peered":"2024-04-24T08:18:47.293623+0000","last_unstale":"2024-04-24T08:19:56.447372+0000","last_undegraded":"2024-04-24T08:19:56.447372+0000","last_fullsized":"2024-04-24T08:19:56.447372+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:23.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":112,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:12.650189+0000","last_change":"2024-04-24T08:18:46.439909+0000","last_active":"2024-04-24T08:20:12.650189+0000","last_peered":"2024-04-24T08:20:12.650189+0000","last_clean":"2024-04-24T08:20:12.650189+0000","last_became_active":"2024-04-24T08:18:46.439597+0000","last_became_peered":"2024-04-24T08:18:46.439597+0000","last_unstale":"2024-04-24T08:20:12.650189+0000","last_undegraded":"2024-04-24T08:20:12.650189+0000","last_fullsized":"2024-04-24T08:20:12.650189+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:49:11.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.443437+0000","last_change":"2024-04-24T08:18:47.306480+0000","last_active":"2024-04-24T08:19:56.443437+0000","last_peered":"2024-04-24T08:19:56.443437+0000","last_clean":"2024-04-24T08:19:56.443437+0000","last_became_active":"2024-04-24T08:18:47.306104+0000","last_became_peered":"2024-04-24T08:18:47.306104+0000","last_unstale":"2024-04-24T08:19:56.443437+0000","last_undegraded":"2024-04-24T08:19:56.443437+0000","last_fullsized":"2024-04-24T08:19:56.443437+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:56.340836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.723501+0000","last_change":"2024-04-24T08:18:46.439381+0000","last_active":"2024-04-24T08:18:57.723501+0000","last_peered":"2024-04-24T08:18:57.723501+0000","last_clean":"2024-04-24T08:18:57.723501+0000","last_became_active":"2024-04-24T08:18:46.439033+0000","last_became_peered":"2024-04-24T08:18:46.439033+0000","last_unstale":"2024-04-24T08:18:57.723501+0000","last_undegraded":"2024-04-24T08:18:57.723501+0000","last_fullsized":"2024-04-24T08:18:57.723501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:48:28.833066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.465908+0000","last_change":"2024-04-24T08:18:57.595549+0000","last_active":"2024-04-24T08:19:56.465908+0000","last_peered":"2024-04-24T08:19:56.465908+0000","last_clean":"2024-04-24T08:19:56.465908+0000","last_became_active":"2024-04-24T08:18:57.595098+0000","last_became_peered":"2024-04-24T08:18:57.595098+0000","last_unstale":"2024-04-24T08:19:56.465908+0000","last_undegraded":"2024-04-24T08:19:56.465908+0000","last_fullsized":"2024-04-24T08:19:56.465908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:09:22.722996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577343+0000","last_change":"2024-04-24T08:18:46.448151+0000","last_active":"2024-04-24T08:18:57.577343+0000","last_peered":"2024-04-24T08:18:57.577343+0000","last_clean":"2024-04-24T08:18:57.577343+0000","last_became_active":"2024-04-24T08:18:46.447077+0000","last_became_peered":"2024-04-24T08:18:46.447077+0000","last_unstale":"2024-04-24T08:18:57.577343+0000","last_undegraded":"2024-04-24T08:18:57.577343+0000","last_fullsized":"2024-04-24T08:18:57.577343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:00.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.474596+0000","last_change":"2024-04-24T08:18:47.319423+0000","last_active":"2024-04-24T08:19:56.474596+0000","last_peered":"2024-04-24T08:19:56.474596+0000","last_clean":"2024-04-24T08:19:56.474596+0000","last_became_active":"2024-04-24T08:18:47.316284+0000","last_became_peered":"2024-04-24T08:18:47.316284+0000","last_unstale":"2024-04-24T08:19:56.474596+0000","last_undegraded":"2024-04-24T08:19:56.474596+0000","last_fullsized":"2024-04-24T08:19:56.474596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:55.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584609+0000","last_change":"2024-04-24T08:18:46.436152+0000","last_active":"2024-04-24T08:18:57.584609+0000","last_peered":"2024-04-24T08:18:57.584609+0000","last_clean":"2024-04-24T08:18:57.584609+0000","last_became_active":"2024-04-24T08:18:46.435835+0000","last_became_peered":"2024-04-24T08:18:46.435835+0000","last_unstale":"2024-04-24T08:18:57.584609+0000","last_undegraded":"2024-04-24T08:18:57.584609+0000","last_fullsized":"2024-04-24T08:18:57.584609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:12:30.536455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.457938+0000","last_change":"2024-04-24T08:18:57.725728+0000","last_active":"2024-04-24T08:19:56.457938+0000","last_peered":"2024-04-24T08:19:56.457938+0000","last_clean":"2024-04-24T08:19:56.457938+0000","last_became_active":"2024-04-24T08:18:57.724704+0000","last_became_peered":"2024-04-24T08:18:57.724704+0000","last_unstale":"2024-04-24T08:19:56.457938+0000","last_undegraded":"2024-04-24T08:19:56.457938+0000","last_fullsized":"2024-04-24T08:19:56.457938+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:27.501006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":102,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:09.644932+0000","last_change":"2024-04-24T08:18:46.446652+0000","last_active":"2024-04-24T08:20:09.644932+0000","last_peered":"2024-04-24T08:20:09.644932+0000","last_clean":"2024-04-24T08:20:09.644932+0000","last_became_active":"2024-04-24T08:18:46.446348+0000","last_became_peered":"2024-04-24T08:18:46.446348+0000","last_unstale":"2024-04-24T08:20:09.644932+0000","last_undegraded":"2024-04-24T08:20:09.644932+0000","last_fullsized":"2024-04-24T08:20:09.644932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:25.791359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445344+0000","last_change":"2024-04-24T08:18:47.309072+0000","last_active":"2024-04-24T08:19:56.445344+0000","last_peered":"2024-04-24T08:19:56.445344+0000","last_clean":"2024-04-24T08:19:56.445344+0000","last_became_active":"2024-04-24T08:18:47.308729+0000","last_became_peered":"2024-04-24T08:18:47.308729+0000","last_unstale":"2024-04-24T08:19:56.445344+0000","last_undegraded":"2024-04-24T08:19:56.445344+0000","last_fullsized":"2024-04-24T08:19:56.445344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:58.256876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.582322+0000","last_change":"2024-04-24T08:18:46.439190+0000","last_active":"2024-04-24T08:18:57.582322+0000","last_peered":"2024-04-24T08:18:57.582322+0000","last_clean":"2024-04-24T08:18:57.582322+0000","last_became_active":"2024-04-24T08:18:46.438442+0000","last_became_peered":"2024-04-24T08:18:46.438442+0000","last_unstale":"2024-04-24T08:18:57.582322+0000","last_undegraded":"2024-04-24T08:18:57.582322+0000","last_fullsized":"2024-04-24T08:18:57.582322+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:42.614563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.484174+0000","last_change":"2024-04-24T08:18:47.311447+0000","last_active":"2024-04-24T08:19:56.484174+0000","last_peered":"2024-04-24T08:19:56.484174+0000","last_clean":"2024-04-24T08:19:56.484174+0000","last_became_active":"2024-04-24T08:18:47.310939+0000","last_became_peered":"2024-04-24T08:18:47.310939+0000","last_unstale":"2024-04-24T08:19:56.484174+0000","last_undegraded":"2024-04-24T08:19:56.484174+0000","last_fullsized":"2024-04-24T08:19:56.484174+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:42.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584998+0000","last_change":"2024-04-24T08:18:46.450596+0000","last_active":"2024-04-24T08:18:57.584998+0000","last_peered":"2024-04-24T08:18:57.584998+0000","last_clean":"2024-04-24T08:18:57.584998+0000","last_became_active":"2024-04-24T08:18:46.450226+0000","last_became_peered":"2024-04-24T08:18:46.450226+0000","last_unstale":"2024-04-24T08:18:57.584998+0000","last_undegraded":"2024-04-24T08:18:57.584998+0000","last_fullsized":"2024-04-24T08:18:57.584998+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:05.722507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.447318+0000","last_change":"2024-04-24T08:18:47.295266+0000","last_active":"2024-04-24T08:19:56.447318+0000","last_peered":"2024-04-24T08:19:56.447318+0000","last_clean":"2024-04-24T08:19:56.447318+0000","last_became_active":"2024-04-24T08:18:47.294963+0000","last_became_peered":"2024-04-24T08:18:47.294963+0000","last_unstale":"2024-04-24T08:19:56.447318+0000","last_undegraded":"2024-04-24T08:19:56.447318+0000","last_fullsized":"2024-04-24T08:19:56.447318+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:07.638921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'31","reported_seq":217,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.652961+0000","last_change":"2024-04-24T08:18:46.435524+0000","last_active":"2024-04-24T08:20:10.652961+0000","last_peered":"2024-04-24T08:20:10.652961+0000","last_clean":"2024-04-24T08:20:10.652961+0000","last_became_active":"2024-04-24T08:18:46.435212+0000","last_became_peered":"2024-04-24T08:18:46.435212+0000","last_unstale":"2024-04-24T08:20:10.652961+0000","last_undegraded":"2024-04-24T08:20:10.652961+0000","last_fullsized":"2024-04-24T08:20:10.652961+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:45.575905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":167,"num_read_kb":143,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.454437+0000","last_change":"2024-04-24T08:18:47.293648+0000","last_active":"2024-04-24T08:19:56.454437+0000","last_peered":"2024-04-24T08:19:56.454437+0000","last_clean":"2024-04-24T08:19:56.454437+0000","last_became_active":"2024-04-24T08:18:47.293260+0000","last_became_peered":"2024-04-24T08:18:47.293260+0000","last_unstale":"2024-04-24T08:19:56.454437+0000","last_undegraded":"2024-04-24T08:19:56.454437+0000","last_fullsized":"2024-04-24T08:19:56.454437+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:36.462606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577836+0000","last_change":"2024-04-24T08:18:46.441401+0000","last_active":"2024-04-24T08:18:57.577836+0000","last_peered":"2024-04-24T08:18:57.577836+0000","last_clean":"2024-04-24T08:18:57.577836+0000","last_became_active":"2024-04-24T08:18:46.440985+0000","last_became_peered":"2024-04-24T08:18:46.440985+0000","last_unstale":"2024-04-24T08:18:57.577836+0000","last_undegraded":"2024-04-24T08:18:57.577836+0000","last_fullsized":"2024-04-24T08:18:57.577836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:45.201518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.476644+0000","last_change":"2024-04-24T08:18:47.320928+0000","last_active":"2024-04-24T08:19:56.476644+0000","last_peered":"2024-04-24T08:19:56.476644+0000","last_clean":"2024-04-24T08:19:56.476644+0000","last_became_active":"2024-04-24T08:18:47.320515+0000","last_became_peered":"2024-04-24T08:18:47.320515+0000","last_unstale":"2024-04-24T08:19:56.476644+0000","last_undegraded":"2024-04-24T08:19:56.476644+0000","last_fullsized":"2024-04-24T08:19:56.476644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:55:42.899438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584699+0000","last_change":"2024-04-24T08:18:46.437677+0000","last_active":"2024-04-24T08:18:57.584699+0000","last_peered":"2024-04-24T08:18:57.584699+0000","last_clean":"2024-04-24T08:18:57.584699+0000","last_became_active":"2024-04-24T08:18:46.437364+0000","last_became_peered":"2024-04-24T08:18:46.437364+0000","last_unstale":"2024-04-24T08:18:57.584699+0000","last_undegraded":"2024-04-24T08:18:57.584699+0000","last_fullsized":"2024-04-24T08:18:57.584699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:25:55.408887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.466029+0000","last_change":"2024-04-24T08:18:47.298396+0000","last_active":"2024-04-24T08:19:56.466029+0000","last_peered":"2024-04-24T08:19:56.466029+0000","last_clean":"2024-04-24T08:19:56.466029+0000","last_became_active":"2024-04-24T08:18:47.298021+0000","last_became_peered":"2024-04-24T08:18:47.298021+0000","last_unstale":"2024-04-24T08:19:56.466029+0000","last_undegraded":"2024-04-24T08:19:56.466029+0000","last_fullsized":"2024-04-24T08:19:56.466029+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:24.015897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":106,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:12.705698+0000","last_change":"2024-04-24T08:18:46.436612+0000","last_active":"2024-04-24T08:20:12.705698+0000","last_peered":"2024-04-24T08:20:12.705698+0000","last_clean":"2024-04-24T08:20:12.705698+0000","last_became_active":"2024-04-24T08:18:46.436211+0000","last_became_peered":"2024-04-24T08:18:46.436211+0000","last_unstale":"2024-04-24T08:20:12.705698+0000","last_undegraded":"2024-04-24T08:20:12.705698+0000","last_fullsized":"2024-04-24T08:20:12.705698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:49:05.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.453480+0000","last_change":"2024-04-24T08:18:47.290849+0000","last_active":"2024-04-24T08:19:56.453480+0000","last_peered":"2024-04-24T08:19:56.453480+0000","last_clean":"2024-04-24T08:19:56.453480+0000","last_became_active":"2024-04-24T08:18:47.290514+0000","last_became_peered":"2024-04-24T08:18:47.290514+0000","last_unstale":"2024-04-24T08:19:56.453480+0000","last_undegraded":"2024-04-24T08:19:56.453480+0000","last_fullsized":"2024-04-24T08:19:56.453480+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:39:12.527477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.579682+0000","last_change":"2024-04-24T08:18:46.451815+0000","last_active":"2024-04-24T08:18:57.579682+0000","last_peered":"2024-04-24T08:18:57.579682+0000","last_clean":"2024-04-24T08:18:57.579682+0000","last_became_active":"2024-04-24T08:18:46.451478+0000","last_became_peered":"2024-04-24T08:18:46.451478+0000","last_unstale":"2024-04-24T08:18:57.579682+0000","last_undegraded":"2024-04-24T08:18:57.579682+0000","last_fullsized":"2024-04-24T08:18:57.579682+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:54:22.907143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.446838+0000","last_change":"2024-04-24T08:18:47.295181+0000","last_active":"2024-04-24T08:19:56.446838+0000","last_peered":"2024-04-24T08:19:56.446838+0000","last_clean":"2024-04-24T08:19:56.446838+0000","last_became_active":"2024-04-24T08:18:47.294842+0000","last_became_peered":"2024-04-24T08:18:47.294842+0000","last_unstale":"2024-04-24T08:19:56.446838+0000","last_undegraded":"2024-04-24T08:19:56.446838+0000","last_fullsized":"2024-04-24T08:19:56.446838+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:37.058025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.580128+0000","last_change":"2024-04-24T08:18:46.437967+0000","last_active":"2024-04-24T08:18:57.580128+0000","last_peered":"2024-04-24T08:18:57.580128+0000","last_clean":"2024-04-24T08:18:57.580128+0000","last_became_active":"2024-04-24T08:18:46.437617+0000","last_became_peered":"2024-04-24T08:18:46.437617+0000","last_unstale":"2024-04-24T08:18:57.580128+0000","last_undegraded":"2024-04-24T08:18:57.580128+0000","last_fullsized":"2024-04-24T08:18:57.580128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:07.303827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.443717+0000","last_change":"2024-04-24T08:18:57.726323+0000","last_active":"2024-04-24T08:19:56.443717+0000","last_peered":"2024-04-24T08:19:56.443717+0000","last_clean":"2024-04-24T08:19:56.443717+0000","last_became_active":"2024-04-24T08:18:57.725924+0000","last_became_peered":"2024-04-24T08:18:57.725924+0000","last_unstale":"2024-04-24T08:19:56.443717+0000","last_undegraded":"2024-04-24T08:19:56.443717+0000","last_fullsized":"2024-04-24T08:19:56.443717+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:15:33.756804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.591169+0000","last_change":"2024-04-24T08:18:57.591169+0000","last_active":"2024-04-24T08:18:57.591169+0000","last_peered":"2024-04-24T08:18:57.591169+0000","last_clean":"2024-04-24T08:18:57.591169+0000","last_became_active":"2024-04-24T08:18:57.590300+0000","last_became_peered":"2024-04-24T08:18:57.590300+0000","last_unstale":"2024-04-24T08:18:57.591169+0000","last_undegraded":"2024-04-24T08:18:57.591169+0000","last_fullsized":"2024-04-24T08:18:57.591169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:13.632780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.473340+0000","last_change":"2024-04-24T08:18:47.316791+0000","last_active":"2024-04-24T08:19:56.473340+0000","last_peered":"2024-04-24T08:19:56.473340+0000","last_clean":"2024-04-24T08:19:56.473340+0000","last_became_active":"2024-04-24T08:18:47.316486+0000","last_became_peered":"2024-04-24T08:18:47.316486+0000","last_unstale":"2024-04-24T08:19:56.473340+0000","last_undegraded":"2024-04-24T08:19:56.473340+0000","last_fullsized":"2024-04-24T08:19:56.473340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:36:37.190732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.582062+0000","last_change":"2024-04-24T08:18:46.432831+0000","last_active":"2024-04-24T08:18:57.582062+0000","last_peered":"2024-04-24T08:18:57.582062+0000","last_clean":"2024-04-24T08:18:57.582062+0000","last_became_active":"2024-04-24T08:18:46.432483+0000","last_became_peered":"2024-04-24T08:18:46.432483+0000","last_unstale":"2024-04-24T08:18:57.582062+0000","last_undegraded":"2024-04-24T08:18:57.582062+0000","last_fullsized":"2024-04-24T08:18:57.582062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:43:24.802099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.451323+0000","last_change":"2024-04-24T08:18:47.309408+0000","last_active":"2024-04-24T08:19:56.451323+0000","last_peered":"2024-04-24T08:19:56.451323+0000","last_clean":"2024-04-24T08:19:56.451323+0000","last_became_active":"2024-04-24T08:18:47.309083+0000","last_became_peered":"2024-04-24T08:18:47.309083+0000","last_unstale":"2024-04-24T08:19:56.451323+0000","last_undegraded":"2024-04-24T08:19:56.451323+0000","last_fullsized":"2024-04-24T08:19:56.451323+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:15:44.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.436736+0000","last_change":"2024-04-24T08:18:47.291060+0000","last_active":"2024-04-24T08:19:56.436736+0000","last_peered":"2024-04-24T08:19:56.436736+0000","last_clean":"2024-04-24T08:19:56.436736+0000","last_became_active":"2024-04-24T08:18:47.290635+0000","last_became_peered":"2024-04-24T08:18:47.290635+0000","last_unstale":"2024-04-24T08:19:56.436736+0000","last_undegraded":"2024-04-24T08:19:56.436736+0000","last_fullsized":"2024-04-24T08:19:56.436736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:20.372853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":106,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:20:13.706985+0000","last_change":"2024-04-24T08:18:46.449850+0000","last_active":"2024-04-24T08:20:13.706985+0000","last_peered":"2024-04-24T08:20:13.706985+0000","last_clean":"2024-04-24T08:20:13.706985+0000","last_became_active":"2024-04-24T08:18:46.448486+0000","last_became_peered":"2024-04-24T08:18:46.448486+0000","last_unstale":"2024-04-24T08:20:13.706985+0000","last_undegraded":"2024-04-24T08:20:13.706985+0000","last_fullsized":"2024-04-24T08:20:13.706985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:54.611867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.466651+0000","last_change":"2024-04-24T08:18:47.328051+0000","last_active":"2024-04-24T08:19:56.466651+0000","last_peered":"2024-04-24T08:19:56.466651+0000","last_clean":"2024-04-24T08:19:56.466651+0000","last_became_active":"2024-04-24T08:18:47.327474+0000","last_became_peered":"2024-04-24T08:18:47.327474+0000","last_unstale":"2024-04-24T08:19:56.466651+0000","last_undegraded":"2024-04-24T08:19:56.466651+0000","last_fullsized":"2024-04-24T08:19:56.466651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:07.577741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.724791+0000","last_change":"2024-04-24T08:18:46.444375+0000","last_active":"2024-04-24T08:18:57.724791+0000","last_peered":"2024-04-24T08:18:57.724791+0000","last_clean":"2024-04-24T08:18:57.724791+0000","last_became_active":"2024-04-24T08:18:46.443963+0000","last_became_peered":"2024-04-24T08:18:46.443963+0000","last_unstale":"2024-04-24T08:18:57.724791+0000","last_undegraded":"2024-04-24T08:18:57.724791+0000","last_fullsized":"2024-04-24T08:18:57.724791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:22.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.581987+0000","last_change":"2024-04-24T08:17:29.632219+0000","last_active":"2024-04-24T08:18:57.581987+0000","last_peered":"2024-04-24T08:18:57.581987+0000","last_clean":"2024-04-24T08:18:57.581987+0000","last_became_active":"2024-04-24T08:17:29.631841+0000","last_became_peered":"2024-04-24T08:17:29.631841+0000","last_unstale":"2024-04-24T08:18:57.581987+0000","last_undegraded":"2024-04-24T08:18:57.581987+0000","last_fullsized":"2024-04-24T08:18:57.581987+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_clean_scrub_stamp":"2024-04-24T08:13:41.228801+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:02:55.415959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.460885+0000","last_change":"2024-04-24T08:18:47.301876+0000","last_active":"2024-04-24T08:19:56.460885+0000","last_peered":"2024-04-24T08:19:56.460885+0000","last_clean":"2024-04-24T08:19:56.460885+0000","last_became_active":"2024-04-24T08:18:47.301545+0000","last_became_peered":"2024-04-24T08:18:47.301545+0000","last_unstale":"2024-04-24T08:19:56.460885+0000","last_undegraded":"2024-04-24T08:19:56.460885+0000","last_fullsized":"2024-04-24T08:19:56.460885+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:34.650014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.581866+0000","last_change":"2024-04-24T08:18:46.437915+0000","last_active":"2024-04-24T08:18:57.581866+0000","last_peered":"2024-04-24T08:18:57.581866+0000","last_clean":"2024-04-24T08:18:57.581866+0000","last_became_active":"2024-04-24T08:18:46.437567+0000","last_became_peered":"2024-04-24T08:18:46.437567+0000","last_unstale":"2024-04-24T08:18:57.581866+0000","last_undegraded":"2024-04-24T08:18:57.581866+0000","last_fullsized":"2024-04-24T08:18:57.581866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:36.801535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.580177+0000","last_change":"2024-04-24T08:18:46.451834+0000","last_active":"2024-04-24T08:18:57.580177+0000","last_peered":"2024-04-24T08:18:57.580177+0000","last_clean":"2024-04-24T08:18:57.580177+0000","last_became_active":"2024-04-24T08:18:46.451505+0000","last_became_peered":"2024-04-24T08:18:46.451505+0000","last_unstale":"2024-04-24T08:18:57.580177+0000","last_undegraded":"2024-04-24T08:18:57.580177+0000","last_fullsized":"2024-04-24T08:18:57.580177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:26.468176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.462457+0000","last_change":"2024-04-24T08:18:47.310490+0000","last_active":"2024-04-24T08:19:56.462457+0000","last_peered":"2024-04-24T08:19:56.462457+0000","last_clean":"2024-04-24T08:19:56.462457+0000","last_became_active":"2024-04-24T08:18:47.309615+0000","last_became_peered":"2024-04-24T08:18:47.309615+0000","last_unstale":"2024-04-24T08:19:56.462457+0000","last_undegraded":"2024-04-24T08:19:56.462457+0000","last_fullsized":"2024-04-24T08:19:56.462457+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:22:55.063485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'11","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:01.461007+0000","last_change":"2024-04-24T08:18:57.602098+0000","last_active":"2024-04-24T08:19:01.461007+0000","last_peered":"2024-04-24T08:19:01.461007+0000","last_clean":"2024-04-24T08:19:01.461007+0000","last_became_active":"2024-04-24T08:18:57.595899+0000","last_became_peered":"2024-04-24T08:18:57.595899+0000","last_unstale":"2024-04-24T08:19:01.461007+0000","last_undegraded":"2024-04-24T08:19:01.461007+0000","last_fullsized":"2024-04-24T08:19:01.461007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:31.951803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.469016+0000","last_change":"2024-04-24T08:18:47.319022+0000","last_active":"2024-04-24T08:19:56.469016+0000","last_peered":"2024-04-24T08:19:56.469016+0000","last_clean":"2024-04-24T08:19:56.469016+0000","last_became_active":"2024-04-24T08:18:47.318167+0000","last_became_peered":"2024-04-24T08:18:47.318167+0000","last_unstale":"2024-04-24T08:19:56.469016+0000","last_undegraded":"2024-04-24T08:19:56.469016+0000","last_fullsized":"2024-04-24T08:19:56.469016+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:18.980856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.725014+0000","last_change":"2024-04-24T08:18:46.433018+0000","last_active":"2024-04-24T08:18:57.725014+0000","last_peered":"2024-04-24T08:18:57.725014+0000","last_clean":"2024-04-24T08:18:57.725014+0000","last_became_active":"2024-04-24T08:18:46.432681+0000","last_became_peered":"2024-04-24T08:18:46.432681+0000","last_unstale":"2024-04-24T08:18:57.725014+0000","last_undegraded":"2024-04-24T08:18:57.725014+0000","last_fullsized":"2024-04-24T08:18:57.725014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:53.691684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.485040+0000","last_change":"2024-04-24T08:18:47.326055+0000","last_active":"2024-04-24T08:19:56.485040+0000","last_peered":"2024-04-24T08:19:56.485040+0000","last_clean":"2024-04-24T08:19:56.485040+0000","last_became_active":"2024-04-24T08:18:47.324956+0000","last_became_peered":"2024-04-24T08:18:47.324956+0000","last_unstale":"2024-04-24T08:19:56.485040+0000","last_undegraded":"2024-04-24T08:19:56.485040+0000","last_fullsized":"2024-04-24T08:19:56.485040+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:44:37.431264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.575383+0000","last_change":"2024-04-24T08:18:46.450710+0000","last_active":"2024-04-24T08:18:57.575383+0000","last_peered":"2024-04-24T08:18:57.575383+0000","last_clean":"2024-04-24T08:18:57.575383+0000","last_became_active":"2024-04-24T08:18:46.450391+0000","last_became_peered":"2024-04-24T08:18:46.450391+0000","last_unstale":"2024-04-24T08:18:57.575383+0000","last_undegraded":"2024-04-24T08:18:57.575383+0000","last_fullsized":"2024-04-24T08:18:57.575383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:19.905524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.414318+0000","last_change":"2024-04-24T08:18:47.292114+0000","last_active":"2024-04-24T08:19:56.414318+0000","last_peered":"2024-04-24T08:19:56.414318+0000","last_clean":"2024-04-24T08:19:56.414318+0000","last_became_active":"2024-04-24T08:18:47.291787+0000","last_became_peered":"2024-04-24T08:18:47.291787+0000","last_unstale":"2024-04-24T08:19:56.414318+0000","last_undegraded":"2024-04-24T08:19:56.414318+0000","last_fullsized":"2024-04-24T08:19:56.414318+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:37.020368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.593098+0000","last_change":"2024-04-24T08:18:46.446027+0000","last_active":"2024-04-24T08:18:57.593098+0000","last_peered":"2024-04-24T08:18:57.593098+0000","last_clean":"2024-04-24T08:18:57.593098+0000","last_became_active":"2024-04-24T08:18:46.445634+0000","last_became_peered":"2024-04-24T08:18:46.445634+0000","last_unstale":"2024-04-24T08:18:57.593098+0000","last_undegraded":"2024-04-24T08:18:57.593098+0000","last_fullsized":"2024-04-24T08:18:57.593098+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:46.208560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.450275+0000","last_change":"2024-04-24T08:18:47.307800+0000","last_active":"2024-04-24T08:19:56.450275+0000","last_peered":"2024-04-24T08:19:56.450275+0000","last_clean":"2024-04-24T08:19:56.450275+0000","last_became_active":"2024-04-24T08:18:47.307501+0000","last_became_peered":"2024-04-24T08:18:47.307501+0000","last_unstale":"2024-04-24T08:19:56.450275+0000","last_undegraded":"2024-04-24T08:19:56.450275+0000","last_fullsized":"2024-04-24T08:19:56.450275+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:04:05.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577702+0000","last_change":"2024-04-24T08:18:46.442811+0000","last_active":"2024-04-24T08:18:57.577702+0000","last_peered":"2024-04-24T08:18:57.577702+0000","last_clean":"2024-04-24T08:18:57.577702+0000","last_became_active":"2024-04-24T08:18:46.442479+0000","last_became_peered":"2024-04-24T08:18:46.442479+0000","last_unstale":"2024-04-24T08:18:57.577702+0000","last_undegraded":"2024-04-24T08:18:57.577702+0000","last_fullsized":"2024-04-24T08:18:57.577702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:24:07.236529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.478705+0000","last_change":"2024-04-24T08:18:47.325046+0000","last_active":"2024-04-24T08:19:56.478705+0000","last_peered":"2024-04-24T08:19:56.478705+0000","last_clean":"2024-04-24T08:19:56.478705+0000","last_became_active":"2024-04-24T08:18:47.324720+0000","last_became_peered":"2024-04-24T08:18:47.324720+0000","last_unstale":"2024-04-24T08:19:56.478705+0000","last_undegraded":"2024-04-24T08:19:56.478705+0000","last_fullsized":"2024-04-24T08:19:56.478705+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:11.240388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.579744+0000","last_change":"2024-04-24T08:18:46.440572+0000","last_active":"2024-04-24T08:18:57.579744+0000","last_peered":"2024-04-24T08:18:57.579744+0000","last_clean":"2024-04-24T08:18:57.579744+0000","last_became_active":"2024-04-24T08:18:46.440216+0000","last_became_peered":"2024-04-24T08:18:46.440216+0000","last_unstale":"2024-04-24T08:18:57.579744+0000","last_undegraded":"2024-04-24T08:18:57.579744+0000","last_fullsized":"2024-04-24T08:18:57.579744+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:51.352874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.463196+0000","last_change":"2024-04-24T08:18:47.293465+0000","last_active":"2024-04-24T08:19:56.463196+0000","last_peered":"2024-04-24T08:19:56.463196+0000","last_clean":"2024-04-24T08:19:56.463196+0000","last_became_active":"2024-04-24T08:18:47.293062+0000","last_became_peered":"2024-04-24T08:18:47.293062+0000","last_unstale":"2024-04-24T08:19:56.463196+0000","last_undegraded":"2024-04-24T08:19:56.463196+0000","last_fullsized":"2024-04-24T08:19:56.463196+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:36.029511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.578157+0000","last_change":"2024-04-24T08:18:46.451007+0000","last_active":"2024-04-24T08:18:57.578157+0000","last_peered":"2024-04-24T08:18:57.578157+0000","last_clean":"2024-04-24T08:18:57.578157+0000","last_became_active":"2024-04-24T08:18:46.450676+0000","last_became_peered":"2024-04-24T08:18:46.450676+0000","last_unstale":"2024-04-24T08:18:57.578157+0000","last_undegraded":"2024-04-24T08:18:57.578157+0000","last_fullsized":"2024-04-24T08:18:57.578157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:40:00.329311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.456457+0000","last_change":"2024-04-24T08:18:47.293859+0000","last_active":"2024-04-24T08:19:56.456457+0000","last_peered":"2024-04-24T08:19:56.456457+0000","last_clean":"2024-04-24T08:19:56.456457+0000","last_became_active":"2024-04-24T08:18:47.293509+0000","last_became_peered":"2024-04-24T08:18:47.293509+0000","last_unstale":"2024-04-24T08:19:56.456457+0000","last_undegraded":"2024-04-24T08:19:56.456457+0000","last_fullsized":"2024-04-24T08:19:56.456457+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:41.224037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.584722+0000","last_change":"2024-04-24T08:18:46.436661+0000","last_active":"2024-04-24T08:18:57.584722+0000","last_peered":"2024-04-24T08:18:57.584722+0000","last_clean":"2024-04-24T08:18:57.584722+0000","last_became_active":"2024-04-24T08:18:46.436353+0000","last_became_peered":"2024-04-24T08:18:46.436353+0000","last_unstale":"2024-04-24T08:18:57.584722+0000","last_undegraded":"2024-04-24T08:18:57.584722+0000","last_fullsized":"2024-04-24T08:18:57.584722+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:57:25.559706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.451593+0000","last_change":"2024-04-24T08:18:57.597302+0000","last_active":"2024-04-24T08:19:56.451593+0000","last_peered":"2024-04-24T08:19:56.451593+0000","last_clean":"2024-04-24T08:19:56.451593+0000","last_became_active":"2024-04-24T08:18:57.591420+0000","last_became_peered":"2024-04-24T08:18:57.591420+0000","last_unstale":"2024-04-24T08:19:56.451593+0000","last_undegraded":"2024-04-24T08:19:56.451593+0000","last_fullsized":"2024-04-24T08:19:56.451593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:03:34.286436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:18:57.577353+0000","last_change":"2024-04-24T08:18:46.432499+0000","last_active":"2024-04-24T08:18:57.577353+0000","last_peered":"2024-04-24T08:18:57.577353+0000","last_clean":"2024-04-24T08:18:57.577353+0000","last_became_active":"2024-04-24T08:18:46.432085+0000","last_became_peered":"2024-04-24T08:18:46.432085+0000","last_unstale":"2024-04-24T08:18:57.577353+0000","last_undegraded":"2024-04-24T08:18:57.577353+0000","last_fullsized":"2024-04-24T08:18:57.577353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:03.794091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.455423+0000","last_change":"2024-04-24T08:18:47.291193+0000","last_active":"2024-04-24T08:19:56.455423+0000","last_peered":"2024-04-24T08:19:56.455423+0000","last_clean":"2024-04-24T08:19:56.455423+0000","last_became_active":"2024-04-24T08:18:47.289903+0000","last_became_peered":"2024-04-24T08:18:47.289903+0000","last_unstale":"2024-04-24T08:19:56.455423+0000","last_undegraded":"2024-04-24T08:19:56.455423+0000","last_fullsized":"2024-04-24T08:19:56.455423+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:35:18.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.439071+0000","last_change":"2024-04-24T08:18:47.292628+0000","last_active":"2024-04-24T08:19:56.439071+0000","last_peered":"2024-04-24T08:19:56.439071+0000","last_clean":"2024-04-24T08:19:56.439071+0000","last_became_active":"2024-04-24T08:18:47.292060+0000","last_became_peered":"2024-04-24T08:18:47.292060+0000","last_unstale":"2024-04-24T08:19:56.439071+0000","last_undegraded":"2024-04-24T08:19:56.439071+0000","last_fullsized":"2024-04-24T08:19:56.439071+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:53.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.445363+0000","last_change":"2024-04-24T08:18:47.297121+0000","last_active":"2024-04-24T08:19:56.445363+0000","last_peered":"2024-04-24T08:19:56.445363+0000","last_clean":"2024-04-24T08:19:56.445363+0000","last_became_active":"2024-04-24T08:18:47.296787+0000","last_became_peered":"2024-04-24T08:18:47.296787+0000","last_unstale":"2024-04-24T08:19:56.445363+0000","last_undegraded":"2024-04-24T08:19:56.445363+0000","last_fullsized":"2024-04-24T08:19:56.445363+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:44:10.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.473806+0000","last_change":"2024-04-24T08:18:47.320669+0000","last_active":"2024-04-24T08:19:56.473806+0000","last_peered":"2024-04-24T08:19:56.473806+0000","last_clean":"2024-04-24T08:19:56.473806+0000","last_became_active":"2024-04-24T08:18:47.320287+0000","last_became_peered":"2024-04-24T08:18:47.320287+0000","last_unstale":"2024-04-24T08:19:56.473806+0000","last_undegraded":"2024-04-24T08:19:56.473806+0000","last_fullsized":"2024-04-24T08:19:56.473806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:39:03.275189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.481107+0000","last_change":"2024-04-24T08:18:57.585568+0000","last_active":"2024-04-24T08:19:56.481107+0000","last_peered":"2024-04-24T08:19:56.481107+0000","last_clean":"2024-04-24T08:19:56.481107+0000","last_became_active":"2024-04-24T08:18:57.584799+0000","last_became_peered":"2024-04-24T08:18:57.584799+0000","last_unstale":"2024-04-24T08:19:56.481107+0000","last_undegraded":"2024-04-24T08:19:56.481107+0000","last_fullsized":"2024-04-24T08:19:56.481107+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:47.202344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.461915+0000","last_change":"2024-04-24T08:18:47.316068+0000","last_active":"2024-04-24T08:19:56.461915+0000","last_peered":"2024-04-24T08:19:56.461915+0000","last_clean":"2024-04-24T08:19:56.461915+0000","last_became_active":"2024-04-24T08:18:47.315647+0000","last_became_peered":"2024-04-24T08:18:47.315647+0000","last_unstale":"2024-04-24T08:19:56.461915+0000","last_undegraded":"2024-04-24T08:19:56.461915+0000","last_fullsized":"2024-04-24T08:19:56.461915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:33.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.436673+0000","last_change":"2024-04-24T08:18:47.294966+0000","last_active":"2024-04-24T08:19:56.436673+0000","last_peered":"2024-04-24T08:19:56.436673+0000","last_clean":"2024-04-24T08:19:56.436673+0000","last_became_active":"2024-04-24T08:18:47.294672+0000","last_became_peered":"2024-04-24T08:18:47.294672+0000","last_unstale":"2024-04-24T08:19:56.436673+0000","last_undegraded":"2024-04-24T08:19:56.436673+0000","last_fullsized":"2024-04-24T08:19:56.436673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:37.359114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.443180+0000","last_change":"2024-04-24T08:18:47.294548+0000","last_active":"2024-04-24T08:19:56.443180+0000","last_peered":"2024-04-24T08:19:56.443180+0000","last_clean":"2024-04-24T08:19:56.443180+0000","last_became_active":"2024-04-24T08:18:47.293589+0000","last_became_peered":"2024-04-24T08:18:47.293589+0000","last_unstale":"2024-04-24T08:19:56.443180+0000","last_undegraded":"2024-04-24T08:19:56.443180+0000","last_fullsized":"2024-04-24T08:19:56.443180+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:34:56.208453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.469129+0000","last_change":"2024-04-24T08:18:47.313156+0000","last_active":"2024-04-24T08:19:56.469129+0000","last_peered":"2024-04-24T08:19:56.469129+0000","last_clean":"2024-04-24T08:19:56.469129+0000","last_became_active":"2024-04-24T08:18:47.312829+0000","last_became_peered":"2024-04-24T08:18:47.312829+0000","last_unstale":"2024-04-24T08:19:56.469129+0000","last_undegraded":"2024-04-24T08:19:56.469129+0000","last_fullsized":"2024-04-24T08:19:56.469129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:28:36.638514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.467098+0000","last_change":"2024-04-24T08:18:47.320022+0000","last_active":"2024-04-24T08:19:56.467098+0000","last_peered":"2024-04-24T08:19:56.467098+0000","last_clean":"2024-04-24T08:19:56.467098+0000","last_became_active":"2024-04-24T08:18:47.319707+0000","last_became_peered":"2024-04-24T08:18:47.319707+0000","last_unstale":"2024-04-24T08:19:56.467098+0000","last_undegraded":"2024-04-24T08:19:56.467098+0000","last_fullsized":"2024-04-24T08:19:56.467098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:51.330874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.470982+0000","last_change":"2024-04-24T08:18:47.323187+0000","last_active":"2024-04-24T08:19:56.470982+0000","last_peered":"2024-04-24T08:19:56.470982+0000","last_clean":"2024-04-24T08:19:56.470982+0000","last_became_active":"2024-04-24T08:18:47.320258+0000","last_became_peered":"2024-04-24T08:18:47.320258+0000","last_unstale":"2024-04-24T08:19:56.470982+0000","last_undegraded":"2024-04-24T08:19:56.470982+0000","last_fullsized":"2024-04-24T08:19:56.470982+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:42:32.770950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.456892+0000","last_change":"2024-04-24T08:18:47.321926+0000","last_active":"2024-04-24T08:19:56.456892+0000","last_peered":"2024-04-24T08:19:56.456892+0000","last_clean":"2024-04-24T08:19:56.456892+0000","last_became_active":"2024-04-24T08:18:47.321584+0000","last_became_peered":"2024-04-24T08:18:47.321584+0000","last_unstale":"2024-04-24T08:19:56.456892+0000","last_undegraded":"2024-04-24T08:19:56.456892+0000","last_fullsized":"2024-04-24T08:19:56.456892+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:24.357157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.463681+0000","last_change":"2024-04-24T08:18:47.313310+0000","last_active":"2024-04-24T08:19:56.463681+0000","last_peered":"2024-04-24T08:19:56.463681+0000","last_clean":"2024-04-24T08:19:56.463681+0000","last_became_active":"2024-04-24T08:18:47.312968+0000","last_became_peered":"2024-04-24T08:18:47.312968+0000","last_unstale":"2024-04-24T08:19:56.463681+0000","last_undegraded":"2024-04-24T08:19:56.463681+0000","last_fullsized":"2024-04-24T08:19:56.463681+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:08.967577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-24T08:19:56.444201+0000","last_change":"2024-04-24T08:18:57.591543+0000","last_active":"2024-04-24T08:19:56.444201+0000","last_peered":"2024-04-24T08:19:56.444201+0000","last_clean":"2024-04-24T08:19:56.444201+0000","last_became_active":"2024-04-24T08:18:57.591043+0000","last_became_peered":"2024-04-24T08:18:57.591043+0000","last_unstale":"2024-04-24T08:19:56.444201+0000","last_undegraded":"2024-04-24T08:19:56.444201+0000","last_fullsized":"2024-04-24T08:19:56.444201+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:52:38.669469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":513,"num_read_kb":325,"num_write":167,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":196311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285599,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":469627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547235,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1679360,"data_stored":1122403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":469980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972401,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":469783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266741,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1593344,"data_stored":1061854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430267,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":469252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":530340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":470523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":532222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":469446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":469689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691905,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1605632,"data_stored":1061538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:20:18.833 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-24T08:20:18.833 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T08:20:18.833 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-24T08:20:18.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-24T08:20:19.316 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:19.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:19 smithi146 ceph-mon[27805]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:20:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:19 smithi146 ceph-mon[27805]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:19.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:19 smithi177 ceph-mon[33046]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:20:19.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:19 smithi177 ceph-mon[33046]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:19 smithi153 ceph-mon[32746]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:20:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:19 smithi153 ceph-mon[32746]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:21.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 -- 172.21.15.146:0/1934978205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406b1c0 msgr2=0x7f537406b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 --2- 172.21.15.146:0/1934978205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406b1c0 0x7f537406b620 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f535c009990 tx=0x7f535c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 -- 172.21.15.146:0/1934978205 shutdown_connections 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 --2- 172.21.15.146:0/1934978205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f537406bb60 0x7f537410f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 --2- 172.21.15.146:0/1934978205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5374108b90 0x7f5374108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 --2- 172.21.15.146:0/1934978205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406b1c0 0x7f537406b620 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 -- 172.21.15.146:0/1934978205 >> 172.21.15.146:0/1934978205 conn(0x7f53740789a0 msgr2=0x7f5374078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 -- 172.21.15.146:0/1934978205 shutdown_connections 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.488+0000 7f5378c00700 1 -- 172.21.15.146:0/1934978205 wait complete. 2024-04-24T08:20:21.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.489+0000 7f5378c00700 1 Processor -- start 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.489+0000 7f5378c00700 1 -- start start 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5378c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f537406b1c0 0x7f537419b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5378c00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406bb60 0x7f537419ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5378c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5374108b90 0x7f537419fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5378c00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5374112e30 con 0x7f537406bb60 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5378c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5374112cb0 con 0x7f5374108b90 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5378c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5374112fb0 con 0x7f537406b1c0 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5369400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406bb60 0x7f537419ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5369400700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406bb60 0x7f537419ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37752/0 (socket says 172.21.15.146:37752) 2024-04-24T08:20:21.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5369400700 1 -- 172.21.15.146:0/1521079733 learned_addr learned my addr 172.21.15.146:0/1521079733 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5371e00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f537406b1c0 0x7f537419b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5372800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5374108b90 0x7f537419fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5369400700 1 -- 172.21.15.146:0/1521079733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f537406b1c0 msgr2=0x7f537419b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.490+0000 7f5369400700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f537406b1c0 0x7f537419b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5369400700 1 -- 172.21.15.146:0/1521079733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5374108b90 msgr2=0x7f537419fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5369400700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5374108b90 0x7f537419fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5369400700 1 -- 172.21.15.146:0/1521079733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53741a0540 con 0x7f537406bb60 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5371e00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f537406b1c0 0x7f537419b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5372800700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5374108b90 0x7f537419fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5369400700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406bb60 0x7f537419ba70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f535c005910 tx=0x7f535c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f536b400700 1 -- 172.21.15.146:0/1521079733 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c027070 con 0x7f537406bb60 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f536b400700 1 -- 172.21.15.146:0/1521079733 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f535c003d70 con 0x7f537406bb60 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53741a07d0 con 0x7f537406bb60 2024-04-24T08:20:21.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f536b400700 1 -- 172.21.15.146:0/1521079733 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c00f460 con 0x7f537406bb60 2024-04-24T08:20:21.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.491+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53741a80b0 con 0x7f537406bb60 2024-04-24T08:20:21.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.492+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f537406aa70 con 0x7f537406bb60 2024-04-24T08:20:21.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.495+0000 7f536b400700 1 -- 172.21.15.146:0/1521079733 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f535c02a070 con 0x7f537406bb60 2024-04-24T08:20:21.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.497+0000 7f536b400700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53580a3c40 0x7f53580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.497+0000 7f536b400700 1 -- 172.21.15.146:0/1521079733 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f535c0d19d0 con 0x7f537406bb60 2024-04-24T08:20:21.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.497+0000 7f5371e00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53580a3c40 0x7f53580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.498+0000 7f5371e00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53580a3c40 0x7f53580a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5364009680 tx=0x7f5364009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.500+0000 7f536b400700 1 -- 172.21.15.146:0/1521079733 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f535c031070 con 0x7f537406bb60 2024-04-24T08:20:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:21 smithi146 ceph-mon[27805]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:21.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.782+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5374104240 con 0x7f537406bb60 2024-04-24T08:20:21.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.783+0000 7f536b400700 1 -- 172.21.15.146:0/1521079733 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5374104240 con 0x7f537406bb60 2024-04-24T08:20:21.785 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:21.785 INFO:teuthology.orchestra.run.smithi146.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T08:20:21.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53580a3c40 msgr2=0x7f53580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53580a3c40 0x7f53580a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5364009680 tx=0x7f5364009260 comp rx=0 tx=0).stop 2024-04-24T08:20:21.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406bb60 msgr2=0x7f537419ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406bb60 0x7f537419ba70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f535c005910 tx=0x7f535c00bab0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 shutdown_connections 2024-04-24T08:20:21.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5374108b90 0x7f537419fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f537406b1c0 0x7f537419b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f53580a3c40 0x7f53580a6100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 --2- 172.21.15.146:0/1521079733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f537406bb60 0x7f537419ba70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.786+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 >> 172.21.15.146:0/1521079733 conn(0x7f53740789a0 msgr2=0x7f537410dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.787+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 shutdown_connections 2024-04-24T08:20:21.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:21.787+0000 7f5378c00700 1 -- 172.21.15.146:0/1521079733 wait complete. 2024-04-24T08:20:21.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:21 smithi177 ceph-mon[33046]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:21.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:21 smithi153 ceph-mon[32746]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:22.552 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-24T08:20:22.559 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:22 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1521079733' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 -- 172.21.15.146:0/1782927221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 msgr2=0x7fd22c102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 --2- 172.21.15.146:0/1782927221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c102470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd220009900 tx=0x7fd22002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 -- 172.21.15.146:0/1782927221 shutdown_connections 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 --2- 172.21.15.146:0/1782927221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd22c103c00 0x7fd22c10a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 --2- 172.21.15.146:0/1782927221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd22c103240 0x7fd22c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 --2- 172.21.15.146:0/1782927221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c102470 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 -- 172.21.15.146:0/1782927221 >> 172.21.15.146:0/1782927221 conn(0x7fd22c0fd5e0 msgr2=0x7fd22c0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 -- 172.21.15.146:0/1782927221 shutdown_connections 2024-04-24T08:20:22.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.769+0000 7fd234200640 1 -- 172.21.15.146:0/1782927221 wait complete. 2024-04-24T08:20:22.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.770+0000 7fd234200640 1 Processor -- start 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.770+0000 7fd234200640 1 -- start start 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd234200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd234200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd22c103240 0x7fd22c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd234200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd22c103c00 0x7fd22c1a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd234200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c10dd30 con 0x7fd22c102050 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd234200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c10dbb0 con 0x7fd22c103c00 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd234200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c10deb0 con 0x7fd22c103240 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c19a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37772/0 (socket says 172.21.15.146:37772) 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 -- 172.21.15.146:0/2090772096 learned_addr learned my addr 172.21.15.146:0/2090772096 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd232200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd22c103c00 0x7fd22c1a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:22.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd230e00640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd22c103240 0x7fd22c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 -- 172.21.15.146:0/2090772096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd22c103240 msgr2=0x7fd22c19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd22c103240 0x7fd22c19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 -- 172.21.15.146:0/2090772096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd22c103c00 msgr2=0x7fd22c1a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd22c103c00 0x7fd22c1a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd231800640 1 -- 172.21.15.146:0/2090772096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22c1a2170 con 0x7fd22c102050 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.771+0000 7fd232200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd22c103c00 0x7fd22c1a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.772+0000 7fd231800640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c19a550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd220031440 tx=0x7fd220032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.772+0000 7fd22a000640 1 -- 172.21.15.146:0/2090772096 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22003f030 con 0x7fd22c102050 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.772+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd22c1a2400 con 0x7fd22c102050 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.772+0000 7fd22a000640 1 -- 172.21.15.146:0/2090772096 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd220003ce0 con 0x7fd22c102050 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.772+0000 7fd22a000640 1 -- 172.21.15.146:0/2090772096 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd220035070 con 0x7fd22c102050 2024-04-24T08:20:22.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.772+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd22c1a2940 con 0x7fd22c102050 2024-04-24T08:20:22.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.773+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd22c1a2ba0 con 0x7fd22c102050 2024-04-24T08:20:22.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.775+0000 7fd22a000640 1 -- 172.21.15.146:0/2090772096 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fd220042030 con 0x7fd22c102050 2024-04-24T08:20:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.777+0000 7fd22a000640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd2140a3bb0 0x7fd2140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.778+0000 7fd22a000640 1 -- 172.21.15.146:0/2090772096 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fd2200f0890 con 0x7fd22c102050 2024-04-24T08:20:22.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.778+0000 7fd230e00640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd2140a3bb0 0x7fd2140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:22.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.778+0000 7fd230e00640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd2140a3bb0 0x7fd2140a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd2180095f0 tx=0x7fd218005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:22.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.782+0000 7fd22a000640 1 -- 172.21.15.146:0/2090772096 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd2200bcbe0 con 0x7fd22c102050 2024-04-24T08:20:22.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:22 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1521079733' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T08:20:22.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:22 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1521079733' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T08:20:22.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.984+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd22c1a2d70 con 0x7fd22c102050 2024-04-24T08:20:22.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.986+0000 7fd22a000640 1 -- 172.21.15.146:0/2090772096 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7fd2200c0ee0 con 0x7fd22c102050 2024-04-24T08:20:22.988 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:22.988 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd2140a3bb0 msgr2=0x7fd2140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd2140a3bb0 0x7fd2140a6070 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd2180095f0 tx=0x7fd218005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 msgr2=0x7fd22c19a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c19a550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd220031440 tx=0x7fd220032ce0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 shutdown_connections 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd22c103240 0x7fd22c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fd2140a3bb0 0x7fd2140a6070 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd22c103c00 0x7fd22c1a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 --2- 172.21.15.146:0/2090772096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd22c102050 0x7fd22c19a550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 >> 172.21.15.146:0/2090772096 conn(0x7fd22c0fd5e0 msgr2=0x7fd22c074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 shutdown_connections 2024-04-24T08:20:22.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:22.989+0000 7fd234200640 1 -- 172.21.15.146:0/2090772096 wait complete. 2024-04-24T08:20:23.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:23.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:23.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 -- 172.21.15.146:0/3159980009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc102ac0 msgr2=0x7fc2dc104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/3159980009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc104eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc009ca0 tx=0x7fc2cc02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 -- 172.21.15.146:0/3159980009 shutdown_connections 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/3159980009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc1053f0 0x7fc2dc1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/3159980009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2dc107d20 0x7fc2dc10a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/3159980009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc104eb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 -- 172.21.15.146:0/3159980009 >> 172.21.15.146:0/3159980009 conn(0x7fc2dc0fc4b0 msgr2=0x7fc2dc0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 -- 172.21.15.146:0/3159980009 shutdown_connections 2024-04-24T08:20:23.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.218+0000 7fc2e0a00640 1 -- 172.21.15.146:0/3159980009 wait complete. 2024-04-24T08:20:23.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 Processor -- start 2024-04-24T08:20:23.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 -- start start 2024-04-24T08:20:23.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2dc1053f0 0x7fc2dc19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc107d20 0x7fc2dc1a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc10dd20 con 0x7fc2dc107d20 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc10dba0 con 0x7fc2dc1053f0 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.219+0000 7fc2e0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc10dea0 con 0x7fc2dc102ac0 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2da800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc107d20 0x7fc2dc1a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc19a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50682/0 (socket says 172.21.15.146:50682) 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 -- 172.21.15.146:0/1065644026 learned_addr learned my addr 172.21.15.146:0/1065644026 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9400640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2dc1053f0 0x7fc2dc19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 -- 172.21.15.146:0/1065644026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2dc1053f0 msgr2=0x7fc2dc19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2dc1053f0 0x7fc2dc19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 -- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc107d20 msgr2=0x7fc2dc1a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc107d20 0x7fc2dc1a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 -- 172.21.15.146:0/1065644026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc1a2210 con 0x7fc2dc102ac0 2024-04-24T08:20:23.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2da800640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc107d20 0x7fc2dc1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:23.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.220+0000 7fc2d9e00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc19a5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc000f80 tx=0x7fc2cc0319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:23.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.221+0000 7fc2d2a00640 1 -- 172.21.15.146:0/1065644026 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc003b70 con 0x7fc2dc102ac0 2024-04-24T08:20:23.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.221+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2dc1a24a0 con 0x7fc2dc102ac0 2024-04-24T08:20:23.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.221+0000 7fc2d2a00640 1 -- 172.21.15.146:0/1065644026 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2cc003d10 con 0x7fc2dc102ac0 2024-04-24T08:20:23.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.221+0000 7fc2d2a00640 1 -- 172.21.15.146:0/1065644026 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc02fb50 con 0x7fc2dc102ac0 2024-04-24T08:20:23.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.221+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc1a29f0 con 0x7fc2dc102ac0 2024-04-24T08:20:23.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.222+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc109870 con 0x7fc2dc102ac0 2024-04-24T08:20:23.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.224+0000 7fc2d2a00640 1 -- 172.21.15.146:0/1065644026 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fc2cc02fcf0 con 0x7fc2dc102ac0 2024-04-24T08:20:23.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.225+0000 7fc2d2a00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc2c00a3bb0 0x7fc2c00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.226+0000 7fc2d2a00640 1 -- 172.21.15.146:0/1065644026 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc2cc0f19c0 con 0x7fc2dc102ac0 2024-04-24T08:20:23.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.226+0000 7fc2d9400640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc2c00a3bb0 0x7fc2c00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.226+0000 7fc2d9400640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc2c00a3bb0 0x7fc2c00a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc19bb00 tx=0x7fc2c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:23.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.227+0000 7fc2d2a00640 1 -- 172.21.15.146:0/1065644026 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2cc0be360 con 0x7fc2dc102ac0 2024-04-24T08:20:23.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.438+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc2dc062c00 con 0x7fc2dc102ac0 2024-04-24T08:20:23.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.439+0000 7fc2d2a00640 1 -- 172.21.15.146:0/1065644026 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fc2cc0dce20 con 0x7fc2dc102ac0 2024-04-24T08:20:23.441 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:20:23.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc2c00a3bb0 msgr2=0x7fc2c00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc2c00a3bb0 0x7fc2c00a6070 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc19bb00 tx=0x7fc2c8005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:23.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc102ac0 msgr2=0x7fc2dc19a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc19a5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc000f80 tx=0x7fc2cc0319f0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 shutdown_connections 2024-04-24T08:20:23.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc2c00a3bb0 0x7fc2c00a6070 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2dc107d20 0x7fc2dc1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2dc1053f0 0x7fc2dc19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 --2- 172.21.15.146:0/1065644026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2dc102ac0 0x7fc2dc19a5f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 >> 172.21.15.146:0/1065644026 conn(0x7fc2dc0fc4b0 msgr2=0x7fc2dc102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:23.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 shutdown_connections 2024-04-24T08:20:23.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.442+0000 7fc2e0a00640 1 -- 172.21.15.146:0/1065644026 wait complete. 2024-04-24T08:20:23.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.670+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2542907624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 msgr2=0x7f3b18109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2542907624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b18109cd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b04009900 tx=0x7f3b0402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2542907624 shutdown_connections 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2542907624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b18109cd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2542907624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b1810a210 0x7f3b18110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2542907624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b18108660 0x7f3b18108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2542907624 >> 172.21.15.146:0/2542907624 conn(0x7f3b18077600 msgr2=0x7f3b18079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2542907624 shutdown_connections 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.671+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2542907624 wait complete. 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.672+0000 7f3b1fc00640 1 Processor -- start 2024-04-24T08:20:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.672+0000 7f3b1fc00640 1 -- start start 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.672+0000 7f3b1fc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b18108660 0x7f3b1819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1fc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b1819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1fc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b1810a210 0x7f3b181a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1fc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3b18114330 con 0x7f3b18108660 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1fc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3b181141b0 con 0x7f3b1810a210 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1fc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b181144b0 con 0x7f3b18109850 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b17e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b1819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b1810a210 0x7f3b181a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b18108660 0x7f3b1819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b17e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b1819eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50708/0 (socket says 172.21.15.146:50708) 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b17e00640 1 -- 172.21.15.146:0/2219458204 learned_addr learned my addr 172.21.15.146:0/2219458204 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1dc00640 1 -- 172.21.15.146:0/2219458204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 msgr2=0x7f3b1819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1dc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b1819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1dc00640 1 -- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b18108660 msgr2=0x7f3b1819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1dc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b18108660 0x7f3b1819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1dc00640 1 -- 172.21.15.146:0/2219458204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b181a6590 con 0x7f3b1810a210 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b17e00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b1819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.673+0000 7f3b1d200640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b18108660 0x7f3b1819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:23.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.674+0000 7f3b1dc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b1810a210 0x7f3b181a5f20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009fd0 tx=0x7f3b1000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.674+0000 7f3b15600640 1 -- 172.21.15.146:0/2219458204 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1000ce20 con 0x7f3b1810a210 2024-04-24T08:20:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.674+0000 7f3b15600640 1 -- 172.21.15.146:0/2219458204 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b10005020 con 0x7f3b1810a210 2024-04-24T08:20:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.674+0000 7f3b15600640 1 -- 172.21.15.146:0/2219458204 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10010720 con 0x7f3b1810a210 2024-04-24T08:20:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.674+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b181a6880 con 0x7f3b1810a210 2024-04-24T08:20:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.674+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b181a6dd0 con 0x7f3b1810a210 2024-04-24T08:20:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.675+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1810f980 con 0x7f3b1810a210 2024-04-24T08:20:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.677+0000 7f3b15600640 1 -- 172.21.15.146:0/2219458204 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f3b10005540 con 0x7f3b1810a210 2024-04-24T08:20:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.679+0000 7f3b15600640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b000a3bb0 0x7f3b000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.679+0000 7f3b15600640 1 -- 172.21.15.146:0/2219458204 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f3b100d0a10 con 0x7f3b1810a210 2024-04-24T08:20:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.679+0000 7f3b1d200640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b000a3bb0 0x7f3b000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.679+0000 7f3b1d200640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b000a3bb0 0x7f3b000a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3b181096b0 tx=0x7f3b0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:23.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.683+0000 7f3b15600640 1 -- 172.21.15.146:0/2219458204 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b1009d3b0 con 0x7f3b1810a210 2024-04-24T08:20:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:23 smithi146 ceph-mon[27805]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:23 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2090772096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:23 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1065644026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:23 smithi177 ceph-mon[33046]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:23 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2090772096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:23 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1065644026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:23.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.886+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3b18062c00 con 0x7f3b1810a210 2024-04-24T08:20:23.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.888+0000 7f3b15600640 1 -- 172.21.15.146:0/2219458204 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f3b1009ccb0 con 0x7f3b1810a210 2024-04-24T08:20:23.889 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b000a3bb0 msgr2=0x7f3b000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b000a3bb0 0x7f3b000a6070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3b181096b0 tx=0x7f3b0c005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b1810a210 msgr2=0x7f3b181a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b1810a210 0x7f3b181a5f20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009fd0 tx=0x7f3b1000ef10 comp rx=0 tx=0).stop 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 shutdown_connections 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b18109850 0x7f3b1819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f3b000a3bb0 0x7f3b000a6070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b1810a210 0x7f3b181a5f20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 --2- 172.21.15.146:0/2219458204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b18108660 0x7f3b1819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 >> 172.21.15.146:0/2219458204 conn(0x7f3b18077600 msgr2=0x7f3b1810cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 shutdown_connections 2024-04-24T08:20:23.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:23.890+0000 7f3b1fc00640 1 -- 172.21.15.146:0/2219458204 wait complete. 2024-04-24T08:20:23.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:23.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:23 smithi153 ceph-mon[32746]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:23 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2090772096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:23 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1065644026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 -- 172.21.15.146:0/3952241570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e8102990 msgr2=0x7f84e8104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 --2- 172.21.15.146:0/3952241570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e8102990 0x7f84e8104d80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009900 tx=0x7f84d802a410 comp rx=0 tx=0).stop 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 -- 172.21.15.146:0/3952241570 shutdown_connections 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 --2- 172.21.15.146:0/3952241570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84e8107bf0 0x7f84e810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 --2- 172.21.15.146:0/3952241570 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e81052c0 0x7f84e81076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 --2- 172.21.15.146:0/3952241570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e8102990 0x7f84e8104d80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 -- 172.21.15.146:0/3952241570 >> 172.21.15.146:0/3952241570 conn(0x7f84e80fc3a0 msgr2=0x7f84e80fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 -- 172.21.15.146:0/3952241570 shutdown_connections 2024-04-24T08:20:24.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.121+0000 7f84eda00640 1 -- 172.21.15.146:0/3952241570 wait complete. 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.122+0000 7f84eda00640 1 Processor -- start 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.122+0000 7f84eda00640 1 -- start start 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84eda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84e8102990 0x7f84e8198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84eda00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e81052c0 0x7f84e8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84eda00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8107bf0 0x7f84e819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84eda00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f84e810dba0 con 0x7f84e81052c0 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84eda00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84e810da20 con 0x7f84e8107bf0 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84eda00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84e810dd20 con 0x7f84e8102990 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84dfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e81052c0 0x7f84e8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84dfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e81052c0 0x7f84e8198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37824/0 (socket says 172.21.15.146:37824) 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84ed000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8107bf0 0x7f84e819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84dfe00640 1 -- 172.21.15.146:0/3095012637 learned_addr learned my addr 172.21.15.146:0/3095012637 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84e7e00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84e8102990 0x7f84e8198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84ed000640 1 -- 172.21.15.146:0/3095012637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84e8102990 msgr2=0x7f84e8198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.123+0000 7f84ed000640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84e8102990 0x7f84e8198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84ed000640 1 -- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e81052c0 msgr2=0x7f84e8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84ed000640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e81052c0 0x7f84e8198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84ed000640 1 -- 172.21.15.146:0/3095012637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e819fe30 con 0x7f84e8107bf0 2024-04-24T08:20:24.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84dfe00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e81052c0 0x7f84e8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:24.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84ed000640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8107bf0 0x7f84e819f7c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009fd0 tx=0x7f84e000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:24.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84e5600640 1 -- 172.21.15.146:0/3095012637 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e000ce20 con 0x7f84e8107bf0 2024-04-24T08:20:24.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e81a0120 con 0x7f84e8107bf0 2024-04-24T08:20:24.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84e5600640 1 -- 172.21.15.146:0/3095012637 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84e0005020 con 0x7f84e8107bf0 2024-04-24T08:20:24.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84e5600640 1 -- 172.21.15.146:0/3095012637 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e0010720 con 0x7f84e8107bf0 2024-04-24T08:20:24.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.124+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e81a0670 con 0x7f84e8107bf0 2024-04-24T08:20:24.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.125+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e80fde90 con 0x7f84e8107bf0 2024-04-24T08:20:24.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.128+0000 7f84e5600640 1 -- 172.21.15.146:0/3095012637 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f84e0005540 con 0x7f84e8107bf0 2024-04-24T08:20:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.129+0000 7f84e5600640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f84cc0a3bb0 0x7f84cc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.129+0000 7f84e5600640 1 -- 172.21.15.146:0/3095012637 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f84e00d0c40 con 0x7f84e8107bf0 2024-04-24T08:20:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.129+0000 7f84e7e00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f84cc0a3bb0 0x7f84cc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.130+0000 7f84e7e00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f84cc0a3bb0 0x7f84cc0a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f84d80314e0 tx=0x7f84d8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:24.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.131+0000 7f84e5600640 1 -- 172.21.15.146:0/3095012637 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84e009d5e0 con 0x7f84e8107bf0 2024-04-24T08:20:24.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.338+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84e802d620 con 0x7f84e8107bf0 2024-04-24T08:20:24.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.339+0000 7f84e5600640 1 -- 172.21.15.146:0/3095012637 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f84e009cee0 con 0x7f84e8107bf0 2024-04-24T08:20:24.341 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f84cc0a3bb0 msgr2=0x7f84cc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f84cc0a3bb0 0x7f84cc0a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f84d80314e0 tx=0x7f84d8000bc0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8107bf0 msgr2=0x7f84e819f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8107bf0 0x7f84e819f7c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009fd0 tx=0x7f84e000ef10 comp rx=0 tx=0).stop 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 shutdown_connections 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f84cc0a3bb0 0x7f84cc0a6070 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84e8107bf0 0x7f84e819f7c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84e81052c0 0x7f84e8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 --2- 172.21.15.146:0/3095012637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84e8102990 0x7f84e8198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 >> 172.21.15.146:0/3095012637 conn(0x7f84e80fc3a0 msgr2=0x7f84e8103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 shutdown_connections 2024-04-24T08:20:24.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.342+0000 7f84eda00640 1 -- 172.21.15.146:0/3095012637 wait complete. 2024-04-24T08:20:24.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:24.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.567+0000 7f92ab200640 1 -- 172.21.15.146:0/1595170420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac076040 msgr2=0x7f92ac074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.567+0000 7f92ab200640 1 --2- 172.21.15.146:0/1595170420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac076040 0x7f92ac074490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f929c009900 tx=0x7f929c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.568+0000 7f92ab200640 1 -- 172.21.15.146:0/1595170420 shutdown_connections 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.568+0000 7f92ab200640 1 --2- 172.21.15.146:0/1595170420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac109d60 0x7f92ac10c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.568+0000 7f92ab200640 1 --2- 172.21.15.146:0/1595170420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92ac074a60 0x7f92ac074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.568+0000 7f92ab200640 1 --2- 172.21.15.146:0/1595170420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac076040 0x7f92ac074490 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.568+0000 7f92ab200640 1 -- 172.21.15.146:0/1595170420 >> 172.21.15.146:0/1595170420 conn(0x7f92ac0fd410 msgr2=0x7f92ac0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.568+0000 7f92ab200640 1 -- 172.21.15.146:0/1595170420 shutdown_connections 2024-04-24T08:20:24.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.568+0000 7f92ab200640 1 -- 172.21.15.146:0/1595170420 wait complete. 2024-04-24T08:20:24.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 Processor -- start 2024-04-24T08:20:24.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 -- start start 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92ac074a60 0x7f92ac19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac076040 0x7f92ac19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac109d60 0x7f92ac1a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac10fdd0 con 0x7f92ac076040 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac10fc50 con 0x7f92ac074a60 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92ab200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac10ff50 con 0x7f92ac109d60 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92a1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac076040 0x7f92ac19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92a1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac076040 0x7f92ac19a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37852/0 (socket says 172.21.15.146:37852) 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.569+0000 7f92a1400640 1 -- 172.21.15.146:0/1918283819 learned_addr learned my addr 172.21.15.146:0/1918283819 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a9e00640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92ac074a60 0x7f92ac19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92aa800640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac109d60 0x7f92ac1a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a1400640 1 -- 172.21.15.146:0/1918283819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac109d60 msgr2=0x7f92ac1a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a1400640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac109d60 0x7f92ac1a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a1400640 1 -- 172.21.15.146:0/1918283819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92ac074a60 msgr2=0x7f92ac19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a1400640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92ac074a60 0x7f92ac19a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a1400640 1 -- 172.21.15.146:0/1918283819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac1a2040 con 0x7f92ac076040 2024-04-24T08:20:24.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a9e00640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92ac074a60 0x7f92ac19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:24.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a1400640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac076040 0x7f92ac19a960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f929400d770 tx=0x7f929400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:24.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.570+0000 7f92a3400640 1 -- 172.21.15.146:0/1918283819 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294017070 con 0x7f92ac076040 2024-04-24T08:20:24.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.571+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ac1a2330 con 0x7f92ac076040 2024-04-24T08:20:24.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.571+0000 7f92a3400640 1 -- 172.21.15.146:0/1918283819 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9294005500 con 0x7f92ac076040 2024-04-24T08:20:24.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.571+0000 7f92a3400640 1 -- 172.21.15.146:0/1918283819 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294004e00 con 0x7f92ac076040 2024-04-24T08:20:24.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.571+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ac1a2900 con 0x7f92ac076040 2024-04-24T08:20:24.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.572+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92ac108a90 con 0x7f92ac076040 2024-04-24T08:20:24.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.574+0000 7f92a3400640 1 -- 172.21.15.146:0/1918283819 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f9294004fa0 con 0x7f92ac076040 2024-04-24T08:20:24.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.575+0000 7f92a3400640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f92900a3bb0 0x7f92900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:24.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.575+0000 7f92a3400640 1 -- 172.21.15.146:0/1918283819 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f92940cfe00 con 0x7f92ac076040 2024-04-24T08:20:24.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.575+0000 7f92a9e00640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f92900a3bb0 0x7f92900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:24.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.576+0000 7f92a9e00640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f92900a3bb0 0x7f92900a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f929c031440 tx=0x7f929c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:24.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.579+0000 7f92a3400640 1 -- 172.21.15.146:0/1918283819 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f929409c150 con 0x7f92ac076040 2024-04-24T08:20:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:24 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2219458204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:24 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3095012637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:24 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2219458204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:24 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3095012637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:24.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.823+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92ac02d620 con 0x7f92ac076040 2024-04-24T08:20:24.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.825+0000 7f92a3400640 1 -- 172.21.15.146:0/1918283819 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11891 (secure 0 0 0) 0x7f92940a0450 con 0x7f92ac076040 2024-04-24T08:20:24.827 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:24.827 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:18:52.518710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:20:24.827 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f92900a3bb0 msgr2=0x7f92900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f92900a3bb0 0x7f92900a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f929c031440 tx=0x7f929c0315b0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac076040 msgr2=0x7f92ac19a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac076040 0x7f92ac19a960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f929400d770 tx=0x7f929400dc30 comp rx=0 tx=0).stop 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 shutdown_connections 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92ac109d60 0x7f92ac1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f92900a3bb0 0x7f92900a6070 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92ac076040 0x7f92ac19a960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 --2- 172.21.15.146:0/1918283819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92ac074a60 0x7f92ac19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 >> 172.21.15.146:0/1918283819 conn(0x7f92ac0fd410 msgr2=0x7f92ac105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 shutdown_connections 2024-04-24T08:20:24.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:24.828+0000 7f92ab200640 1 -- 172.21.15.146:0/1918283819 wait complete. 2024-04-24T08:20:24.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:24.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:24 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2219458204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:24.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:24 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3095012637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:25.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.044+0000 7f9d65e00640 1 -- 172.21.15.146:0/3074382316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d60109670 msgr2=0x7f9d60109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.044+0000 7f9d65e00640 1 --2- 172.21.15.146:0/3074382316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d60109670 0x7f9d60109af0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009900 tx=0x7f9d5002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.044+0000 7f9d65e00640 1 -- 172.21.15.146:0/3074382316 shutdown_connections 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.044+0000 7f9d65e00640 1 --2- 172.21.15.146:0/3074382316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d6010a030 0x7f9d601108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.044+0000 7f9d65e00640 1 --2- 172.21.15.146:0/3074382316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d60109670 0x7f9d60109af0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.044+0000 7f9d65e00640 1 --2- 172.21.15.146:0/3074382316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d60108480 0x7f9d601088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.044+0000 7f9d65e00640 1 -- 172.21.15.146:0/3074382316 >> 172.21.15.146:0/3074382316 conn(0x7f9d60077440 msgr2=0x7f9d60079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.045+0000 7f9d65e00640 1 -- 172.21.15.146:0/3074382316 shutdown_connections 2024-04-24T08:20:25.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.045+0000 7f9d65e00640 1 -- 172.21.15.146:0/3074382316 wait complete. 2024-04-24T08:20:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 Processor -- start 2024-04-24T08:20:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 -- start start 2024-04-24T08:20:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d60108480 0x7f9d6019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d60109670 0x7f9d6019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d6010a030 0x7f9d601a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d60114110 con 0x7f9d6010a030 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d60113f90 con 0x7f9d60109670 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9d60114290 con 0x7f9d60108480 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d6010a030 0x7f9d601a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d57e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d60109670 0x7f9d6019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d6010a030 0x7f9d601a5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37880/0 (socket says 172.21.15.146:37880) 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.046+0000 7f9d65400640 1 -- 172.21.15.146:0/4028334085 learned_addr learned my addr 172.21.15.146:0/4028334085 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d64a00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d60108480 0x7f9d6019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d57e00640 1 -- 172.21.15.146:0/4028334085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d60108480 msgr2=0x7f9d6019e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d57e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d60108480 0x7f9d6019e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d57e00640 1 -- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d6010a030 msgr2=0x7f9d601a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d57e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d6010a030 0x7f9d601a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d57e00640 1 -- 172.21.15.146:0/4028334085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d601a6340 con 0x7f9d60109670 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d65400640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d6010a030 0x7f9d601a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d57e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d60109670 0x7f9d6019ecb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d50031440 tx=0x7f9d50004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d5e000640 1 -- 172.21.15.146:0/4028334085 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5003f070 con 0x7f9d60109670 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d5e000640 1 -- 172.21.15.146:0/4028334085 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d50003cd0 con 0x7f9d60109670 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d601a65d0 con 0x7f9d60109670 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.047+0000 7f9d5e000640 1 -- 172.21.15.146:0/4028334085 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50035070 con 0x7f9d60109670 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.048+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d601a6a90 con 0x7f9d60109670 2024-04-24T08:20:25.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.049+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6010f7a0 con 0x7f9d60109670 2024-04-24T08:20:25.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.051+0000 7f9d5e000640 1 -- 172.21.15.146:0/4028334085 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f9d5003b4f0 con 0x7f9d60109670 2024-04-24T08:20:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.052+0000 7f9d5e000640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9d400a3bb0 0x7f9d400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.052+0000 7f9d5e000640 1 -- 172.21.15.146:0/4028334085 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9d500f0d50 con 0x7f9d60109670 2024-04-24T08:20:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.052+0000 7f9d64a00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9d400a3bb0 0x7f9d400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.053+0000 7f9d64a00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9d400a3bb0 0x7f9d400a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d601094d0 tx=0x7f9d48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.055+0000 7f9d5e000640 1 -- 172.21.15.146:0/4028334085 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d500bd0a0 con 0x7f9d60109670 2024-04-24T08:20:25.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.261+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d6002d7f0 con 0x7f9d60109670 2024-04-24T08:20:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.264+0000 7f9d5e000640 1 -- 172.21.15.146:0/4028334085 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7f9d500dd070 con 0x7f9d60109670 2024-04-24T08:20:25.265 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:25.265 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9d400a3bb0 msgr2=0x7f9d400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9d400a3bb0 0x7f9d400a6070 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d601094d0 tx=0x7f9d48005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d60109670 msgr2=0x7f9d6019ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d60109670 0x7f9d6019ecb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d50031440 tx=0x7f9d50004270 comp rx=0 tx=0).stop 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 shutdown_connections 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9d400a3bb0 0x7f9d400a6070 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d6010a030 0x7f9d601a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d60109670 0x7f9d6019ecb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 --2- 172.21.15.146:0/4028334085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d60108480 0x7f9d6019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 >> 172.21.15.146:0/4028334085 conn(0x7f9d60077440 msgr2=0x7f9d6010c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 shutdown_connections 2024-04-24T08:20:25.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.266+0000 7f9d65e00640 1 -- 172.21.15.146:0/4028334085 wait complete. 2024-04-24T08:20:25.287 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:25.288 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:25.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 -- 172.21.15.146:0/2225909243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30105330 msgr2=0x7fac301057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 --2- 172.21.15.146:0/2225909243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30105330 0x7fac301057b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009900 tx=0x7fac1c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 -- 172.21.15.146:0/2225909243 shutdown_connections 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 --2- 172.21.15.146:0/2225909243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30105330 0x7fac301057b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 --2- 172.21.15.146:0/2225909243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105cf0 0x7fac3010c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 --2- 172.21.15.146:0/2225909243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac30104140 0x7fac30104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 -- 172.21.15.146:0/2225909243 >> 172.21.15.146:0/2225909243 conn(0x7fac300ff6b0 msgr2=0x7fac30101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.485+0000 7fac2f200640 1 -- 172.21.15.146:0/2225909243 shutdown_connections 2024-04-24T08:20:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.486+0000 7fac2f200640 1 -- 172.21.15.146:0/2225909243 wait complete. 2024-04-24T08:20:25.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 Processor -- start 2024-04-24T08:20:25.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 -- start start 2024-04-24T08:20:25.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30104140 0x7fac3019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105330 0x7fac3019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac30105cf0 0x7fac301a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac3010fdd0 con 0x7fac30105330 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac3010fc50 con 0x7fac30105cf0 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac3010ff50 con 0x7fac30104140 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105330 0x7fac3019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105330 0x7fac3019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37894/0 (socket says 172.21.15.146:37894) 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 -- 172.21.15.146:0/2430552720 learned_addr learned my addr 172.21.15.146:0/2430552720 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2de00640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30104140 0x7fac3019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2e800640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac30105cf0 0x7fac301a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 -- 172.21.15.146:0/2430552720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30104140 msgr2=0x7fac3019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30104140 0x7fac3019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 -- 172.21.15.146:0/2430552720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac30105cf0 msgr2=0x7fac301a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac30105cf0 0x7fac301a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 -- 172.21.15.146:0/2430552720 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac301a2050 con 0x7fac30105330 2024-04-24T08:20:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac2d400640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105330 0x7fac3019a970 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009230 tx=0x7fac1c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.488+0000 7fac26a00640 1 -- 172.21.15.146:0/2430552720 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c03f030 con 0x7fac30105330 2024-04-24T08:20:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.489+0000 7fac26a00640 1 -- 172.21.15.146:0/2430552720 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac1c003ce0 con 0x7fac30105330 2024-04-24T08:20:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.489+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac301a22e0 con 0x7fac30105330 2024-04-24T08:20:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.489+0000 7fac26a00640 1 -- 172.21.15.146:0/2430552720 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c035070 con 0x7fac30105330 2024-04-24T08:20:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.489+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac301a27d0 con 0x7fac30105330 2024-04-24T08:20:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.490+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac3010b4b0 con 0x7fac30105330 2024-04-24T08:20:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.492+0000 7fac26a00640 1 -- 172.21.15.146:0/2430552720 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fac1c042030 con 0x7fac30105330 2024-04-24T08:20:25.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.494+0000 7fac26a00640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fac140a3bb0 0x7fac140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.494+0000 7fac26a00640 1 -- 172.21.15.146:0/2430552720 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fac1c0f0d70 con 0x7fac30105330 2024-04-24T08:20:25.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.494+0000 7fac2de00640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fac140a3bb0 0x7fac140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.495+0000 7fac2de00640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fac140a3bb0 0x7fac140a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fac30105190 tx=0x7fac2000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.497+0000 7fac26a00640 1 -- 172.21.15.146:0/2430552720 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac1c0bd0c0 con 0x7fac30105330 2024-04-24T08:20:25.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.692+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac30062c00 con 0x7fac30105330 2024-04-24T08:20:25.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.693+0000 7fac26a00640 1 -- 172.21.15.146:0/2430552720 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fac1c0c13c0 con 0x7fac30105330 2024-04-24T08:20:25.695 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fac140a3bb0 msgr2=0x7fac140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fac140a3bb0 0x7fac140a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fac30105190 tx=0x7fac2000b390 comp rx=0 tx=0).stop 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105330 msgr2=0x7fac3019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105330 0x7fac3019a970 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009230 tx=0x7fac1c004360 comp rx=0 tx=0).stop 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 shutdown_connections 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fac140a3bb0 0x7fac140a6070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac30105cf0 0x7fac301a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac30105330 0x7fac3019a970 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 --2- 172.21.15.146:0/2430552720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac30104140 0x7fac3019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 >> 172.21.15.146:0/2430552720 conn(0x7fac300ff6b0 msgr2=0x7fac30108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 shutdown_connections 2024-04-24T08:20:25.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.696+0000 7fac2f200640 1 -- 172.21.15.146:0/2430552720 wait complete. 2024-04-24T08:20:25.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 -- 172.21.15.146:0/1074395126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 msgr2=0x7f3020108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 --2- 172.21.15.146:0/1074395126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 0x7f3020108960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3010009900 tx=0x7f301002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 -- 172.21.15.146:0/1074395126 shutdown_connections 2024-04-24T08:20:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 --2- 172.21.15.146:0/1074395126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f302010a0f0 0x7f3020110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 --2- 172.21.15.146:0/1074395126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 0x7f3020109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 --2- 172.21.15.146:0/1074395126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 0x7f3020108960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 -- 172.21.15.146:0/1074395126 >> 172.21.15.146:0/1074395126 conn(0x7f30200774d0 msgr2=0x7f3020079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.911+0000 7f301f200640 1 -- 172.21.15.146:0/1074395126 shutdown_connections 2024-04-24T08:20:25.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.912+0000 7f301f200640 1 -- 172.21.15.146:0/1074395126 wait complete. 2024-04-24T08:20:25.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 Processor -- start 2024-04-24T08:20:25.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 -- start start 2024-04-24T08:20:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 0x7f302019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 0x7f302019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f302010a0f0 0x7f30201a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f30201141d0 con 0x7f3020109730 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3020114050 con 0x7f3020108540 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3020114350 con 0x7f302010a0f0 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.913+0000 7f301d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 0x7f302019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 0x7f302019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37904/0 (socket says 172.21.15.146:37904) 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 -- 172.21.15.146:0/3670861617 learned_addr learned my addr 172.21.15.146:0/3670861617 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301de00640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 0x7f302019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301e800640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f302010a0f0 0x7f30201a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 -- 172.21.15.146:0/3670861617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f302010a0f0 msgr2=0x7f30201a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f302010a0f0 0x7f30201a5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 -- 172.21.15.146:0/3670861617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 msgr2=0x7f302019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 0x7f302019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 -- 172.21.15.146:0/3670861617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30201a6460 con 0x7f3020109730 2024-04-24T08:20:25.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301de00640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 0x7f302019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:25.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f301d400640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 0x7f302019ed80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f300c00e9f0 tx=0x7f300c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.914+0000 7f3016a00640 1 -- 172.21.15.146:0/3670861617 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c00cd80 con 0x7f3020109730 2024-04-24T08:20:25.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.915+0000 7f3016a00640 1 -- 172.21.15.146:0/3670861617 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c010910 con 0x7f3020109730 2024-04-24T08:20:25.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.915+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30201a6750 con 0x7f3020109730 2024-04-24T08:20:25.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.915+0000 7f3016a00640 1 -- 172.21.15.146:0/3670861617 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c018990 con 0x7f3020109730 2024-04-24T08:20:25.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.915+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30201a6ca0 con 0x7f3020109730 2024-04-24T08:20:25.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.916+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f302010f8e0 con 0x7f3020109730 2024-04-24T08:20:25.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.919+0000 7f3016a00640 1 -- 172.21.15.146:0/3670861617 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f300c010ac0 con 0x7f3020109730 2024-04-24T08:20:25.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.921+0000 7f3016a00640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f30040a3bb0 0x7f30040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.921+0000 7f3016a00640 1 -- 172.21.15.146:0/3670861617 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f300c0d0760 con 0x7f3020109730 2024-04-24T08:20:25.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.921+0000 7f301de00640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f30040a3bb0 0x7f30040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.921+0000 7f301de00640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f30040a3bb0 0x7f30040a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3010005ab0 tx=0x7f3010000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:25.925+0000 7f3016a00640 1 -- 172.21.15.146:0/3670861617 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f300c09cab0 con 0x7f3020109730 2024-04-24T08:20:25.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:25 smithi153 ceph-mon[32746]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:25.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:25 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1918283819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:25.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:25 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4028334085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:25.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:25 smithi146 ceph-mon[27805]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:25.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:25 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1918283819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:25.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:25 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4028334085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:26.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:25 smithi177 ceph-mon[33046]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:26.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:25 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1918283819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:26.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:25 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4028334085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:26.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.126+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3020062c00 con 0x7f3020109730 2024-04-24T08:20:26.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.127+0000 7f3016a00640 1 -- 172.21.15.146:0/3670861617 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f300c0a0db0 con 0x7f3020109730 2024-04-24T08:20:26.129 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f30040a3bb0 msgr2=0x7f30040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f30040a3bb0 0x7f30040a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3010005ab0 tx=0x7f3010000bc0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 msgr2=0x7f302019ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 0x7f302019ed80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f300c00e9f0 tx=0x7f300c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 shutdown_connections 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f302010a0f0 0x7f30201a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f30040a3bb0 0x7f30040a6070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3020109730 0x7f302019ed80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 --2- 172.21.15.146:0/3670861617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3020108540 0x7f302019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 >> 172.21.15.146:0/3670861617 conn(0x7f30200774d0 msgr2=0x7f302010cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 shutdown_connections 2024-04-24T08:20:26.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.130+0000 7f301f200640 1 -- 172.21.15.146:0/3670861617 wait complete. 2024-04-24T08:20:26.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:26.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.349+0000 7f422d000640 1 -- 172.21.15.146:0/876848174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42281052c0 msgr2=0x7f42281076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.349+0000 7f422d000640 1 --2- 172.21.15.146:0/876848174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42281052c0 0x7f42281076b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4210009860 tx=0x7f421002a340 comp rx=0 tx=0).stop 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.350+0000 7f422d000640 1 -- 172.21.15.146:0/876848174 shutdown_connections 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.350+0000 7f422d000640 1 --2- 172.21.15.146:0/876848174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42281052c0 0x7f42281076b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.350+0000 7f422d000640 1 --2- 172.21.15.146:0/876848174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4228107bf0 0x7f422810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.350+0000 7f422d000640 1 --2- 172.21.15.146:0/876848174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4228102990 0x7f4228104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.350+0000 7f422d000640 1 -- 172.21.15.146:0/876848174 >> 172.21.15.146:0/876848174 conn(0x7f42280fc3a0 msgr2=0x7f42280fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.350+0000 7f422d000640 1 -- 172.21.15.146:0/876848174 shutdown_connections 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.350+0000 7f422d000640 1 -- 172.21.15.146:0/876848174 wait complete. 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 Processor -- start 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 -- start start 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4228102990 0x7f422819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42281052c0 0x7f422819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4228107bf0 0x7f42281a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f422810db80 con 0x7f42281052c0 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f422810da00 con 0x7f4228107bf0 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f422d000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f422810dd00 con 0x7f4228102990 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f4226a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42281052c0 0x7f422819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.351+0000 7f4227400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4228102990 0x7f422819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42281052c0 0x7f422819a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37912/0 (socket says 172.21.15.146:37912) 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4227400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4228102990 0x7f422819a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50834/0 (socket says 172.21.15.146:50834) 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4227e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4228107bf0 0x7f42281a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 -- 172.21.15.146:0/926522072 learned_addr learned my addr 172.21.15.146:0/926522072 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 -- 172.21.15.146:0/926522072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4228102990 msgr2=0x7f422819a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4228102990 0x7f422819a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 -- 172.21.15.146:0/926522072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4228107bf0 msgr2=0x7f42281a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4228107bf0 0x7f42281a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 -- 172.21.15.146:0/926522072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42281a2030 con 0x7f42281052c0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4227e00640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4228107bf0 0x7f42281a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.352+0000 7f4226a00640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42281052c0 0x7f422819a950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4210035040 tx=0x7f4210033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.353+0000 7f421fe00640 1 -- 172.21.15.146:0/926522072 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4210037040 con 0x7f42281052c0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.353+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42281a22c0 con 0x7f42281052c0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.353+0000 7f421fe00640 1 -- 172.21.15.146:0/926522072 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4210030850 con 0x7f42281052c0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.353+0000 7f421fe00640 1 -- 172.21.15.146:0/926522072 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4210004920 con 0x7f42281052c0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.353+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42281a2800 con 0x7f42281052c0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.354+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4228109740 con 0x7f42281052c0 2024-04-24T08:20:26.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.357+0000 7f421fe00640 1 -- 172.21.15.146:0/926522072 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f4210004050 con 0x7f42281052c0 2024-04-24T08:20:26.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.358+0000 7f421fe00640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f420c0a3bb0 0x7f420c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.358+0000 7f421fe00640 1 -- 172.21.15.146:0/926522072 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f42100f05a0 con 0x7f42281052c0 2024-04-24T08:20:26.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.359+0000 7f4227400640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f420c0a3bb0 0x7f420c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.359+0000 7f4227400640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f420c0a3bb0 0x7f420c0a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4218005fd0 tx=0x7f4218010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.361+0000 7f421fe00640 1 -- 172.21.15.146:0/926522072 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f42100bcf40 con 0x7f42281052c0 2024-04-24T08:20:26.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.564+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f422802d820 con 0x7f42281052c0 2024-04-24T08:20:26.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.565+0000 7f421fe00640 1 -- 172.21.15.146:0/926522072 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f42100bc840 con 0x7f42281052c0 2024-04-24T08:20:26.566 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:20:26.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.567+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f420c0a3bb0 msgr2=0x7f420c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.567+0000 7f422d000640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f420c0a3bb0 0x7f420c0a6070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4218005fd0 tx=0x7f4218010040 comp rx=0 tx=0).stop 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42281052c0 msgr2=0x7f422819a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42281052c0 0x7f422819a950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4210035040 tx=0x7f4210033d90 comp rx=0 tx=0).stop 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 shutdown_connections 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f420c0a3bb0 0x7f420c0a6070 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4228107bf0 0x7f42281a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42281052c0 0x7f422819a950 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 --2- 172.21.15.146:0/926522072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4228102990 0x7f422819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 >> 172.21.15.146:0/926522072 conn(0x7f42280fc3a0 msgr2=0x7f4228102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 shutdown_connections 2024-04-24T08:20:26.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.568+0000 7f422d000640 1 -- 172.21.15.146:0/926522072 wait complete. 2024-04-24T08:20:26.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 -- 172.21.15.146:0/2896630495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788104080 msgr2=0x7ff7881044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 --2- 172.21.15.146:0/2896630495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788104080 0x7ff7881044a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff778009900 tx=0x7ff77802a410 comp rx=0 tx=0).stop 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 -- 172.21.15.146:0/2896630495 shutdown_connections 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 --2- 172.21.15.146:0/2896630495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788105c30 0x7ff78810c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 --2- 172.21.15.146:0/2896630495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff788105270 0x7ff7881056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 --2- 172.21.15.146:0/2896630495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788104080 0x7ff7881044a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 -- 172.21.15.146:0/2896630495 >> 172.21.15.146:0/2896630495 conn(0x7ff7880ff610 msgr2=0x7ff788101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 -- 172.21.15.146:0/2896630495 shutdown_connections 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.785+0000 7ff78da00640 1 -- 172.21.15.146:0/2896630495 wait complete. 2024-04-24T08:20:26.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 Processor -- start 2024-04-24T08:20:26.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 -- start start 2024-04-24T08:20:26.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788104080 0x7ff78819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788105270 0x7ff78819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff788105c30 0x7ff7881a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff78810fd60 con 0x7ff788104080 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff78810fbe0 con 0x7ff788105c30 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.786+0000 7ff78da00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff78810fee0 con 0x7ff788105270 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788104080 0x7ff78819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788104080 0x7ff78819a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37922/0 (socket says 172.21.15.146:37922) 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 -- 172.21.15.146:0/1977406497 learned_addr learned my addr 172.21.15.146:0/1977406497 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787400640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788105270 0x7ff78819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff78d000640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff788105c30 0x7ff7881a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 -- 172.21.15.146:0/1977406497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788105270 msgr2=0x7ff78819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788105270 0x7ff78819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 -- 172.21.15.146:0/1977406497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff788105c30 msgr2=0x7ff7881a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff788105c30 0x7ff7881a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 -- 172.21.15.146:0/1977406497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7881a1fc0 con 0x7ff788104080 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff787e00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788104080 0x7ff78819a3a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff778031440 tx=0x7ff778032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff784c00640 1 -- 172.21.15.146:0/1977406497 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77803f030 con 0x7ff788104080 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7881a2250 con 0x7ff788104080 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff784c00640 1 -- 172.21.15.146:0/1977406497 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff778003ad0 con 0x7ff788104080 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff784c00640 1 -- 172.21.15.146:0/1977406497 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778035070 con 0x7ff788104080 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.787+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7881a2790 con 0x7ff788104080 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.789+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff78810b3a0 con 0x7ff788104080 2024-04-24T08:20:26.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.790+0000 7ff784c00640 1 -- 172.21.15.146:0/1977406497 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7ff778042030 con 0x7ff788104080 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.792+0000 7ff784c00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff76c0a3c80 0x7ff76c0a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.793+0000 7ff784c00640 1 -- 172.21.15.146:0/1977406497 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff7780f0880 con 0x7ff788104080 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.793+0000 7ff787400640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff76c0a3c80 0x7ff76c0a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.793+0000 7ff787400640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff76c0a3c80 0x7ff76c0a6140 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff78819b8b0 tx=0x7ff770009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:26.796+0000 7ff784c00640 1 -- 172.21.15.146:0/1977406497 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7780bcbd0 con 0x7ff788104080 2024-04-24T08:20:26.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:26 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2430552720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:26.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:26 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3670861617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:26.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:26 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2430552720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:26.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:26 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3670861617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.042+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff78802d620 con 0x7ff788104080 2024-04-24T08:20:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.044+0000 7ff784c00640 1 -- 172.21.15.146:0/1977406497 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11891 (secure 0 0 0) 0x7ff7780c0ed0 con 0x7ff788104080 2024-04-24T08:20:27.045 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:27.045 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:18:52.518710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:20:27.045 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff76c0a3c80 msgr2=0x7ff76c0a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff76c0a3c80 0x7ff76c0a6140 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff78819b8b0 tx=0x7ff770009500 comp rx=0 tx=0).stop 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788104080 msgr2=0x7ff78819a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788104080 0x7ff78819a3a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff778031440 tx=0x7ff778032c60 comp rx=0 tx=0).stop 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 shutdown_connections 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff788105270 0x7ff78819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff76c0a3c80 0x7ff76c0a6140 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff788105c30 0x7ff7881a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 --2- 172.21.15.146:0/1977406497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff788104080 0x7ff78819a3a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 >> 172.21.15.146:0/1977406497 conn(0x7ff7880ff610 msgr2=0x7ff788108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 shutdown_connections 2024-04-24T08:20:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.047+0000 7ff78da00640 1 -- 172.21.15.146:0/1977406497 wait complete. 2024-04-24T08:20:27.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:26 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2430552720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:27.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:26 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3670861617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:27.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:27.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 -- 172.21.15.146:0/732938664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 msgr2=0x7f33981088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 --2- 172.21.15.146:0/732938664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 0x7f33981088a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3388009900 tx=0x7f338802a410 comp rx=0 tx=0).stop 2024-04-24T08:20:27.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 -- 172.21.15.146:0/732938664 shutdown_connections 2024-04-24T08:20:27.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 --2- 172.21.15.146:0/732938664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3398109670 0x7f3398109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 --2- 172.21.15.146:0/732938664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f339810a030 0x7f33981108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 --2- 172.21.15.146:0/732938664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 0x7f33981088a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 -- 172.21.15.146:0/732938664 >> 172.21.15.146:0/732938664 conn(0x7f3398077440 msgr2=0x7f3398079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 -- 172.21.15.146:0/732938664 shutdown_connections 2024-04-24T08:20:27.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.269+0000 7f3397e00640 1 -- 172.21.15.146:0/732938664 wait complete. 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.270+0000 7f3397e00640 1 Processor -- start 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.270+0000 7f3397e00640 1 -- start start 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 0x7f339819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3398109670 0x7f339819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f339810a030 0x7f33981a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3398114160 con 0x7f3398108480 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3398113fe0 con 0x7f3398109670 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33981142e0 con 0x7f339810a030 2024-04-24T08:20:27.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3396a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 0x7f339819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f338e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3398109670 0x7f339819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3396a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 0x7f339819e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37928/0 (socket says 172.21.15.146:37928) 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3396a00640 1 -- 172.21.15.146:0/3855367770 learned_addr learned my addr 172.21.15.146:0/3855367770 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397400640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f339810a030 0x7f33981a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f338e000640 1 -- 172.21.15.146:0/3855367770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f339810a030 msgr2=0x7f33981a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f338e000640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f339810a030 0x7f33981a5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f338e000640 1 -- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 msgr2=0x7f339819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f338e000640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 0x7f339819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f338e000640 1 -- 172.21.15.146:0/3855367770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33981a6400 con 0x7f3398109670 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.271+0000 7f3397400640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f339810a030 0x7f33981a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.272+0000 7f338e000640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3398109670 0x7f339819ed70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3380009fd0 tx=0x7f338000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.272+0000 7f338fe00640 1 -- 172.21.15.146:0/3855367770 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380004b20 con 0x7f3398109670 2024-04-24T08:20:27.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.272+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33981a66f0 con 0x7f3398109670 2024-04-24T08:20:27.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.272+0000 7f338fe00640 1 -- 172.21.15.146:0/3855367770 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3380004cc0 con 0x7f3398109670 2024-04-24T08:20:27.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.272+0000 7f338fe00640 1 -- 172.21.15.146:0/3855367770 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33800107b0 con 0x7f3398109670 2024-04-24T08:20:27.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.272+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33981a6c40 con 0x7f3398109670 2024-04-24T08:20:27.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.273+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f339810f7a0 con 0x7f3398109670 2024-04-24T08:20:27.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.275+0000 7f338fe00640 1 -- 172.21.15.146:0/3855367770 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f3380005020 con 0x7f3398109670 2024-04-24T08:20:27.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.277+0000 7f338fe00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f337c0a3bb0 0x7f337c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.277+0000 7f338fe00640 1 -- 172.21.15.146:0/3855367770 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f33800d0c70 con 0x7f3398109670 2024-04-24T08:20:27.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.277+0000 7f3396a00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f337c0a3bb0 0x7f337c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.278+0000 7f3396a00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f337c0a3bb0 0x7f337c0a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3388000c00 tx=0x7f33880315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.281+0000 7f338fe00640 1 -- 172.21.15.146:0/3855367770 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f338009d610 con 0x7f3398109670 2024-04-24T08:20:27.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.487+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3398062c00 con 0x7f3398109670 2024-04-24T08:20:27.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.489+0000 7f338fe00640 1 -- 172.21.15.146:0/3855367770 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7f338009cf10 con 0x7f3398109670 2024-04-24T08:20:27.490 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:27.491 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f337c0a3bb0 msgr2=0x7f337c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f337c0a3bb0 0x7f337c0a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3388000c00 tx=0x7f33880315b0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3398109670 msgr2=0x7f339819ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3398109670 0x7f339819ed70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3380009fd0 tx=0x7f338000ef90 comp rx=0 tx=0).stop 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 shutdown_connections 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f339810a030 0x7f33981a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f337c0a3bb0 0x7f337c0a6070 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3398109670 0x7f339819ed70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 --2- 172.21.15.146:0/3855367770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3398108480 0x7f339819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 >> 172.21.15.146:0/3855367770 conn(0x7f3398077440 msgr2=0x7f339810c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 shutdown_connections 2024-04-24T08:20:27.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.492+0000 7f3397e00640 1 -- 172.21.15.146:0/3855367770 wait complete. 2024-04-24T08:20:27.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 -- 172.21.15.146:0/1696681338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8104080 msgr2=0x7f17b81044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 --2- 172.21.15.146:0/1696681338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8104080 0x7f17b81044a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009900 tx=0x7f17a802a410 comp rx=0 tx=0).stop 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 -- 172.21.15.146:0/1696681338 shutdown_connections 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 --2- 172.21.15.146:0/1696681338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b8105c30 0x7f17b810c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 --2- 172.21.15.146:0/1696681338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8105270 0x7f17b81056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 --2- 172.21.15.146:0/1696681338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8104080 0x7f17b81044a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 -- 172.21.15.146:0/1696681338 >> 172.21.15.146:0/1696681338 conn(0x7f17b80ff610 msgr2=0x7f17b8101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 -- 172.21.15.146:0/1696681338 shutdown_connections 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.706+0000 7f17be000640 1 -- 172.21.15.146:0/1696681338 wait complete. 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.707+0000 7f17be000640 1 Processor -- start 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.707+0000 7f17be000640 1 -- start start 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17be000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8104080 0x7f17b819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17be000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8105270 0x7f17b819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17be000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b8105c30 0x7f17b81a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17be000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810fd10 con 0x7f17b8104080 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17be000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810fb90 con 0x7f17b8105c30 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17be000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810fe90 con 0x7f17b8105270 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8104080 0x7f17b819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8104080 0x7f17b819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37944/0 (socket says 172.21.15.146:37944) 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 -- 172.21.15.146:0/3233545271 learned_addr learned my addr 172.21.15.146:0/3233545271 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:27.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bd600640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b8105c30 0x7f17b81a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17b7e00640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8105270 0x7f17b819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 -- 172.21.15.146:0/3233545271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8105270 msgr2=0x7f17b819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8105270 0x7f17b819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 -- 172.21.15.146:0/3233545271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b8105c30 msgr2=0x7f17b81a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b8105c30 0x7f17b81a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.708+0000 7f17bcc00640 1 -- 172.21.15.146:0/3233545271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b81a1fb0 con 0x7f17b8104080 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.709+0000 7f17bcc00640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8104080 0x7f17b819a390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f17a8031610 tx=0x7f17a8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.709+0000 7f17b5600640 1 -- 172.21.15.146:0/3233545271 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a803f030 con 0x7f17b8104080 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.709+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b81a2240 con 0x7f17b8104080 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.709+0000 7f17b5600640 1 -- 172.21.15.146:0/3233545271 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a8003ad0 con 0x7f17b8104080 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.709+0000 7f17b5600640 1 -- 172.21.15.146:0/3233545271 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a8035070 con 0x7f17b8104080 2024-04-24T08:20:27.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.709+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b81a2730 con 0x7f17b8104080 2024-04-24T08:20:27.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.711+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b810b3a0 con 0x7f17b8104080 2024-04-24T08:20:27.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.713+0000 7f17b5600640 1 -- 172.21.15.146:0/3233545271 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f17a8042030 con 0x7f17b8104080 2024-04-24T08:20:27.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.714+0000 7f17b5600640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f179c0a3bb0 0x7f179c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.715+0000 7f17b5600640 1 -- 172.21.15.146:0/3233545271 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f17a80f0640 con 0x7f17b8104080 2024-04-24T08:20:27.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.715+0000 7f17b7e00640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f179c0a3bb0 0x7f179c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.715+0000 7f17b7e00640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f179c0a3bb0 0x7f179c0a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f17b819b8a0 tx=0x7f17a0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.719+0000 7f17b5600640 1 -- 172.21.15.146:0/3233545271 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f17a80bc990 con 0x7f17b8104080 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.949+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17b802d620 con 0x7f17b8104080 2024-04-24T08:20:27.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.950+0000 7f17b5600640 1 -- 172.21.15.146:0/3233545271 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11891 (secure 0 0 0) 0x7f17a80c0c90 con 0x7f17b8104080 2024-04-24T08:20:27.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:18:52.518710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.953+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f179c0a3bb0 msgr2=0x7f179c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.953+0000 7f17be000640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f179c0a3bb0 0x7f179c0a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f17b819b8a0 tx=0x7f17a0009500 comp rx=0 tx=0).stop 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.953+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8104080 msgr2=0x7f17b819a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.953+0000 7f17be000640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8104080 0x7f17b819a390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f17a8031610 tx=0x7f17a8004270 comp rx=0 tx=0).stop 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 shutdown_connections 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17b8105270 0x7f17b819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f179c0a3bb0 0x7f179c0a6070 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17b8105c30 0x7f17b81a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 --2- 172.21.15.146:0/3233545271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17b8104080 0x7f17b819a390 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 >> 172.21.15.146:0/3233545271 conn(0x7f17b80ff610 msgr2=0x7f17b8108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 shutdown_connections 2024-04-24T08:20:27.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:27.954+0000 7f17be000640 1 -- 172.21.15.146:0/3233545271 wait complete. 2024-04-24T08:20:27.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:27 smithi153 ceph-mon[32746]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:27.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:27 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/926522072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:27.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:27 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1977406497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:27.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:27 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3855367770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:27.970 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:27 smithi146 ceph-mon[27805]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:27.970 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:27 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/926522072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:27.970 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:27 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1977406497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:27.970 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:27 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3855367770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:27.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-24T08:18:46.894792+0000', 'modified': '2024-04-24T08:18:52.518710+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24569, 'mds_1': 24568, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6837/2495345664', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 2495345664}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 2495345664}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24518': {'gid': 24518, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/565470680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 565470680}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 565470680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6839/1533073551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 1533073551}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 1533073551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.146:6839/2207966302', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6838', 'nonce': 2207966302}, {'type': 'v1', 'addr': '172.21.15.146:6839', 'nonce': 2207966302}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24568': {'gid': 24568, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6841/1590374458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6840', 'nonce': 1590374458}, {'type': 'v1', 'addr': '172.21.15.146:6841', 'nonce': 1590374458}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/2023799727', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 2023799727}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 2023799727}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24569, 'qdb_cluster': [24553, 24568, 24569]} 2024-04-24T08:20:27.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-24T08:20:27.973 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-24T08:20:28.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:27 smithi177 ceph-mon[33046]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:28.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:27 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/926522072' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:28.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:27 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1977406497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:28.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:27 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3855367770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 -- 172.21.15.146:0/4051670486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa0077c00 msgr2=0x7f8aa0078060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 --2- 172.21.15.146:0/4051670486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa0077c00 0x7f8aa0078060 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009860 tx=0x7f8a8c02a340 comp rx=0 tx=0).stop 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 -- 172.21.15.146:0/4051670486 shutdown_connections 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 --2- 172.21.15.146:0/4051670486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa0077c00 0x7f8aa0078060 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 --2- 172.21.15.146:0/4051670486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aa00785a0 0x7f8aa0078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 --2- 172.21.15.146:0/4051670486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00769c0 0x7f8aa0076de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 -- 172.21.15.146:0/4051670486 >> 172.21.15.146:0/4051670486 conn(0x7f8aa00ff7b0 msgr2=0x7f8aa0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 -- 172.21.15.146:0/4051670486 shutdown_connections 2024-04-24T08:20:28.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.170+0000 7f8aa5200640 1 -- 172.21.15.146:0/4051670486 wait complete. 2024-04-24T08:20:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 Processor -- start 2024-04-24T08:20:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 -- start start 2024-04-24T08:20:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa00769c0 0x7f8aa019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aa0077c00 0x7f8aa019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00785a0 0x7f8aa01a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01142b0 con 0x7f8aa00785a0 2024-04-24T08:20:28.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa0114130 con 0x7f8aa0077c00 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.171+0000 7f8aa5200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa0114430 con 0x7f8aa00769c0 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00785a0 0x7f8aa01a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00785a0 0x7f8aa01a5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37962/0 (socket says 172.21.15.146:37962) 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9e800640 1 -- 172.21.15.146:0/1434641560 learned_addr learned my addr 172.21.15.146:0/1434641560 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9de00640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa00769c0 0x7f8aa019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9d400640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aa0077c00 0x7f8aa019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9de00640 1 -- 172.21.15.146:0/1434641560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aa0077c00 msgr2=0x7f8aa019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9de00640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aa0077c00 0x7f8aa019eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9de00640 1 -- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00785a0 msgr2=0x7f8aa01a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9de00640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00785a0 0x7f8aa01a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9de00640 1 -- 172.21.15.146:0/1434641560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa01a6580 con 0x7f8aa00769c0 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9e800640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00785a0 0x7f8aa01a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9d400640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aa0077c00 0x7f8aa019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.172+0000 7f8a9de00640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa00769c0 0x7f8aa019e960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000e790 tx=0x7f8a9000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.173+0000 7f8a96a00640 1 -- 172.21.15.146:0/1434641560 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9000eea0 con 0x7f8aa00769c0 2024-04-24T08:20:28.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.173+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa01a6870 con 0x7f8aa00769c0 2024-04-24T08:20:28.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.173+0000 7f8a96a00640 1 -- 172.21.15.146:0/1434641560 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a9000ccc0 con 0x7f8aa00769c0 2024-04-24T08:20:28.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.173+0000 7f8a96a00640 1 -- 172.21.15.146:0/1434641560 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90018780 con 0x7f8aa00769c0 2024-04-24T08:20:28.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.173+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa01a6d90 con 0x7f8aa00769c0 2024-04-24T08:20:28.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 Processor -- start 2024-04-24T08:20:28.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 -- start start 2024-04-24T08:20:28.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56cc54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5a80 0x55bf56cca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56ccd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bf56c2ab10 con 0x55bf56cc5a80 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bf56bd12b0 con 0x55bf56cc5090 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8aa8049740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf56c26ad0 con 0x55bf56ccabf0 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.177+0000 7f8a96a00640 1 -- 172.21.15.146:0/1434641560 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f8a90010430 con 0x7f8aa00769c0 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5a80 0x55bf56cca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56cc54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56cc54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58172/0 (socket says 172.21.15.146:58172) 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5a80 0x55bf56cca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37970/0 (socket says 172.21.15.146:37970) 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 -- 172.21.15.146:0/352940635 learned_addr learned my addr 172.21.15.146:0/352940635 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9d400640 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56ccd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 -- 172.21.15.146:0/352940635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 msgr2=0x55bf56ccd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56ccd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 -- 172.21.15.146:0/352940635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5a80 msgr2=0x55bf56cca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5a80 0x55bf56cca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9e800640 1 -- 172.21.15.146:0/352940635 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf5693d2d0 con 0x55bf56cc5090 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.178+0000 7f8a9de00640 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5a80 0x55bf56cca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.179+0000 7f8a9e800640 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56cc54b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8a980118b0 tx=0x7f8a98011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d159d251fa9a28ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.179+0000 7f8a8b400640 1 -- 172.21.15.146:0/352940635 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9801e070 con 0x55bf56cc5090 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.179+0000 7f8a96a00640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8a840a3bb0 0x7f8a840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.179+0000 7f8a8b400640 1 -- 172.21.15.146:0/352940635 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a98011310 con 0x55bf56cc5090 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.179+0000 7f8a8b400640 1 -- 172.21.15.146:0/352940635 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98010430 con 0x55bf56cc5090 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.179+0000 7f8a96a00640 1 -- 172.21.15.146:0/1434641560 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8a900d0550 con 0x7f8aa00769c0 2024-04-24T08:20:28.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.179+0000 7f8a9de00640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8a840a3bb0 0x7f8a840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 -- 172.21.15.146:0/352940635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 msgr2=0x55bf56cc54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8a9de00640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8a840a3bb0 0x7f8a840a6070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55bf56cc5e80 tx=0x7f8a900d6060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56cc54b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8a980118b0 tx=0x7f8a98011d70 comp rx=0 tx=0).stop 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 -- 172.21.15.146:0/352940635 shutdown_connections 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56ccd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5a80 0x55bf56cca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 --2- 172.21.15.146:0/352940635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56cc54b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 -- 172.21.15.146:0/352940635 >> 172.21.15.146:0/352940635 conn(0x55bf56cc0700 msgr2=0x55bf56cc0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 -- 172.21.15.146:0/352940635 shutdown_connections 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 -- 172.21.15.146:0/352940635 wait complete. 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 -- start start 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.180+0000 7f8aa8049740 1 Processor -- start 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56ccf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56cc5a80 0x55bf56ccf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56ccabf0 0x55bf56cd6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8aa8049740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bf56ccdba0 con 0x55bf56ccabf0 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8aa8049740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bf56be4590 con 0x55bf56cc5090 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8aa8049740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf56ccdd20 con 0x55bf56cc5a80 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8aa8049740 10 client.0 _ll_register_callbacks cb 0x55bf56bc73a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56ccf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56ccabf0 0x55bf56cd6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56ccabf0 0x55bf56cd6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37982/0 (socket says 172.21.15.146:37982) 2024-04-24T08:20:28.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 -- 172.21.15.146:0/1007791097 learned_addr learned my addr 172.21.15.146:0/1007791097 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9d400640 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56cc5a80 0x55bf56ccf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 -- 172.21.15.146:0/1007791097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56cc5a80 msgr2=0x55bf56ccf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56cc5a80 0x55bf56ccf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 -- 172.21.15.146:0/1007791097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 msgr2=0x55bf56ccf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56ccf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.181+0000 7f8a9e800640 1 -- 172.21.15.146:0/1007791097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf56a076b0 con 0x55bf56ccabf0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8a9e800640 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56ccabf0 0x55bf56cd6980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8a9801b000 tx=0x7f8a98000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78eb1ce6c3c2540d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8a7fe00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8a7fe00640 1 -- 172.21.15.146:0/1007791097 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9802d060 con 0x55bf56ccabf0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8a7fe00640 1 -- 172.21.15.146:0/1007791097 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8a98019dd0 con 0x55bf56ccabf0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a98011310 con 0x55bf56ccabf0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8a7fe00640 1 -- 172.21.15.146:0/1007791097 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8a9801e040 con 0x55bf56ccabf0 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8aa8049740 10 client.14862 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:28.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bf56cd9cf0 con 0x55bf56ccabf0 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.182+0000 7f8a7fe00640 1 -- 172.21.15.146:0/1007791097 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a98011310 con 0x55bf56ccabf0 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8a7fe00640 1 -- 172.21.15.146:0/1007791097 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8a98032070 con 0x55bf56ccabf0 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8aa8049740 10 client.14862 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8aa8049740 10 client.14862 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8aa8049740 20 client.14862 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8aa8049740 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56cdc0c0 0x55bf56cdd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8aa8049740 4 client.14862 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf56c38530 con 0x55bf56cdc0c0 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.183+0000 7f8a9d400640 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56cdc0c0 0x55bf56cdd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.184+0000 7f8a7fe00640 10 client.14862 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.184+0000 7f8a9d400640 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56cdc0c0 0x55bf56cdd530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=680b5f4a650e5e01 server_cookie=48c49fc2a69806f6 in_seq=0 out_seq=0 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8a7fe00640 1 -- 172.21.15.146:0/1007791097 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf56c38530 con 0x55bf56cdc0c0 2024-04-24T08:20:28.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8a7fe00640 10 client.14862 handle_command_reply: tid=0 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 client.14862 shutdown 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56ccabf0 msgr2=0x55bf56cd6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56ccabf0 0x55bf56cd6980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8a9801b000 tx=0x7f8a98000f40 comp rx=0 tx=0).stop 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 shutdown_connections 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56cdc0c0 0x55bf56cdd530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56cc5a80 0x55bf56ccf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56ccabf0 0x55bf56cd6980 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 --2- 172.21.15.146:0/1007791097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5090 0x55bf56ccf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8a7fe00640 0 client.14862 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 >> 172.21.15.146:0/1007791097 conn(0x55bf56cc0700 msgr2=0x55bf56cbf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 shutdown_connections 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 1 -- 172.21.15.146:0/1007791097 wait complete. 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.185+0000 7f8aa8049740 20 client.14862 trim_cache size 0 max 16384 2024-04-24T08:20:28.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 Processor -- start 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 -- start start 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56ccf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56ccf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bf56cdb870 con 0x55bf56cc5090 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bf56cdbf40 con 0x55bf56cc5a80 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.204+0000 7f8aa8049740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf56cdc610 con 0x55bf56ccabf0 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58192/0 (socket says 172.21.15.146:58192) 2024-04-24T08:20:28.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 -- 172.21.15.146:0/2239310791 learned_addr learned my addr 172.21.15.146:0/2239310791 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9e800640 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56ccf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 -- 172.21.15.146:0/2239310791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 msgr2=0x55bf56ccf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56ccf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 -- 172.21.15.146:0/2239310791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 msgr2=0x55bf56ccf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56ccf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 -- 172.21.15.146:0/2239310791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf56cca8d0 con 0x55bf56cc5a80 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.205+0000 7f8a9d400640 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd68a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c031000 tx=0x7f8a8c02aac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a01559b5360289 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8a8aa00640 1 -- 172.21.15.146:0/2239310791 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c037040 con 0x55bf56cc5a80 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8a8aa00640 1 -- 172.21.15.146:0/2239310791 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a8c005520 con 0x55bf56cc5a80 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8a8aa00640 1 -- 172.21.15.146:0/2239310791 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c004c20 con 0x55bf56cc5a80 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 -- 172.21.15.146:0/2239310791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 msgr2=0x55bf56cd68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd68a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c031000 tx=0x7f8a8c02aac0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 -- 172.21.15.146:0/2239310791 shutdown_connections 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56ccf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd68a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2239310791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56ccf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 -- 172.21.15.146:0/2239310791 >> 172.21.15.146:0/2239310791 conn(0x55bf56cc0700 msgr2=0x55bf56cbb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 -- 172.21.15.146:0/2239310791 shutdown_connections 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.206+0000 7f8aa8049740 1 -- 172.21.15.146:0/2239310791 wait complete. 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 -- start start 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 Processor -- start 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56cd2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56cd5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bf56cdd3b0 con 0x55bf56cc5090 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bf56cdcce0 con 0x55bf56cc5a80 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf56cdda80 con 0x55bf56ccabf0 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.207+0000 7f8aa8049740 10 client.0 _ll_register_callbacks cb 0x7f8a90005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56cd2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56cd2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:37994/0 (socket says 172.21.15.146:37994) 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 -- 172.21.15.146:0/2941168108 learned_addr learned my addr 172.21.15.146:0/2941168108 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9d400640 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56cd5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 -- 172.21.15.146:0/2941168108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 msgr2=0x55bf56cd5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56cd5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 -- 172.21.15.146:0/2941168108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 msgr2=0x55bf56cd3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 -- 172.21.15.146:0/2941168108 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf56cd63a0 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a9e800640 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56cd2ad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8a980118b0 tx=0x7f8a980021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405571fe520937 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.208+0000 7f8a7fe00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8a7fe00640 1 -- 172.21.15.146:0/2941168108 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98038040 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8a7fe00640 1 -- 172.21.15.146:0/2941168108 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f8a98004e80 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a8c005520 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8a7fe00640 1 -- 172.21.15.146:0/2941168108 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f8a9802d040 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 10 client.14868 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55bf56d42930 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8a7fe00640 1 -- 172.21.15.146:0/2941168108 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a8c005520 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8a7fe00640 1 -- 172.21.15.146:0/2941168108 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f8a98032050 con 0x55bf56cc5090 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 10 client.14868 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 10 client.14868 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 20 client.14868 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56d47480 0x55bf56d46190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 4 client.14868 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.209+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "status"}) -- 0x55bf56c323a0 con 0x55bf56d47480 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.210+0000 7f8a9d400640 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56d47480 0x55bf56d46190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.210+0000 7f8a9d400640 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56d47480 0x55bf56d46190 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af7389be56e50dba server_cookie=e676d6bcee40882b in_seq=0 out_seq=0 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.210+0000 7f8a7fe00640 10 client.14868 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.210+0000 7f8a7fe00640 1 -- 172.21.15.146:0/2941168108 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55bf56c323a0 con 0x55bf56d47480 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.210+0000 7f8a7fe00640 10 client.14868 handle_command_reply: tid=0 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.210+0000 7f8aa8049740 1 client.14868 shutdown 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.210+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 msgr2=0x55bf56cd2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56cd2ad0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8a980118b0 tx=0x7f8a980021d0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 shutdown_connections 2024-04-24T08:20:28.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55bf56d47480 0x55bf56d46190 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf56ccabf0 0x55bf56cd5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bf56cc5a80 0x55bf56cd3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 --2- 172.21.15.146:0/2941168108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bf56cc5090 0x55bf56cd2ad0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8a7fe00640 0 client.14868 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 >> 172.21.15.146:0/2941168108 conn(0x55bf56cc0700 msgr2=0x55bf56ccf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 shutdown_connections 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 1 -- 172.21.15.146:0/2941168108 wait complete. 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa8049740 20 client.14868 trim_cache size 0 max 16384 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 2, 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 24553, 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 96.701830389999998, 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 23, 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 90, 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 88, 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 112.96696983299999 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8a840a3bb0 msgr2=0x7f8a840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa5200640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8a840a3bb0 0x7f8a840a6070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55bf56cc5e80 tx=0x7f8a900d6060 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa00769c0 msgr2=0x7f8aa019e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.211+0000 7f8aa5200640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa00769c0 0x7f8aa019e960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000e790 tx=0x7f8a9000ec50 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 shutdown_connections 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f8a840a3bb0 0x7f8a840a6070 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8aa00785a0 0x7f8aa01a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8aa0077c00 0x7f8aa019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 --2- 172.21.15.146:0/1434641560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8aa00769c0 0x7f8aa019e960 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 >> 172.21.15.146:0/1434641560 conn(0x7f8aa00ff7b0 msgr2=0x7f8aa0100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 shutdown_connections 2024-04-24T08:20:28.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.212+0000 7f8aa5200640 1 -- 172.21.15.146:0/1434641560 wait complete. 2024-04-24T08:20:28.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 -- 172.21.15.146:0/489909307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280769c0 msgr2=0x7fc228076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 --2- 172.21.15.146:0/489909307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280769c0 0x7fc228076de0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc210009860 tx=0x7fc21002a340 comp rx=0 tx=0).stop 2024-04-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 -- 172.21.15.146:0/489909307 shutdown_connections 2024-04-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 --2- 172.21.15.146:0/489909307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2280785a0 0x7fc228078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 --2- 172.21.15.146:0/489909307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc228077c00 0x7fc228078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 --2- 172.21.15.146:0/489909307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280769c0 0x7fc228076de0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 -- 172.21.15.146:0/489909307 >> 172.21.15.146:0/489909307 conn(0x7fc2280ff6b0 msgr2=0x7fc228101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 -- 172.21.15.146:0/489909307 shutdown_connections 2024-04-24T08:20:28.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.419+0000 7fc22cc00640 1 -- 172.21.15.146:0/489909307 wait complete. 2024-04-24T08:20:28.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.420+0000 7fc22cc00640 1 Processor -- start 2024-04-24T08:20:28.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.420+0000 7fc22cc00640 1 -- start start 2024-04-24T08:20:28.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc22cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2280769c0 0x7fc22819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc22cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc228077c00 0x7fc22819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc22cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280785a0 0x7fc2281a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc22cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc2281141d0 con 0x7fc228077c00 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc22cc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc228114050 con 0x7fc2280769c0 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc22cc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc228114350 con 0x7fc2280785a0 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc228077c00 0x7fc22819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2280769c0 0x7fc22819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc228077c00 0x7fc22819ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38016/0 (socket says 172.21.15.146:38016) 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280785a0 0x7fc2281a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc21fe00640 1 -- 172.21.15.146:0/1881833208 learned_addr learned my addr 172.21.15.146:0/1881833208 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2280769c0 0x7fc22819e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58212/0 (socket says 172.21.15.146:58212) 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227e00640 1 -- 172.21.15.146:0/1881833208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2280769c0 msgr2=0x7fc22819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227e00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2280769c0 0x7fc22819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227e00640 1 -- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc228077c00 msgr2=0x7fc22819ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227e00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc228077c00 0x7fc22819ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227e00640 1 -- 172.21.15.146:0/1881833208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2281a6500 con 0x7fc2280785a0 2024-04-24T08:20:28.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc227400640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2280769c0 0x7fc22819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.421+0000 7fc21fe00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc228077c00 0x7fc22819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.422+0000 7fc227e00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280785a0 0x7fc2281a5e90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc22000efc0 tx=0x7fc22000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.422+0000 7fc224c00640 1 -- 172.21.15.146:0/1881833208 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220003970 con 0x7fc2280785a0 2024-04-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.422+0000 7fc224c00640 1 -- 172.21.15.146:0/1881833208 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc220004020 con 0x7fc2280785a0 2024-04-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.422+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2281a67f0 con 0x7fc2280785a0 2024-04-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.422+0000 7fc224c00640 1 -- 172.21.15.146:0/1881833208 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22000b9f0 con 0x7fc2280785a0 2024-04-24T08:20:28.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.422+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2281a6cf0 con 0x7fc2280785a0 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.425+0000 7fc2311db740 1 Processor -- start 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.425+0000 7fc224c00640 1 -- 172.21.15.146:0/1881833208 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fc22001d020 con 0x7fc2280785a0 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.425+0000 7fc2311db740 1 -- start start 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc2311db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433a260 0x564a8433a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc2311db740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8433f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc2311db740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433fdc0 0x564a84342250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc2311db740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564a8429fb10 con 0x564a8433ac50 2024-04-24T08:20:28.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc2311db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a842462b0 con 0x564a8433a260 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc2311db740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a8429bad0 con 0x564a8433fdc0 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8433f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8433f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38030/0 (socket says 172.21.15.146:38030) 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 -- 172.21.15.146:0/1373924455 learned_addr learned my addr 172.21.15.146:0/1373924455 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc21fe00640 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433fdc0 0x564a84342250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc21fe00640 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433a260 0x564a8433a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 -- 172.21.15.146:0/1373924455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433fdc0 msgr2=0x564a84342250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433fdc0 0x564a84342250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 -- 172.21.15.146:0/1373924455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433a260 msgr2=0x564a8433a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433a260 0x564a8433a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc227400640 1 -- 172.21.15.146:0/1373924455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a83fb22d0 con 0x564a8433ac50 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.426+0000 7fc21fe00640 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433a260 0x564a8433a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.427+0000 7fc224c00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc20c0a3bb0 0x7fc20c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.427+0000 7fc227400640 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8433f750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc210009830 tx=0x7fc210009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92c8a3f51a38ee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.427+0000 7fc224c00640 1 -- 172.21.15.146:0/1881833208 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc2200d02c0 con 0x7fc2280785a0 2024-04-24T08:20:28.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.427+0000 7fc21fe00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc20c0a3bb0 0x7fc20c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.427+0000 7fc217e00640 1 -- 172.21.15.146:0/1373924455 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210037040 con 0x564a8433ac50 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.427+0000 7fc217e00640 1 -- 172.21.15.146:0/1373924455 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc210042070 con 0x564a8433ac50 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.427+0000 7fc217e00640 1 -- 172.21.15.146:0/1373924455 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2100046c0 con 0x564a8433ac50 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 -- 172.21.15.146:0/1373924455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 msgr2=0x564a8433f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc21fe00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc20c0a3bb0 0x7fc20c0a6070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x564a8433b4b0 tx=0x7fc218013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8433f750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc210009830 tx=0x7fc210009560 comp rx=0 tx=0).stop 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 -- 172.21.15.146:0/1373924455 shutdown_connections 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433fdc0 0x564a84342250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8433f750 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 --2- 172.21.15.146:0/1373924455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433a260 0x564a8433a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 -- 172.21.15.146:0/1373924455 >> 172.21.15.146:0/1373924455 conn(0x564a843358d0 msgr2=0x564a84335cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 -- 172.21.15.146:0/1373924455 shutdown_connections 2024-04-24T08:20:28.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 -- 172.21.15.146:0/1373924455 wait complete. 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.428+0000 7fc2311db740 1 -- start start 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 1 Processor -- start 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84344620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a84344b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a8434bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564a84342d10 con 0x564a8433ac50 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a84259590 con 0x564a8433fdc0 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a84342e90 con 0x564a8433a260 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc2311db740 10 client.0 _ll_register_callbacks cb 0x564a8423c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a84344b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc227e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a8434bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a84344b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38046/0 (socket says 172.21.15.146:38046) 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc21fe00640 1 -- 172.21.15.146:0/4109072350 learned_addr learned my addr 172.21.15.146:0/4109072350 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.429+0000 7fc227400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84344620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc21fe00640 1 -- 172.21.15.146:0/4109072350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 msgr2=0x564a84344620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc21fe00640 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84344620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc21fe00640 1 -- 172.21.15.146:0/4109072350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 msgr2=0x564a8434bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc21fe00640 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a8434bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc21fe00640 1 -- 172.21.15.146:0/4109072350 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a8407c6b0 con 0x564a8433ac50 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc227e00640 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a8434bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc21fe00640 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a84344b60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc2180206f0 tx=0x7fc218004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265287b74f4e8a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc214c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc214c00640 1 -- 172.21.15.146:0/4109072350 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218030060 con 0x564a8433ac50 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc214c00640 1 -- 172.21.15.146:0/4109072350 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc21801a070 con 0x564a8433ac50 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc210042070 con 0x564a8433ac50 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc214c00640 1 -- 172.21.15.146:0/4109072350 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc21802b420 con 0x564a8433ac50 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc2311db740 10 client.14880 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:28.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.430+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a8434f030 con 0x564a8433ac50 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc214c00640 1 -- 172.21.15.146:0/4109072350 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc210042070 con 0x564a8433ac50 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc214c00640 1 -- 172.21.15.146:0/4109072350 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc218035070 con 0x564a8433ac50 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc2311db740 10 client.14880 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc2311db740 10 client.14880 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc2311db740 20 client.14880 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc2311db740 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a84351710 0x564a84352b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc2311db740 4 client.14880 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a842ad530 con 0x564a84351710 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.431+0000 7fc227400640 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a84351710 0x564a84352b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.432+0000 7fc227400640 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a84351710 0x564a84352b80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fdec9009f41a977d server_cookie=4d3c2a999d3232b1 in_seq=0 out_seq=0 2024-04-24T08:20:28.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.432+0000 7fc214c00640 10 client.14880 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:28.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc214c00640 1 -- 172.21.15.146:0/4109072350 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a842ad530 con 0x564a84351710 2024-04-24T08:20:28.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc214c00640 10 client.14880 handle_command_reply: tid=0 2024-04-24T08:20:28.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 client.14880 shutdown 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 msgr2=0x564a84344b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a84344b60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc2180206f0 tx=0x7fc218004730 comp rx=0 tx=0).stop 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 shutdown_connections 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a84351710 0x564a84352b80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a8434bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a84344b60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 --2- 172.21.15.146:0/4109072350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84344620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 >> 172.21.15.146:0/4109072350 conn(0x564a843358d0 msgr2=0x564a84334970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc214c00640 0 client.14880 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 shutdown_connections 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 1 -- 172.21.15.146:0/4109072350 wait complete. 2024-04-24T08:20:28.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.433+0000 7fc2311db740 20 client.14880 trim_cache size 0 max 16384 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.452+0000 7fc2311db740 1 Processor -- start 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc2311db740 1 -- start start 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc2311db740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84343950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc2311db740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8434bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc2311db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a84343e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc2311db740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564a84350ec0 con 0x564a8433ac50 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc2311db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a84351590 con 0x564a8433fdc0 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc2311db740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a84351c60 con 0x564a8433a260 2024-04-24T08:20:28.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8434bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8434bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38056/0 (socket says 172.21.15.146:38056) 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc21fe00640 1 -- 172.21.15.146:0/3602696694 learned_addr learned my addr 172.21.15.146:0/3602696694 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc227e00640 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a84343e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.453+0000 7fc227400640 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84343950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc21fe00640 1 -- 172.21.15.146:0/3602696694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 msgr2=0x564a84343950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc21fe00640 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84343950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc21fe00640 1 -- 172.21.15.146:0/3602696694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 msgr2=0x564a84343e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc21fe00640 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a84343e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc21fe00640 1 -- 172.21.15.146:0/3602696694 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a84335700 con 0x564a8433ac50 2024-04-24T08:20:28.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc21fe00640 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8434bb60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc21801e5e0 tx=0x7fc21801e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b523defbb77c2fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc217400640 1 -- 172.21.15.146:0/3602696694 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21803b070 con 0x564a8433ac50 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc217400640 1 -- 172.21.15.146:0/3602696694 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc21801ecb0 con 0x564a8433ac50 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.454+0000 7fc217400640 1 -- 172.21.15.146:0/3602696694 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21802b860 con 0x564a8433ac50 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 -- 172.21.15.146:0/3602696694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 msgr2=0x564a8434bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8434bb60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc21801e5e0 tx=0x7fc21801e3c0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 -- 172.21.15.146:0/3602696694 shutdown_connections 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a84343e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a8434bb60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 --2- 172.21.15.146:0/3602696694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84343950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 -- 172.21.15.146:0/3602696694 >> 172.21.15.146:0/3602696694 conn(0x564a843358d0 msgr2=0x564a84344620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 -- 172.21.15.146:0/3602696694 shutdown_connections 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 -- 172.21.15.146:0/3602696694 wait complete. 2024-04-24T08:20:28.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 -- start start 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.455+0000 7fc2311db740 1 Processor -- start 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc2311db740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84349ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc2311db740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a843b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc2311db740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a843b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc2311db740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564a84352a00 con 0x564a8433ac50 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc2311db740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a84352330 con 0x564a8433fdc0 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc2311db740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a843530d0 con 0x564a8433a260 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc2311db740 10 client.0 _ll_register_callbacks cb 0x7fc21800fd00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a843b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc227400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84349ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc21fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a843b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38060/0 (socket says 172.21.15.146:38060) 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc21fe00640 1 -- 172.21.15.146:0/3087209274 learned_addr learned my addr 172.21.15.146:0/3087209274 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc227e00640 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a843b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc227400640 1 -- 172.21.15.146:0/3087209274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 msgr2=0x564a843b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc227400640 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a843b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc227400640 1 -- 172.21.15.146:0/3087209274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 msgr2=0x564a843b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc227400640 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a843b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.456+0000 7fc227400640 1 -- 172.21.15.146:0/3087209274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a843b64f0 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc227400640 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84349ab0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x564a843367a0 tx=0x7fc210005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d7417b96bb327fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc214c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc214c00640 1 -- 172.21.15.146:0/3087209274 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210037040 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc21801ecb0 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc214c00640 1 -- 172.21.15.146:0/3087209274 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fc210030750 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc214c00640 1 -- 172.21.15.146:0/3087209274 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc210030950 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc2311db740 10 client.25006 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564a843b77f0 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.457+0000 7fc214c00640 1 -- 172.21.15.146:0/3087209274 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc21801ecb0 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc214c00640 1 -- 172.21.15.146:0/3087209274 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc2100035f0 con 0x564a8433a260 2024-04-24T08:20:28.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc2311db740 10 client.25006 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:28.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc2311db740 10 client.25006 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:20:28.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc2311db740 20 client.25006 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:28.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc2311db740 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a843bbf00 0x564a843bac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc2311db740 4 client.25006 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:20:28.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "status"}) -- 0x564a842a73a0 con 0x564a843bbf00 2024-04-24T08:20:28.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc21fe00640 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a843bbf00 0x564a843bac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc214c00640 10 client.25006 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:28.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.458+0000 7fc21fe00640 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a843bbf00 0x564a843bac10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f195c40b78a057c4 server_cookie=8811e0b97b2c393d in_seq=0 out_seq=0 2024-04-24T08:20:28.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc214c00640 1 -- 172.21.15.146:0/3087209274 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564a842a73a0 con 0x564a843bbf00 2024-04-24T08:20:28.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc214c00640 10 client.25006 handle_command_reply: tid=0 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 client.25006 shutdown 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 msgr2=0x564a84349ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84349ab0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x564a843367a0 tx=0x7fc210005f10 comp rx=0 tx=0).stop 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 shutdown_connections 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x564a843bbf00 0x564a843bac10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a8433fdc0 0x564a843b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc214c00640 0 client.25006 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a8433ac50 0x564a843b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 --2- 172.21.15.146:0/3087209274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8433a260 0x564a84349ab0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 >> 172.21.15.146:0/3087209274 conn(0x564a843358d0 msgr2=0x564a84343950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 shutdown_connections 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 1 -- 172.21.15.146:0/3087209274 wait complete. 2024-04-24T08:20:28.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.459+0000 7fc2311db740 20 client.25006 trim_cache size 0 max 16384 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 1, 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 24568, 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 97.965027734000003, 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 23, 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 90, 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 88, 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 109.024263178 2024-04-24T08:20:28.465 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc20c0a3bb0 msgr2=0x7fc20c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc20c0a3bb0 0x7fc20c0a6070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x564a8433b4b0 tx=0x7fc218013040 comp rx=0 tx=0).stop 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280785a0 msgr2=0x7fc2281a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280785a0 0x7fc2281a5e90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc22000efc0 tx=0x7fc22000c800 comp rx=0 tx=0).stop 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 shutdown_connections 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2280785a0 0x7fc2281a5e90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fc20c0a3bb0 0x7fc20c0a6070 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc228077c00 0x7fc22819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 --2- 172.21.15.146:0/1881833208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2280769c0 0x7fc22819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 >> 172.21.15.146:0/1881833208 conn(0x7fc2280ff6b0 msgr2=0x7fc2281011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 shutdown_connections 2024-04-24T08:20:28.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.460+0000 7fc22cc00640 1 -- 172.21.15.146:0/1881833208 wait complete. 2024-04-24T08:20:28.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T08:20:28.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.659+0000 7f830f400640 1 -- 172.21.15.146:0/2874796455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 msgr2=0x7f83081046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.659+0000 7f830f400640 1 --2- 172.21.15.146:0/2874796455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 0x7f83081046a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009900 tx=0x7f82f802a410 comp rx=0 tx=0).stop 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.660+0000 7f830f400640 1 -- 172.21.15.146:0/2874796455 shutdown_connections 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.660+0000 7f830f400640 1 --2- 172.21.15.146:0/2874796455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308105470 0x7f83081058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.660+0000 7f830f400640 1 --2- 172.21.15.146:0/2874796455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 0x7f830810c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.660+0000 7f830f400640 1 --2- 172.21.15.146:0/2874796455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 0x7f83081046a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.660+0000 7f830f400640 1 -- 172.21.15.146:0/2874796455 >> 172.21.15.146:0/2874796455 conn(0x7f83080ff7f0 msgr2=0x7f8308101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.660+0000 7f830f400640 1 -- 172.21.15.146:0/2874796455 shutdown_connections 2024-04-24T08:20:28.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.660+0000 7f830f400640 1 -- 172.21.15.146:0/2874796455 wait complete. 2024-04-24T08:20:28.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 Processor -- start 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 -- start start 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 0x7f830819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308105470 0x7f830819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 0x7f83081a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f830810ff60 con 0x7f8308105e30 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f830810fde0 con 0x7f8308104280 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830f400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f83081100e0 con 0x7f8308105470 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 0x7f83081a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 0x7f83081a1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38070/0 (socket says 172.21.15.146:38070) 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830d400640 1 -- 172.21.15.146:0/3820601882 learned_addr learned my addr 172.21.15.146:0/3820601882 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f8307e00640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308105470 0x7f830819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.661+0000 7f830ca00640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 0x7f830819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f8307e00640 1 -- 172.21.15.146:0/3820601882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 msgr2=0x7f830819a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f8307e00640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 0x7f830819a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f8307e00640 1 -- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 msgr2=0x7f83081a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f8307e00640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 0x7f83081a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f8307e00640 1 -- 172.21.15.146:0/3820601882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83081a21e0 con 0x7f8308105470 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f830d400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 0x7f83081a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f8307e00640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308105470 0x7f830819ab00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009fd0 tx=0x7f82f000ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.662+0000 7f8305600640 1 -- 172.21.15.146:0/3820601882 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0004b20 con 0x7f8308105470 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.663+0000 7f8305600640 1 -- 172.21.15.146:0/3820601882 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f0004cc0 con 0x7f8308105470 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.663+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83081a24d0 con 0x7f8308105470 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.663+0000 7f8305600640 1 -- 172.21.15.146:0/3820601882 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00107b0 con 0x7f8308105470 2024-04-24T08:20:28.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.663+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83081a2a20 con 0x7f8308105470 2024-04-24T08:20:28.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 Processor -- start 2024-04-24T08:20:28.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8305600640 1 -- 172.21.15.146:0/3820601882 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f82f0005020 con 0x7f8308105470 2024-04-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 -- start start 2024-04-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b3a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3aa80 0x5585f0b3f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3fbf0 0x5585f0b42080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5585f0a9fb10 con 0x5585f0b3fbf0 2024-04-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5585f0a462b0 con 0x5585f0b3a090 2024-04-24T08:20:28.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8312399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5585f0a9bad0 con 0x5585f0b3aa80 2024-04-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8307e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3fbf0 0x5585f0b42080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f830ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3aa80 0x5585f0b3f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f830ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3aa80 0x5585f0b3f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:51042/0 (socket says 172.21.15.146:51042) 2024-04-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f830ca00640 1 -- 172.21.15.146:0/2691590136 learned_addr learned my addr 172.21.15.146:0/2691590136 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f8307e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3fbf0 0x5585f0b42080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38086/0 (socket says 172.21.15.146:38086) 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.666+0000 7f830d400640 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b3a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f830ca00640 1 -- 172.21.15.146:0/2691590136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 msgr2=0x5585f0b3a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f830ca00640 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b3a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f830ca00640 1 -- 172.21.15.146:0/2691590136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3fbf0 msgr2=0x5585f0b42080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f830ca00640 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3fbf0 0x5585f0b42080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f830ca00640 1 -- 172.21.15.146:0/2691590136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f07b22d0 con 0x5585f0b3aa80 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f830ca00640 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3aa80 0x5585f0b3f580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f8004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81dcd7af8c3777fb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f8305600640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82e80a3c30 0x7f82e80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f82fdc00640 1 -- 172.21.15.146:0/2691590136 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8035040 con 0x5585f0b3aa80 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f82fdc00640 1 -- 172.21.15.146:0/2691590136 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f8004430 con 0x5585f0b3aa80 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f82fdc00640 1 -- 172.21.15.146:0/2691590136 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f80316b0 con 0x5585f0b3aa80 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f8305600640 1 -- 172.21.15.146:0/3820601882 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f82f00d0b40 con 0x7f8308105470 2024-04-24T08:20:28.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.667+0000 7f830d400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82e80a3c30 0x7f82e80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 -- 172.21.15.146:0/2691590136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3aa80 msgr2=0x5585f0b3f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3aa80 0x5585f0b3f580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f8004040 comp rx=0 tx=0).stop 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f830d400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82e80a3c30 0x7f82e80a60f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8300009fd0 tx=0x7f8300005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 -- 172.21.15.146:0/2691590136 shutdown_connections 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3aa80 0x5585f0b3f580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3fbf0 0x5585f0b42080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 --2- 172.21.15.146:0/2691590136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b3a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 -- 172.21.15.146:0/2691590136 >> 172.21.15.146:0/2691590136 conn(0x5585f0b35700 msgr2=0x5585f0b35b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 -- 172.21.15.146:0/2691590136 shutdown_connections 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.668+0000 7f8312399740 1 -- 172.21.15.146:0/2691590136 wait complete. 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 -- start start 2024-04-24T08:20:28.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 Processor -- start 2024-04-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b44530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3aa80 0x5585f0b44a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b4ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b42b40 con 0x5585f0b3aa80 2024-04-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5585f0a59590 con 0x5585f0b3a090 2024-04-24T08:20:28.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.669+0000 7f8312399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b42cc0 con 0x5585f0b3fbf0 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8312399740 10 client.0 _ll_register_callbacks cb 0x5585f0a3c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f830ca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b44530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8307e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3aa80 0x5585f0b44a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f830ca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b44530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58284/0 (socket says 172.21.15.146:58284) 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f830ca00640 1 -- 172.21.15.146:0/3425608955 learned_addr learned my addr 172.21.15.146:0/3425608955 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f830d400640 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b4ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8307e00640 1 -- 172.21.15.146:0/3425608955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 msgr2=0x5585f0b4ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8307e00640 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b4ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8307e00640 1 -- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 msgr2=0x5585f0b44530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8307e00640 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b44530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8307e00640 1 -- 172.21.15.146:0/3425608955 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f087c6b0 con 0x5585f0b3aa80 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f830ca00640 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b44530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8307e00640 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3aa80 0x5585f0b44a70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ce600 tx=0x7f82f00058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd3ee69e844e084 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f82f6000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f82f6000640 1 -- 172.21.15.146:0/3425608955 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f009cbb0 con 0x5585f0b3aa80 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f82f6000640 1 -- 172.21.15.146:0/3425608955 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f82f0005590 con 0x5585f0b3aa80 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.670+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f8004430 con 0x5585f0b3aa80 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f82f6000640 1 -- 172.21.15.146:0/3425608955 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f82f009c740 con 0x5585f0b3aa80 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 10 client.14898 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5585f0b4eed0 con 0x5585f0b3aa80 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f82f6000640 1 -- 172.21.15.146:0/3425608955 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f8004430 con 0x5585f0b3aa80 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f82f6000640 1 -- 172.21.15.146:0/3425608955 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f82f00b6150 con 0x5585f0b3aa80 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 10 client.14898 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 10 client.14898 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 20 client.14898 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0b515e0 0x5585f0b52a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 4 client.14898 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:28.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.671+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585f0aad530 con 0x5585f0b515e0 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.672+0000 7f830ca00640 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0b515e0 0x5585f0b52a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.672+0000 7f830ca00640 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0b515e0 0x5585f0b52a50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2d83551f5cbb80e server_cookie=fe6085f6ec77f576 in_seq=0 out_seq=0 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.672+0000 7f82f6000640 10 client.14898 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.673+0000 7f82f6000640 1 -- 172.21.15.146:0/3425608955 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585f0aad530 con 0x5585f0b515e0 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.673+0000 7f82f6000640 10 client.14898 handle_command_reply: tid=0 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 client.14898 shutdown 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3aa80 msgr2=0x5585f0b44a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3aa80 0x5585f0b44a70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ce600 tx=0x7f82f00058e0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 shutdown_connections 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0b515e0 0x5585f0b52a50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b4ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3aa80 0x5585f0b44a70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 --2- 172.21.15.146:0/3425608955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3a090 0x5585f0b44530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f82f6000640 0 client.14898 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:20:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 >> 172.21.15.146:0/3425608955 conn(0x5585f0b35700 msgr2=0x5585f0b348c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 shutdown_connections 2024-04-24T08:20:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 1 -- 172.21.15.146:0/3425608955 wait complete. 2024-04-24T08:20:28.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.674+0000 7f8312399740 20 client.14898 trim_cache size 0 max 16384 2024-04-24T08:20:28.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.693+0000 7f8312399740 1 Processor -- start 2024-04-24T08:20:28.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8312399740 1 -- start start 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8312399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b30750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8312399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0b4ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8312399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b44140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8312399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b50d90 con 0x5585f0b3a090 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8312399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b51460 con 0x5585f0b3aa80 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8312399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b51b30 con 0x5585f0b3fbf0 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f830ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b30750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f830ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b30750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38104/0 (socket says 172.21.15.146:38104) 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f830ca00640 1 -- 172.21.15.146:0/1767134772 learned_addr learned my addr 172.21.15.146:0/1767134772 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f830d400640 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b44140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f8307e00640 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0b4ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f830ca00640 1 -- 172.21.15.146:0/1767134772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 msgr2=0x5585f0b44140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.694+0000 7f830ca00640 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b44140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f830ca00640 1 -- 172.21.15.146:0/1767134772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 msgr2=0x5585f0b4ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f830ca00640 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0b4ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f830ca00640 1 -- 172.21.15.146:0/1767134772 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f0b35530 con 0x5585f0b3a090 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f8307e00640 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0b4ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f830ca00640 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b30750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f82f802aa00 tx=0x7f82f80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17de5279aab2d5a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f82fd200640 1 -- 172.21.15.146:0/1767134772 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8035050 con 0x5585f0b3a090 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f82fd200640 1 -- 172.21.15.146:0/1767134772 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f8004a70 con 0x5585f0b3a090 2024-04-24T08:20:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.695+0000 7f82fd200640 1 -- 172.21.15.146:0/1767134772 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8052630 con 0x5585f0b3a090 2024-04-24T08:20:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 -- 172.21.15.146:0/1767134772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 msgr2=0x5585f0b30750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b30750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f82f802aa00 tx=0x7f82f80045e0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 -- 172.21.15.146:0/1767134772 shutdown_connections 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0b44140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0b4ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 --2- 172.21.15.146:0/1767134772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b30750 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 -- 172.21.15.146:0/1767134772 >> 172.21.15.146:0/1767134772 conn(0x5585f0b35700 msgr2=0x5585f0b35ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 -- 172.21.15.146:0/1767134772 shutdown_connections 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 -- 172.21.15.146:0/1767134772 wait complete. 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 -- start start 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.696+0000 7f8312399740 1 Processor -- start 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8312399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8312399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0bbbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8312399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0bbe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8312399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b528d0 con 0x5585f0b3a090 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8312399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b52200 con 0x5585f0b3aa80 2024-04-24T08:20:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8312399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5585f0b52fa0 con 0x5585f0b3fbf0 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8312399740 10 client.0 _ll_register_callbacks cb 0x7f82f00d4270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f830ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8307e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0bbbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f830ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38106/0 (socket says 172.21.15.146:38106) 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f830ca00640 1 -- 172.21.15.146:0/2535388068 learned_addr learned my addr 172.21.15.146:0/2535388068 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f830d400640 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0bbe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8307e00640 1 -- 172.21.15.146:0/2535388068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 msgr2=0x5585f0bbe2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8307e00640 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0bbe2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8307e00640 1 -- 172.21.15.146:0/2535388068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 msgr2=0x5585f0b499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8307e00640 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f8307e00640 1 -- 172.21.15.146:0/2535388068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f0bbe810 con 0x5585f0b3aa80 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.697+0000 7f830ca00640 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f8307e00640 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0bbbd80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f82f009cae0 tx=0x7f82f00d4320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caf9d9d786b59f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f82f6000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f82f6000640 1 -- 172.21.15.146:0/2535388068 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f007f020 con 0x5585f0b3aa80 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f8004a70 con 0x5585f0b3aa80 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f82f6000640 1 -- 172.21.15.146:0/2535388068 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f82f00b6150 con 0x5585f0b3aa80 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f82f6000640 1 -- 172.21.15.146:0/2535388068 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f82f009c920 con 0x5585f0b3aa80 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f8312399740 10 client.25016 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5585f0bbfa20 con 0x5585f0b3aa80 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.698+0000 7f82f6000640 1 -- 172.21.15.146:0/2535388068 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f8004a70 con 0x5585f0b3aa80 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f82f6000640 1 -- 172.21.15.146:0/2535388068 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f82f00ceac0 con 0x5585f0b3aa80 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f8312399740 10 client.25016 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f8312399740 10 client.25016 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f8312399740 20 client.25016 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f8312399740 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0bc4470 0x5585f0bc3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f8312399740 4 client.25016 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:20:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "status"}) -- 0x5585f0aa73a0 con 0x5585f0bc4470 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f830ca00640 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0bc4470 0x5585f0bc3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f830ca00640 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0bc4470 0x5585f0bc3180 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bf0fa9ae3e6201e server_cookie=318c2306886f2980 in_seq=0 out_seq=0 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.699+0000 7f82f6000640 10 client.25016 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f82f6000640 1 -- 172.21.15.146:0/2535388068 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5585f0aa73a0 con 0x5585f0bc4470 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f82f6000640 10 client.25016 handle_command_reply: tid=0 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 client.25016 shutdown 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 msgr2=0x5585f0bbbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0bbbd80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f82f009cae0 tx=0x7f82f00d4320 comp rx=0 tx=0).stop 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 shutdown_connections 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5585f0bc4470 0x5585f0bc3180 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5585f0b3fbf0 0x5585f0bbe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5585f0b3aa80 0x5585f0bbbd80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 --2- 172.21.15.146:0/2535388068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5585f0b3a090 0x5585f0b499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f82f6000640 0 client.25016 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:20:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 >> 172.21.15.146:0/2535388068 conn(0x5585f0b35700 msgr2=0x5585f0b30750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 shutdown_connections 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 1 -- 172.21.15.146:0/2535388068 wait complete. 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.700+0000 7f8312399740 20 client.25016 trim_cache size 0 max 16384 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 24569, 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-24T08:20:28.706 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-24T08:20:28.707 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 101.246218963, 2024-04-24T08:20:28.707 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 23, 2024-04-24T08:20:28.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 90, 2024-04-24T08:20:28.707 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 88, 2024-04-24T08:20:28.707 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 106.499272073 2024-04-24T08:20:28.707 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:20:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82e80a3c30 msgr2=0x7f82e80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82e80a3c30 0x7f82e80a60f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8300009fd0 tx=0x7f8300005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308105470 msgr2=0x7f830819ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308105470 0x7f830819ab00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009fd0 tx=0x7f82f000ef90 comp rx=0 tx=0).stop 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 shutdown_connections 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8308105470 0x7f830819ab00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f82e80a3c30 0x7f82e80a60f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8308105e30 0x7f83081a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 --2- 172.21.15.146:0/3820601882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8308104280 0x7f830819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 >> 172.21.15.146:0/3820601882 conn(0x7f83080ff7f0 msgr2=0x7f8308108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 shutdown_connections 2024-04-24T08:20:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:28.701+0000 7f830f400640 1 -- 172.21.15.146:0/3820601882 wait complete. 2024-04-24T08:20:28.718 INFO:tasks.cephadm:Setup complete, yielding 2024-04-24T08:20:28.718 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T08:20:28.729 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:20:28.729 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-24T08:20:28.793 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:28 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3233545271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:28.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:28 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3233545271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:28 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3233545271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:29.169 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:29.832 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:29 smithi146 ceph-mon[27805]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:29.861 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:29 smithi153 ceph-mon[32746]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:30.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:29 smithi177 ceph-mon[33046]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.488+0000 7f0152200700 1 -- 172.21.15.146:0/4174786607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 msgr2=0x7f014c1062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.488+0000 7f0152200700 1 --2- 172.21.15.146:0/4174786607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 0x7f014c1062f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01380098f0 tx=0x7f0138009c00 comp rx=0 tx=0).stop 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.489+0000 7f0152200700 1 -- 172.21.15.146:0/4174786607 shutdown_connections 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.489+0000 7f0152200700 1 --2- 172.21.15.146:0/4174786607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f014c106830 0x7f014c10d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.489+0000 7f0152200700 1 --2- 172.21.15.146:0/4174786607 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 0x7f014c1050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.489+0000 7f0152200700 1 --2- 172.21.15.146:0/4174786607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 0x7f014c1062f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.489+0000 7f0152200700 1 -- 172.21.15.146:0/4174786607 >> 172.21.15.146:0/4174786607 conn(0x7f014c100230 msgr2=0x7f014c102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:31.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.489+0000 7f0152200700 1 -- 172.21.15.146:0/4174786607 shutdown_connections 2024-04-24T08:20:31.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.489+0000 7f0152200700 1 -- 172.21.15.146:0/4174786607 wait complete. 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.490+0000 7f0152200700 1 Processor -- start 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.490+0000 7f0152200700 1 -- start start 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0152200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 0x7f014c19aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0152200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 0x7f014c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0152200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f014c106830 0x7f014c1a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0152200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f014c1108a0 con 0x7f014c106830 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0152200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f014c110720 con 0x7f014c104c90 2024-04-24T08:20:31.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0152200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f014c110a20 con 0x7f014c105e90 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 0x7f014c19aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 0x7f014c19aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37822/0 (socket says 172.21.15.146:37822) 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 -- 172.21.15.146:0/500153202 learned_addr learned my addr 172.21.15.146:0/500153202 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0151800700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f014c106830 0x7f014c1a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f014be00700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 0x7f014c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 -- 172.21.15.146:0/500153202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 msgr2=0x7f014c19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 0x7f014c19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 -- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f014c106830 msgr2=0x7f014c1a24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f014c106830 0x7f014c1a24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.491+0000 7f0150e00700 1 -- 172.21.15.146:0/500153202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014c1a2b40 con 0x7f014c104c90 2024-04-24T08:20:31.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.492+0000 7f0150e00700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 0x7f014c19aee0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f014000e970 tx=0x7f014000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:31.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.492+0000 7f0149600700 1 -- 172.21.15.146:0/500153202 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140004220 con 0x7f014c104c90 2024-04-24T08:20:31.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.492+0000 7f0149600700 1 -- 172.21.15.146:0/500153202 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01400043c0 con 0x7f014c104c90 2024-04-24T08:20:31.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.492+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014c1a2e30 con 0x7f014c104c90 2024-04-24T08:20:31.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.492+0000 7f0149600700 1 -- 172.21.15.146:0/500153202 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140003df0 con 0x7f014c104c90 2024-04-24T08:20:31.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.492+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014c1a3290 con 0x7f014c104c90 2024-04-24T08:20:31.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.493+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f014c0697e0 con 0x7f014c104c90 2024-04-24T08:20:31.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.495+0000 7f0149600700 1 -- 172.21.15.146:0/500153202 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f014000c7e0 con 0x7f014c104c90 2024-04-24T08:20:31.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.497+0000 7f0149600700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01340a3cc0 0x7f01340a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:31.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.497+0000 7f0149600700 1 -- 172.21.15.146:0/500153202 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f01400d0750 con 0x7f014c104c90 2024-04-24T08:20:31.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.497+0000 7f014be00700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01340a3cc0 0x7f01340a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:31.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.498+0000 7f014be00700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01340a3cc0 0x7f01340a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01380098c0 tx=0x7f013801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:31.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.499+0000 7f0149600700 1 -- 172.21.15.146:0/500153202 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f014009d0f0 con 0x7f014c104c90 2024-04-24T08:20:31.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.749+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f014c062f30 con 0x7f01340a3cc0 2024-04-24T08:20:31.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.751+0000 7f0149600700 1 -- 172.21.15.146:0/500153202 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f014c062f30 con 0x7f01340a3cc0 2024-04-24T08:20:31.752 INFO:teuthology.orchestra.run.smithi146.stdout:Backend: cephadm 2024-04-24T08:20:31.753 INFO:teuthology.orchestra.run.smithi146.stdout:Available: Yes 2024-04-24T08:20:31.753 INFO:teuthology.orchestra.run.smithi146.stdout:Paused: No 2024-04-24T08:20:31.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01340a3cc0 msgr2=0x7f01340a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:31.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01340a3cc0 0x7f01340a6180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01380098c0 tx=0x7f013801a040 comp rx=0 tx=0).stop 2024-04-24T08:20:31.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 msgr2=0x7f014c19aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:31.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 0x7f014c19aee0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f014000e970 tx=0x7f014000ec80 comp rx=0 tx=0).stop 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 shutdown_connections 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f01340a3cc0 0x7f01340a6180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f014c106830 0x7f014c1a24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f014c104c90 0x7f014c19aee0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 --2- 172.21.15.146:0/500153202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f014c105e90 0x7f014c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.754+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 >> 172.21.15.146:0/500153202 conn(0x7f014c100230 msgr2=0x7f014c109210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.755+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 shutdown_connections 2024-04-24T08:20:31.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:31.755+0000 7f0152200700 1 -- 172.21.15.146:0/500153202 wait complete. 2024-04-24T08:20:31.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:31 smithi153 ceph-mon[32746]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:31.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:31 smithi146 ceph-mon[27805]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:32.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:31 smithi177 ceph-mon[33046]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:32.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-24T08:20:33.004 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:33.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:33 smithi146 ceph-mon[27805]: from='client.25022 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:33.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:33 smithi146 ceph-mon[27805]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:33.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:33 smithi153 ceph-mon[32746]: from='client.25022 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:33 smithi153 ceph-mon[32746]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:34.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:33 smithi177 ceph-mon[33046]: from='client.25022 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:34.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:33 smithi177 ceph-mon[33046]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4255315724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0103c90 msgr2=0x7f0bf01040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4255315724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0103c90 0x7f0bf01040f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80099c0 tx=0x7f0bd8009cd0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4255315724 shutdown_connections 2024-04-24T08:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4255315724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 0x7f0bf010aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4255315724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bf0102a90 0x7f0bf0102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4255315724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0103c90 0x7f0bf01040f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4255315724 >> 172.21.15.146:0/4255315724 conn(0x7f0bf00fe030 msgr2=0x7f0bf0100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4255315724 shutdown_connections 2024-04-24T08:20:34.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.995+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4255315724 wait complete. 2024-04-24T08:20:34.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.996+0000 7f0bf5a00700 1 Processor -- start 2024-04-24T08:20:34.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.996+0000 7f0bf5a00700 1 -- start start 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5a00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0102a90 0x7f0bf019aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bf0103c90 0x7f0bf019b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 0x7f0bf01a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5a00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf010e680 con 0x7f0bf0102a90 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf010e500 con 0x7f0bf0103c90 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf010e800 con 0x7f0bf0104630 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0befe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0102a90 0x7f0bf019aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 0x7f0bf01a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 0x7f0bf01a23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38594/0 (socket says 172.21.15.146:38594) 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0befe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0102a90 0x7f0bf019aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60798/0 (socket says 172.21.15.146:60798) 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0bf5000700 1 -- 172.21.15.146:0/4281530584 learned_addr learned my addr 172.21.15.146:0/4281530584 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.997+0000 7f0be7e00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bf0103c90 0x7f0bf019b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5000700 1 -- 172.21.15.146:0/4281530584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bf0103c90 msgr2=0x7f0bf019b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5000700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bf0103c90 0x7f0bf019b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5000700 1 -- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0102a90 msgr2=0x7f0bf019aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5000700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0102a90 0x7f0bf019aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5000700 1 -- 172.21.15.146:0/4281530584 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf01a2a90 con 0x7f0bf0104630 2024-04-24T08:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5000700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 0x7f0bf01a23f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0be800ef60 tx=0x7f0be800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bed600700 1 -- 172.21.15.146:0/4281530584 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be80092e0 con 0x7f0bf0104630 2024-04-24T08:20:35.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bed600700 1 -- 172.21.15.146:0/4281530584 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0be8004bb0 con 0x7f0bf0104630 2024-04-24T08:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf01a2d20 con 0x7f0bf0104630 2024-04-24T08:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bed600700 1 -- 172.21.15.146:0/4281530584 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be8004280 con 0x7f0bf0104630 2024-04-24T08:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.998+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf01a31b0 con 0x7f0bf0104630 2024-04-24T08:20:35.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:34.999+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf00697e0 con 0x7f0bf0104630 2024-04-24T08:20:35.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.002+0000 7f0bed600700 1 -- 172.21.15.146:0/4281530584 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f0be801b030 con 0x7f0bf0104630 2024-04-24T08:20:35.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.003+0000 7f0bed600700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0bd40a3c40 0x7f0bd40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:35.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.003+0000 7f0bed600700 1 -- 172.21.15.146:0/4281530584 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0be80d0830 con 0x7f0bf0104630 2024-04-24T08:20:35.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.003+0000 7f0befe00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0bd40a3c40 0x7f0bd40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:35.004 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.004+0000 7f0befe00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0bd40a3c40 0x7f0bd40a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009680 tx=0x7f0be0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:35.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.005+0000 7f0bed600700 1 -- 172.21.15.146:0/4281530584 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0be8022020 con 0x7f0bf0104630 2024-04-24T08:20:35.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.243+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f0bf0076440 con 0x7f0bd40a3c40 2024-04-24T08:20:35.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.263+0000 7f0bed600700 1 -- 172.21.15.146:0/4281530584 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f0bf0076440 con 0x7f0bd40a3c40 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.a smithi146 running (116s) 23s ago 115s 17.0M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 16a5089a1089 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.b smithi153 running (113s) 44s ago 112s 26.6M - 19.0.0-3116-g1f81fb39 b98aea5aa06f e1abbd03208f 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.c smithi177 running (2m) 45s ago 2m 19.2M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 6f268dd544ed 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.d smithi146 running (2m) 23s ago 2m 12.5M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 6d99d3073189 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.e smithi153 running (2m) 44s ago 2m 13.9M - 19.0.0-3116-g1f81fb39 b98aea5aa06f aee38ccaeee3 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.f smithi177 running (2m) 45s ago 2m 13.6M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 5c73c36cb987 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.g smithi146 running (2m) 23s ago 119s 16.5M - 19.0.0-3116-g1f81fb39 b98aea5aa06f ad5bb208c020 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.h smithi153 running (2m) 44s ago 2m 19.2M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 8bacd1b36dfe 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.i smithi177 running (2m) 45s ago 2m 19.4M - 19.0.0-3116-g1f81fb39 b98aea5aa06f bdc88fcb6b87 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.j smithi146 running (2m) 23s ago 2m 12.6M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 2e91614dea60 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.k smithi153 running (2m) 44s ago 2m 13.8M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 5015291c49b3 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mds.l smithi177 running (2m) 45s ago 2m 15.2M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 8f4f881ab347 2024-04-24T08:20:35.265 INFO:teuthology.orchestra.run.smithi146.stdout:mgr.x smithi146 *:9283,8765 running (10m) 23s ago 10m 515M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 55a3410c0b3b 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:mgr.y smithi153 *:8443,8765 running (8m) 44s ago 8m 454M - 19.0.0-3116-g1f81fb39 b98aea5aa06f a1ab8f6ee4fb 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:mgr.z smithi177 *:8443,8765 running (8m) 45s ago 8m 454M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 78209c103d09 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:mon.a smithi146 running (10m) 23s ago 10m 48.4M 2048M 19.0.0-3116-g1f81fb39 b98aea5aa06f efc8b69559a5 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:mon.b smithi153 running (8m) 44s ago 8m 41.9M 2048M 19.0.0-3116-g1f81fb39 b98aea5aa06f 42f097a73970 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:mon.c smithi177 running (8m) 45s ago 8m 43.5M 2048M 19.0.0-3116-g1f81fb39 b98aea5aa06f 7230be4a4d0b 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0 smithi146 running (7m) 23s ago 7m 48.7M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 0a1d79605838 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1 smithi153 running (7m) 44s ago 7m 54.8M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 26631e674a31 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2 smithi177 running (7m) 45s ago 7m 51.9M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f e1d67b4161fe 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3 smithi146 running (6m) 23s ago 6m 46.7M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f d239f799fda4 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4 smithi153 running (6m) 44s ago 6m 54.6M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 12e4dc0d8ba3 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5 smithi177 running (5m) 45s ago 5m 55.3M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 31264f21173c 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6 smithi146 running (5m) 23s ago 5m 46.9M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 2c57afc8ac37 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7 smithi153 running (4m) 44s ago 4m 46.4M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 4fdb395010ca 2024-04-24T08:20:35.266 INFO:teuthology.orchestra.run.smithi146.stdout:osd.8 smithi177 running (4m) 45s ago 4m 56.9M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 526e4b73ae85 2024-04-24T08:20:35.267 INFO:teuthology.orchestra.run.smithi146.stdout:osd.9 smithi146 running (3m) 23s ago 3m 50.0M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 5b753f0fa08c 2024-04-24T08:20:35.267 INFO:teuthology.orchestra.run.smithi146.stdout:osd.10 smithi153 running (3m) 44s ago 3m 44.3M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 8ee83a9a3b76 2024-04-24T08:20:35.267 INFO:teuthology.orchestra.run.smithi146.stdout:osd.11 smithi177 running (2m) 45s ago 2m 43.9M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 2c387404e5da 2024-04-24T08:20:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0bd40a3c40 msgr2=0x7f0bd40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0bd40a3c40 0x7f0bd40a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009680 tx=0x7f0be0009260 comp rx=0 tx=0).stop 2024-04-24T08:20:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 msgr2=0x7f0bf01a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 0x7f0bf01a23f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0be800ef60 tx=0x7f0be800ef90 comp rx=0 tx=0).stop 2024-04-24T08:20:35.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 shutdown_connections 2024-04-24T08:20:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bf0104630 0x7f0bf01a23f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0bd40a3c40 0x7f0bd40a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf0102a90 0x7f0bf019aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 --2- 172.21.15.146:0/4281530584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bf0103c90 0x7f0bf019b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.267+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 >> 172.21.15.146:0/4281530584 conn(0x7f0bf00fe030 msgr2=0x7f0bf0074d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.268+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 shutdown_connections 2024-04-24T08:20:35.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:35.268+0000 7f0bf5a00700 1 -- 172.21.15.146:0/4281530584 wait complete. 2024-04-24T08:20:35.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:35 smithi153 ceph-mon[32746]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:35.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:35 smithi146 ceph-mon[27805]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:36.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:35 smithi177 ceph-mon[33046]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:36.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-24T08:20:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:36.862 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:36 smithi146 ceph-mon[27805]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:36.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:36 smithi153 ceph-mon[32746]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:37.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:36 smithi177 ceph-mon[33046]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:37.875 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:37 smithi146 ceph-mon[27805]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:37.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:37 smithi153 ceph-mon[32746]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:38.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:37 smithi177 ceph-mon[33046]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.763+0000 7ff3eca00700 1 -- 172.21.15.146:0/4164501178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8076b20 msgr2=0x7ff3e8074f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.763+0000 7ff3eca00700 1 --2- 172.21.15.146:0/4164501178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8076b20 0x7ff3e8074f80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009990 tx=0x7ff3d8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:20:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.763+0000 7ff3eca00700 1 -- 172.21.15.146:0/4164501178 shutdown_connections 2024-04-24T08:20:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.763+0000 7ff3eca00700 1 --2- 172.21.15.146:0/4164501178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e810aa20 0x7ff3e810ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.763+0000 7ff3eca00700 1 --2- 172.21.15.146:0/4164501178 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8076b20 0x7ff3e8074f80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.763+0000 7ff3eca00700 1 --2- 172.21.15.146:0/4164501178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3e8075550 0x7ff3e80759d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:38.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.763+0000 7ff3eca00700 1 -- 172.21.15.146:0/4164501178 >> 172.21.15.146:0/4164501178 conn(0x7ff3e80fe0f0 msgr2=0x7ff3e8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:38.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.764+0000 7ff3eca00700 1 -- 172.21.15.146:0/4164501178 shutdown_connections 2024-04-24T08:20:38.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.764+0000 7ff3eca00700 1 -- 172.21.15.146:0/4164501178 wait complete. 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.765+0000 7ff3eca00700 1 Processor -- start 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.765+0000 7ff3eca00700 1 -- start start 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3eca00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8075550 0x7ff3e819afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3eca00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e8076b20 0x7ff3e819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3eca00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3e810aa20 0x7ff3e81a2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3eca00700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8110910 con 0x7ff3e8076b20 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3eca00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8110790 con 0x7ff3e8075550 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3eca00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8110a90 con 0x7ff3e810aa20 2024-04-24T08:20:38.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3e5e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8075550 0x7ff3e819afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3dfe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e8076b20 0x7ff3e819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3e5e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8075550 0x7ff3e819afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37850/0 (socket says 172.21.15.146:37850) 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3e5e00700 1 -- 172.21.15.146:0/2886310522 learned_addr learned my addr 172.21.15.146:0/2886310522 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.766+0000 7ff3e6800700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3e810aa20 0x7ff3e81a2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3e5e00700 1 -- 172.21.15.146:0/2886310522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3e810aa20 msgr2=0x7ff3e81a2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3e5e00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3e810aa20 0x7ff3e81a2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3e5e00700 1 -- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e8076b20 msgr2=0x7ff3e819b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:38.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3e5e00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e8076b20 0x7ff3e819b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3e5e00700 1 -- 172.21.15.146:0/2886310522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e81a2c10 con 0x7ff3e8075550 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3e6800700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3e810aa20 0x7ff3e81a2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3e5e00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8075550 0x7ff3e819afb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8005f50 tx=0x7ff3d800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3dea00700 1 -- 172.21.15.146:0/2886310522 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8027070 con 0x7ff3e8075550 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3dea00700 1 -- 172.21.15.146:0/2886310522 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d8022950 con 0x7ff3e8075550 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e81a2ea0 con 0x7ff3e8075550 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.767+0000 7ff3dea00700 1 -- 172.21.15.146:0/2886310522 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8017650 con 0x7ff3e8075550 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.768+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e81a3290 con 0x7ff3e8075550 2024-04-24T08:20:38.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.769+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e80697e0 con 0x7ff3e8075550 2024-04-24T08:20:38.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.770+0000 7ff3dea00700 1 -- 172.21.15.146:0/2886310522 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7ff3d802a070 con 0x7ff3e8075550 2024-04-24T08:20:38.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.772+0000 7ff3dea00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff3cc0a3cc0 0x7ff3cc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:38.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.772+0000 7ff3dea00700 1 -- 172.21.15.146:0/2886310522 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff3d80d1370 con 0x7ff3e8075550 2024-04-24T08:20:38.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.772+0000 7ff3dfe00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff3cc0a3cc0 0x7ff3cc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:38.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.773+0000 7ff3dfe00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff3cc0a3cc0 0x7ff3cc0a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8076850 tx=0x7ff3d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:38.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:38.774+0000 7ff3dea00700 1 -- 172.21.15.146:0/2886310522 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3d8013070 con 0x7ff3e8075550 2024-04-24T08:20:38.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:38 smithi153 ceph-mon[32746]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:38.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:38 smithi146 ceph-mon[27805]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:39.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.007+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ff3e8062f30 con 0x7ff3cc0a3cc0 2024-04-24T08:20:39.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.015+0000 7ff3dea00700 1 -- 172.21.15.146:0/2886310522 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ff3e8062f30 con 0x7ff3cc0a3cc0 2024-04-24T08:20:39.016 INFO:teuthology.orchestra.run.smithi146.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-24T08:20:39.016 INFO:teuthology.orchestra.run.smithi146.stdout:mds.all 12/12 48s ago 2m smithi146=a;smithi146=d;smithi146=g;smithi146=j;smithi153=b;smithi153=e;smithi153=h;smithi153=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-24T08:20:39.016 INFO:teuthology.orchestra.run.smithi146.stdout:mgr 3/3 48s ago 8m smithi146=x;smithi153=y;smithi177=z;count:3 2024-04-24T08:20:39.016 INFO:teuthology.orchestra.run.smithi146.stdout:mon 3/3 48s ago 8m smithi146:172.21.15.146=a;smithi153:172.21.15.153=b;smithi177:172.21.15.177=c;count:3 2024-04-24T08:20:39.016 INFO:teuthology.orchestra.run.smithi146.stdout:osd 12 48s ago - 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff3cc0a3cc0 msgr2=0x7ff3cc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff3cc0a3cc0 0x7ff3cc0a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8076850 tx=0x7ff3d0009260 comp rx=0 tx=0).stop 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8075550 msgr2=0x7ff3e819afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8075550 0x7ff3e819afb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8005f50 tx=0x7ff3d800f690 comp rx=0 tx=0).stop 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 shutdown_connections 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3e810aa20 0x7ff3e81a2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3e8075550 0x7ff3e819afb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff3cc0a3cc0 0x7ff3cc0a6180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 --2- 172.21.15.146:0/2886310522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e8076b20 0x7ff3e819b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 >> 172.21.15.146:0/2886310522 conn(0x7ff3e80fe0f0 msgr2=0x7ff3e8106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 shutdown_connections 2024-04-24T08:20:39.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:39.018+0000 7ff3eca00700 1 -- 172.21.15.146:0/2886310522 wait complete. 2024-04-24T08:20:39.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:38 smithi177 ceph-mon[33046]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:39.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:39 smithi146 ceph-mon[27805]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:39.955 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-24T08:20:39.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:39 smithi153 ceph-mon[32746]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:40.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:39 smithi177 ceph-mon[33046]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:40.420 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:40.906 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:40 smithi146 ceph-mon[27805]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:40.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:40 smithi153 ceph-mon[32746]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:40 smithi177 ceph-mon[33046]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:42.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 -- 172.21.15.146:0/1868132283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800102a90 msgr2=0x7f3800102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 --2- 172.21.15.146:0/1868132283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800102a90 0x7f3800102eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37f00099c0 tx=0x7f37f0009cd0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 -- 172.21.15.146:0/1868132283 shutdown_connections 2024-04-24T08:20:42.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 --2- 172.21.15.146:0/1868132283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3800104630 0x7f380010aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 --2- 172.21.15.146:0/1868132283 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800102a90 0x7f3800102eb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 --2- 172.21.15.146:0/1868132283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800103c90 0x7f38001040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 -- 172.21.15.146:0/1868132283 >> 172.21.15.146:0/1868132283 conn(0x7f38000fe030 msgr2=0x7f3800100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:42.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 -- 172.21.15.146:0/1868132283 shutdown_connections 2024-04-24T08:20:42.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.600+0000 7f3805600700 1 -- 172.21.15.146:0/1868132283 wait complete. 2024-04-24T08:20:42.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.602+0000 7f3805600700 1 Processor -- start 2024-04-24T08:20:42.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.603+0000 7f3805600700 1 -- start start 2024-04-24T08:20:42.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.603+0000 7f3805600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800102a90 0x7f3800076890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.603+0000 7f3805600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3800103c90 0x7f3800074ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.603+0000 7f3805600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800104630 0x7f3800075420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.603+0000 7f3805600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f380010e6c0 con 0x7f3800102a90 2024-04-24T08:20:42.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.603+0000 7f3805600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f380010e540 con 0x7f3800104630 2024-04-24T08:20:42.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.603+0000 7f3805600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f380010e840 con 0x7f3800103c90 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800102a90 0x7f3800076890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37f7400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3800103c90 0x7f3800074ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800102a90 0x7f3800076890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45184/0 (socket says 172.21.15.146:45184) 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 -- 172.21.15.146:0/1447316131 learned_addr learned my addr 172.21.15.146:0/1447316131 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f3804c00700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800104630 0x7f3800075420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 -- 172.21.15.146:0/1447316131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3800103c90 msgr2=0x7f3800074ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3800103c90 0x7f3800074ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 -- 172.21.15.146:0/1447316131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800104630 msgr2=0x7f3800075420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800104630 0x7f3800075420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f37ffe00700 1 -- 172.21.15.146:0/1447316131 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3800075cd0 con 0x7f3800102a90 2024-04-24T08:20:42.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.604+0000 7f3804c00700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800104630 0x7f3800075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:42.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.605+0000 7f37ffe00700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800102a90 0x7f3800076890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37f00098e0 tx=0x7f37f0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.605+0000 7f37fd600700 1 -- 172.21.15.146:0/1447316131 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0027070 con 0x7f3800102a90 2024-04-24T08:20:42.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.605+0000 7f37fd600700 1 -- 172.21.15.146:0/1447316131 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f0022470 con 0x7f3800102a90 2024-04-24T08:20:42.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.605+0000 7f37fd600700 1 -- 172.21.15.146:0/1447316131 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0009e60 con 0x7f3800102a90 2024-04-24T08:20:42.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.605+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38001a2db0 con 0x7f3800102a90 2024-04-24T08:20:42.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.605+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38001a31b0 con 0x7f3800102a90 2024-04-24T08:20:42.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.606+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38000697e0 con 0x7f3800102a90 2024-04-24T08:20:42.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.608+0000 7f37fd600700 1 -- 172.21.15.146:0/1447316131 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f37f002a070 con 0x7f3800102a90 2024-04-24T08:20:42.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.610+0000 7f37fd600700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37e40a3cc0 0x7f37e40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.610+0000 7f37fd600700 1 -- 172.21.15.146:0/1447316131 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f37f00d1380 con 0x7f3800102a90 2024-04-24T08:20:42.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.610+0000 7f37f7400700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37e40a3cc0 0x7f37e40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.611+0000 7f37f7400700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37e40a3cc0 0x7f37e40a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009c60 tx=0x7f37e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.613+0000 7f37fd600700 1 -- 172.21.15.146:0/1447316131 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37f0013070 con 0x7f3800102a90 2024-04-24T08:20:42.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.851+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f3800062f30 con 0x7f37e40a3cc0 2024-04-24T08:20:42.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.855+0000 7f37fd600700 1 -- 172.21.15.146:0/1447316131 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f3800062f30 con 0x7f37e40a3cc0 2024-04-24T08:20:42.856 INFO:teuthology.orchestra.run.smithi146.stdout:HOST ADDR LABELS STATUS 2024-04-24T08:20:42.856 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146 172.21.15.146 2024-04-24T08:20:42.856 INFO:teuthology.orchestra.run.smithi146.stdout:smithi153 172.21.15.153 2024-04-24T08:20:42.856 INFO:teuthology.orchestra.run.smithi146.stdout:smithi177 172.21.15.177 2024-04-24T08:20:42.856 INFO:teuthology.orchestra.run.smithi146.stdout:3 hosts in cluster 2024-04-24T08:20:42.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.858+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37e40a3cc0 msgr2=0x7f37e40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.858+0000 7f3805600700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37e40a3cc0 0x7f37e40a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009c60 tx=0x7f37e8009260 comp rx=0 tx=0).stop 2024-04-24T08:20:42.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.858+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800102a90 msgr2=0x7f3800076890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.858+0000 7f3805600700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800102a90 0x7f3800076890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37f00098e0 tx=0x7f37f0009580 comp rx=0 tx=0).stop 2024-04-24T08:20:42.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 shutdown_connections 2024-04-24T08:20:42.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3800104630 0x7f3800075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37e40a3cc0 0x7f37e40a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3800102a90 0x7f3800076890 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 --2- 172.21.15.146:0/1447316131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3800103c90 0x7f3800074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 >> 172.21.15.146:0/1447316131 conn(0x7f38000fe030 msgr2=0x7f3800107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:42.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 shutdown_connections 2024-04-24T08:20:42.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:42.859+0000 7f3805600700 1 -- 172.21.15.146:0/1447316131 wait complete. 2024-04-24T08:20:43.651 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-24T08:20:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:43 smithi177 ceph-mon[33046]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:43.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:43 smithi153 ceph-mon[32746]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:43.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:43 smithi146 ceph-mon[27805]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:44.108 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:44 smithi177 ceph-mon[33046]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:44.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:44 smithi153 ceph-mon[32746]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:44.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:44 smithi146 ceph-mon[27805]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:45.668 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:45 smithi146 ceph-mon[27805]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:45 smithi177 ceph-mon[33046]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:45.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:45 smithi153 ceph-mon[32746]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.189+0000 7fdca3800700 1 -- 172.21.15.146:0/1049421391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c104d90 msgr2=0x7fdc9c1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.189+0000 7fdca3800700 1 --2- 172.21.15.146:0/1049421391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c104d90 0x7fdc9c1051b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdc900099c0 tx=0x7fdc90009cd0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.190+0000 7fdca3800700 1 -- 172.21.15.146:0/1049421391 shutdown_connections 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.190+0000 7fdca3800700 1 --2- 172.21.15.146:0/1049421391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc9c106930 0x7fdc9c10d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.190+0000 7fdca3800700 1 --2- 172.21.15.146:0/1049421391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c104d90 0x7fdc9c1051b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.190+0000 7fdca3800700 1 --2- 172.21.15.146:0/1049421391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9c105f90 0x7fdc9c1063f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.190+0000 7fdca3800700 1 -- 172.21.15.146:0/1049421391 >> 172.21.15.146:0/1049421391 conn(0x7fdc9c100350 msgr2=0x7fdc9c102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.190+0000 7fdca3800700 1 -- 172.21.15.146:0/1049421391 shutdown_connections 2024-04-24T08:20:46.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.190+0000 7fdca3800700 1 -- 172.21.15.146:0/1049421391 wait complete. 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 Processor -- start 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 -- start start 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9c104d90 0x7fdc9c19afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c105f90 0x7fdc9c19b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc9c106930 0x7fdc9c1a2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c110970 con 0x7fdc9c105f90 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c1107f0 con 0x7fdc9c104d90 2024-04-24T08:20:46.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.191+0000 7fdca3800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c110af0 con 0x7fdc9c106930 2024-04-24T08:20:46.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdc9be00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c105f90 0x7fdc9c19b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:46.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca0e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9c104d90 0x7fdc9c19afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:46.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdc9be00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c105f90 0x7fdc9c19b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45202/0 (socket says 172.21.15.146:45202) 2024-04-24T08:20:46.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdc9be00700 1 -- 172.21.15.146:0/1522268962 learned_addr learned my addr 172.21.15.146:0/1522268962 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:46.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca1800700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc9c106930 0x7fdc9c1a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca0e00700 1 -- 172.21.15.146:0/1522268962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc9c106930 msgr2=0x7fdc9c1a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca0e00700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc9c106930 0x7fdc9c1a2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca0e00700 1 -- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c105f90 msgr2=0x7fdc9c19b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca0e00700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c105f90 0x7fdc9c19b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca0e00700 1 -- 172.21.15.146:0/1522268962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9c1a2c00 con 0x7fdc9c104d90 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.192+0000 7fdca0e00700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9c104d90 0x7fdc9c19afa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000b580 tx=0x7fdc9000bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.193+0000 7fdc99600700 1 -- 172.21.15.146:0/1522268962 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc90009e60 con 0x7fdc9c104d90 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.193+0000 7fdc99600700 1 -- 172.21.15.146:0/1522268962 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc90003e30 con 0x7fdc9c104d90 2024-04-24T08:20:46.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.193+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9c1a2e90 con 0x7fdc9c104d90 2024-04-24T08:20:46.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.193+0000 7fdc99600700 1 -- 172.21.15.146:0/1522268962 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9000fdf0 con 0x7fdc9c104d90 2024-04-24T08:20:46.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.193+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc9c1a3280 con 0x7fdc9c104d90 2024-04-24T08:20:46.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.194+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc9c0697e0 con 0x7fdc9c104d90 2024-04-24T08:20:46.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.196+0000 7fdc99600700 1 -- 172.21.15.146:0/1522268962 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fdc9000f580 con 0x7fdc9c104d90 2024-04-24T08:20:46.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.198+0000 7fdc99600700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc840a3cc0 0x7fdc840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:46.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.198+0000 7fdc99600700 1 -- 172.21.15.146:0/1522268962 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdc900d10b0 con 0x7fdc9c104d90 2024-04-24T08:20:46.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.198+0000 7fdc9be00700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc840a3cc0 0x7fdc840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:46.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.199+0000 7fdc9be00700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc840a3cc0 0x7fdc840a6180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009680 tx=0x7fdc88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:46.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.200+0000 7fdc99600700 1 -- 172.21.15.146:0/1522268962 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc90013070 con 0x7fdc9c104d90 2024-04-24T08:20:46.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.431+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 --> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fdc9c062f30 con 0x7fdc840a3cc0 2024-04-24T08:20:46.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.436+0000 7fdc99600700 1 -- 172.21.15.146:0/1522268962 <== mgr.14152 v2:172.21.15.146:6800/1475219332 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fdc9c062f30 con 0x7fdc840a3cc0 2024-04-24T08:20:46.437 INFO:teuthology.orchestra.run.smithi146.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-24T08:20:46.437 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GB400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-24T08:20:46.437 INFO:teuthology.orchestra.run.smithi146.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-24T08:20:46.437 INFO:teuthology.orchestra.run.smithi146.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-24T08:20:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc840a3cc0 msgr2=0x7fdc840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc840a3cc0 0x7fdc840a6180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009680 tx=0x7fdc88009260 comp rx=0 tx=0).stop 2024-04-24T08:20:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9c104d90 msgr2=0x7fdc9c19afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:46.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9c104d90 0x7fdc9c19afa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000b580 tx=0x7fdc9000bee0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 shutdown_connections 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc9c106930 0x7fdc9c1a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc9c104d90 0x7fdc9c19afa0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.439+0000 7fdca3800700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdc840a3cc0 0x7fdc840a6180 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.440+0000 7fdca3800700 1 --2- 172.21.15.146:0/1522268962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc9c105f90 0x7fdc9c19b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.440+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 >> 172.21.15.146:0/1522268962 conn(0x7fdc9c100350 msgr2=0x7fdc9c109230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.440+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 shutdown_connections 2024-04-24T08:20:46.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:46.440+0000 7fdca3800700 1 -- 172.21.15.146:0/1522268962 wait complete. 2024-04-24T08:20:47.173 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T08:20:47.185 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:20:47.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-24T08:20:47.632 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:47 smithi177 ceph-mon[33046]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:47.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:47 smithi177 ceph-mon[33046]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:47.935 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:47 smithi146 ceph-mon[27805]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:47.935 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:47 smithi146 ceph-mon[27805]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:47.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:47 smithi153 ceph-mon[32746]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:47.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:47 smithi153 ceph-mon[32746]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:49.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.737+0000 7f7dfe600700 1 -- 172.21.15.146:0/229548813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df810aa30 msgr2=0x7f7df810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:49.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.737+0000 7f7dfe600700 1 --2- 172.21.15.146:0/229548813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df810aa30 0x7f7df810cec0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7df000b0a0 tx=0x7f7df000b3b0 comp rx=0 tx=0).stop 2024-04-24T08:20:49.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.738+0000 7f7dfe600700 1 -- 172.21.15.146:0/229548813 shutdown_connections 2024-04-24T08:20:49.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.738+0000 7f7dfe600700 1 --2- 172.21.15.146:0/229548813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df810aa30 0x7f7df810cec0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:49.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.738+0000 7f7dfe600700 1 --2- 172.21.15.146:0/229548813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df8076ac0 0x7f7df8074f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:49.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.738+0000 7f7dfe600700 1 --2- 172.21.15.146:0/229548813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df80754f0 0x7f7df8075970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:49.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.738+0000 7f7dfe600700 1 -- 172.21.15.146:0/229548813 >> 172.21.15.146:0/229548813 conn(0x7f7df80fe0f0 msgr2=0x7f7df8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:49.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.738+0000 7f7dfe600700 1 -- 172.21.15.146:0/229548813 shutdown_connections 2024-04-24T08:20:49.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.738+0000 7f7dfe600700 1 -- 172.21.15.146:0/229548813 wait complete. 2024-04-24T08:20:49.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 Processor -- start 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 -- start start 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df80754f0 0x7f7df819af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df8076ac0 0x7f7df819b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df810aa30 0x7f7df81a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8110970 con 0x7f7df810aa30 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81107f0 con 0x7f7df80754f0 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7dfe600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8110af0 con 0x7f7df8076ac0 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.739+0000 7f7df7e00700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df810aa30 0x7f7df81a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7defe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df8076ac0 0x7f7df819b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7defe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df8076ac0 0x7f7df819b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:49760/0 (socket says 172.21.15.146:49760) 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7df7400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df80754f0 0x7f7df819af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7defe00700 1 -- 172.21.15.146:0/2487532316 learned_addr learned my addr 172.21.15.146:0/2487532316 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:49.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7df7e00700 1 -- 172.21.15.146:0/2487532316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df8076ac0 msgr2=0x7f7df819b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7df7e00700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df8076ac0 0x7f7df819b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7df7e00700 1 -- 172.21.15.146:0/2487532316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df80754f0 msgr2=0x7f7df819af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7df7e00700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df80754f0 0x7f7df819af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7df7e00700 1 -- 172.21.15.146:0/2487532316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df81a2b80 con 0x7f7df810aa30 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.740+0000 7f7df7e00700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df810aa30 0x7f7df81a24e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7df0007fa0 tx=0x7f7df0004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.741+0000 7f7df4c00700 1 -- 172.21.15.146:0/2487532316 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0021070 con 0x7f7df810aa30 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.741+0000 7f7df4c00700 1 -- 172.21.15.146:0/2487532316 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df0004e50 con 0x7f7df810aa30 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.741+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df81a2e10 con 0x7f7df810aa30 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.741+0000 7f7df4c00700 1 -- 172.21.15.146:0/2487532316 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0004780 con 0x7f7df810aa30 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.741+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df81a3210 con 0x7f7df810aa30 2024-04-24T08:20:49.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.742+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df80697e0 con 0x7f7df810aa30 2024-04-24T08:20:49.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.744+0000 7f7df4c00700 1 -- 172.21.15.146:0/2487532316 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f7df0003a00 con 0x7f7df810aa30 2024-04-24T08:20:49.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.745+0000 7f7df4c00700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7ddc0a3c40 0x7f7ddc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:49.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.745+0000 7f7df4c00700 1 -- 172.21.15.146:0/2487532316 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f7df00d1280 con 0x7f7df810aa30 2024-04-24T08:20:49.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.745+0000 7f7df7400700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7ddc0a3c40 0x7f7ddc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:49.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.746+0000 7f7df7400700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7ddc0a3c40 0x7f7ddc0a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009c60 tx=0x7f7de8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:49.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:49.750+0000 7f7df4c00700 1 -- 172.21.15.146:0/2487532316 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7df009dcb0 con 0x7f7df810aa30 2024-04-24T08:20:49.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:49 smithi153 ceph-mon[32746]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:49.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:49 smithi146 ceph-mon[27805]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.028+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f7df8063700 con 0x7f7df810aa30 2024-04-24T08:20:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.029+0000 7f7df4c00700 1 -- 172.21.15.146:0/2487532316 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3064 (secure 0 0 0) 0x7f7df009d5b0 con 0x7f7df810aa30 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:e23 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:legacy client fscid: 1 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:Filesystem 'cephfs' (1) 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:fs_name cephfs 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:epoch 23 2024-04-24T08:20:50.031 INFO:teuthology.orchestra.run.smithi146.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:created 2024-04-24T08:18:46.894792+0000 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:modified 2024-04-24T08:18:52.518710+0000 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:tableserver 0 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:root 0 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:session_timeout 300 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:session_autoclose 300 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:max_file_size 1099511627776 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:max_xattr_size 65536 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:required_client_features {} 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:last_failure 0 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:last_failure_osd_epoch 0 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:max_mds 3 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:in 0,1,2 2024-04-24T08:20:50.032 INFO:teuthology.orchestra.run.smithi146.stdout:up {0=24569,1=24568,2=24553} 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:failed 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:damaged 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:stopped 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:data_pools [3] 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:metadata_pool 2 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:inline_data disabled 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:balancer 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:bal_rank_mask -1 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:standby_count_wanted 0 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:qdb_cluster leader: 24569 members: 24553,24568,24569 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b{0:24569} state up:active seq 3 addr [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.i{0:24538} state up:standby-replay seq 1 addr [v2:172.21.15.177:6838/1533073551,v1:172.21.15.177:6839/1533073551] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a{1:24568} state up:active seq 4 addr [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.h{1:24518} state up:standby-replay seq 1 addr [v2:172.21.15.153:6836/565470680,v1:172.21.15.153:6837/565470680] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.033 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.g{2:24553} state up:active seq 6 addr [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c{2:24517} state up:standby-replay seq 1 addr [v2:172.21.15.177:6836/2495345664,v1:172.21.15.177:6837/2495345664] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:Standby daemons: 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.l{-1:24442} state up:standby seq 1 addr [v2:172.21.15.177:6832/3628851028,v1:172.21.15.177:6833/3628851028] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.k{-1:24446} state up:standby seq 1 addr [v2:172.21.15.153:6832/3377566039,v1:172.21.15.153:6833/3377566039] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.j{-1:24464} state up:standby seq 1 addr [v2:172.21.15.146:6834/2947430860,v1:172.21.15.146:6835/2947430860] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d{-1:24481} state up:standby seq 1 addr [v2:172.21.15.146:6836/1079190046,v1:172.21.15.146:6837/1079190046] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.e{-1:24488} state up:standby seq 1 addr [v2:172.21.15.153:6834/3655889612,v1:172.21.15.153:6835/3655889612] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.034 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.f{-1:24505} state up:standby seq 1 addr [v2:172.21.15.177:6834/2429639426,v1:172.21.15.177:6835/2429639426] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:20:50.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7ddc0a3c40 msgr2=0x7f7ddc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7ddc0a3c40 0x7f7ddc0a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009c60 tx=0x7f7de8009260 comp rx=0 tx=0).stop 2024-04-24T08:20:50.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df810aa30 msgr2=0x7f7df81a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df810aa30 0x7f7df81a24e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7df0007fa0 tx=0x7f7df0004c60 comp rx=0 tx=0).stop 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 shutdown_connections 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f7ddc0a3c40 0x7f7ddc0a6100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df810aa30 0x7f7df81a24e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df80754f0 0x7f7df819af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 --2- 172.21.15.146:0/2487532316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7df8076ac0 0x7f7df819b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.033+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 >> 172.21.15.146:0/2487532316 conn(0x7f7df80fe0f0 msgr2=0x7f7df8106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.034+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 shutdown_connections 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:50.034+0000 7f7dfe600700 1 -- 172.21.15.146:0/2487532316 wait complete. 2024-04-24T08:20:50.036 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-24T08:20:50.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:49 smithi177 ceph-mon[33046]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:50.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:50 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2487532316' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T08:20:50.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cca14746-0211-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-24T08:20:50.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:50 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2487532316' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T08:20:51.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:50 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2487532316' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T08:20:51.257 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:20:51.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:51 smithi146 ceph-mon[27805]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:51.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:51 smithi153 ceph-mon[32746]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:52.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:51 smithi177 ceph-mon[33046]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 -- 172.21.15.146:0/3800315365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f574010ad30 msgr2=0x7f57401115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 --2- 172.21.15.146:0/3800315365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f574010ad30 0x7f57401115e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f573800b0d0 tx=0x7f573800b3e0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 -- 172.21.15.146:0/3800315365 shutdown_connections 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 --2- 172.21.15.146:0/3800315365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f574010ad30 0x7f57401115e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 --2- 172.21.15.146:0/3800315365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5740109190 0x7f57401095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 --2- 172.21.15.146:0/3800315365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f574010a390 0x7f574010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 -- 172.21.15.146:0/3800315365 >> 172.21.15.146:0/3800315365 conn(0x7f5740078160 msgr2=0x7f574007a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 -- 172.21.15.146:0/3800315365 shutdown_connections 2024-04-24T08:20:53.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.507+0000 7f5748200700 1 -- 172.21.15.146:0/3800315365 wait complete. 2024-04-24T08:20:53.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.508+0000 7f5748200700 1 Processor -- start 2024-04-24T08:20:53.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.508+0000 7f5748200700 1 -- start start 2024-04-24T08:20:53.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5748200700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5740109190 0x7f574019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5748200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574010a390 0x7f574019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5748200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f574010ad30 0x7f57401a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5748200700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5740114d80 con 0x7f5740109190 2024-04-24T08:20:53.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5748200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5740114c00 con 0x7f574010ad30 2024-04-24T08:20:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5748200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5740114f00 con 0x7f574010a390 2024-04-24T08:20:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5745800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5740109190 0x7f574019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5745800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5740109190 0x7f574019f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39136/0 (socket says 172.21.15.146:39136) 2024-04-24T08:20:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5745800700 1 -- 172.21.15.146:0/3177160112 learned_addr learned my addr 172.21.15.146:0/3177160112 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:53.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5746200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f574010ad30 0x7f57401a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.509+0000 7f5744e00700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574010a390 0x7f574019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5745800700 1 -- 172.21.15.146:0/3177160112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574010a390 msgr2=0x7f574019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5745800700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574010a390 0x7f574019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5745800700 1 -- 172.21.15.146:0/3177160112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f574010ad30 msgr2=0x7f57401a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5745800700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f574010ad30 0x7f57401a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5745800700 1 -- 172.21.15.146:0/3177160112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57401a6f10 con 0x7f5740109190 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5746200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f574010ad30 0x7f57401a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5745800700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5740109190 0x7f574019f2b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f573400e980 tx=0x7f573400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f573e000700 1 -- 172.21.15.146:0/3177160112 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734004230 con 0x7f5740109190 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f573e000700 1 -- 172.21.15.146:0/3177160112 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57340043d0 con 0x7f5740109190 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57401a7200 con 0x7f5740109190 2024-04-24T08:20:53.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.510+0000 7f573e000700 1 -- 172.21.15.146:0/3177160112 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734003dc0 con 0x7f5740109190 2024-04-24T08:20:53.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.511+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57401a76f0 con 0x7f5740109190 2024-04-24T08:20:53.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.512+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57400697e0 con 0x7f5740109190 2024-04-24T08:20:53.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.514+0000 7f573e000700 1 -- 172.21.15.146:0/3177160112 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f573400c810 con 0x7f5740109190 2024-04-24T08:20:53.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.515+0000 7f573e000700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f57280a3c40 0x7f57280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.515+0000 7f573e000700 1 -- 172.21.15.146:0/3177160112 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f57340d0740 con 0x7f5740109190 2024-04-24T08:20:53.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.516+0000 7f5744e00700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f57280a3c40 0x7f57280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:53.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.516+0000 7f5744e00700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f57280a3c40 0x7f57280a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f572c007be0 tx=0x7f572c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:53.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.518+0000 7f573e000700 1 -- 172.21.15.146:0/3177160112 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f573409d0e0 con 0x7f5740109190 2024-04-24T08:20:53.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:53 smithi146 ceph-mon[27805]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:53.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.748+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f574002e080 con 0x7f5740109190 2024-04-24T08:20:53.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.750+0000 7f573e000700 1 -- 172.21.15.146:0/3177160112 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5327 (secure 0 0 0) 0x7f573409c9e0 con 0x7f5740109190 2024-04-24T08:20:53.752 INFO:teuthology.orchestra.run.smithi146.stdout:epoch 90 2024-04-24T08:20:53.752 INFO:teuthology.orchestra.run.smithi146.stdout:fsid cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:20:53.752 INFO:teuthology.orchestra.run.smithi146.stdout:created 2024-04-24T08:09:48.695117+0000 2024-04-24T08:20:53.752 INFO:teuthology.orchestra.run.smithi146.stdout:modified 2024-04-24T08:18:57.557751+0000 2024-04-24T08:20:53.753 INFO:teuthology.orchestra.run.smithi146.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-24T08:20:53.753 INFO:teuthology.orchestra.run.smithi146.stdout:crush_version 26 2024-04-24T08:20:53.753 INFO:teuthology.orchestra.run.smithi146.stdout:full_ratio 0.95 2024-04-24T08:20:53.753 INFO:teuthology.orchestra.run.smithi146.stdout:backfillfull_ratio 0.9 2024-04-24T08:20:53.753 INFO:teuthology.orchestra.run.smithi146.stdout:nearfull_ratio 0.85 2024-04-24T08:20:53.753 INFO:teuthology.orchestra.run.smithi146.stdout:require_min_compat_client luminous 2024-04-24T08:20:53.754 INFO:teuthology.orchestra.run.smithi146.stdout:min_compat_client luminous 2024-04-24T08:20:53.754 INFO:teuthology.orchestra.run.smithi146.stdout:require_osd_release squid 2024-04-24T08:20:53.754 INFO:teuthology.orchestra.run.smithi146.stdout:stretch_mode_enabled false 2024-04-24T08:20:53.754 INFO:teuthology.orchestra.run.smithi146.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-24T08:20:53.754 INFO:teuthology.orchestra.run.smithi146.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-24T08:20:53.754 INFO:teuthology.orchestra.run.smithi146.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-24T08:20:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:max_osd 12 2024-04-24T08:20:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6802/3767707346,v1:172.21.15.146:6803/3767707346] [v2:172.21.15.146:6804/3767707346,v1:172.21.15.146:6805/3767707346] exists,up 1eee291d-ca8f-49ff-8a55-6e95cfb18c9d 2024-04-24T08:20:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/1792775739,v1:172.21.15.153:6801/1792775739] [v2:172.21.15.153:6802/1792775739,v1:172.21.15.153:6803/1792775739] exists,up 8e5912a1-9d40-4f2b-9651-551bc2797a8d 2024-04-24T08:20:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/96699540,v1:172.21.15.177:6801/96699540] [v2:172.21.15.177:6802/96699540,v1:172.21.15.177:6803/96699540] exists,up a25edc87-3db2-4652-82ee-c406cbe5dd64 2024-04-24T08:20:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:osd.3 up in weight 1 up_from 26 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6810/602409571,v1:172.21.15.146:6811/602409571] [v2:172.21.15.146:6812/602409571,v1:172.21.15.146:6813/602409571] exists,up c0de0716-915a-4554-bb92-b04edb82af4b 2024-04-24T08:20:53.755 INFO:teuthology.orchestra.run.smithi146.stdout:osd.4 up in weight 1 up_from 34 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/3654502994,v1:172.21.15.153:6809/3654502994] [v2:172.21.15.153:6810/3654502994,v1:172.21.15.153:6811/3654502994] exists,up 3f96f1d5-f2a7-4b27-8f7e-1d440866acd3 2024-04-24T08:20:53.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.5 up in weight 1 up_from 40 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/3773516414,v1:172.21.15.177:6809/3773516414] [v2:172.21.15.177:6810/3773516414,v1:172.21.15.177:6811/3773516414] exists,up ad461bd7-3a6c-47cc-aa4c-39793fc77bc6 2024-04-24T08:20:53.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.6 up in weight 1 up_from 48 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6818/3036960082,v1:172.21.15.146:6819/3036960082] [v2:172.21.15.146:6820/3036960082,v1:172.21.15.146:6821/3036960082] exists,up c3257f43-689c-4fa9-b182-d9783ef1aa31 2024-04-24T08:20:53.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.7 up in weight 1 up_from 53 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/2204927697,v1:172.21.15.153:6817/2204927697] [v2:172.21.15.153:6818/2204927697,v1:172.21.15.153:6819/2204927697] exists,up 7b062222-8a11-47ad-9389-ad3a5b1ed19d 2024-04-24T08:20:53.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.8 up in weight 1 up_from 62 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/4270133379,v1:172.21.15.177:6817/4270133379] [v2:172.21.15.177:6818/4270133379,v1:172.21.15.177:6819/4270133379] exists,up 30f8bd53-eba7-4942-8c3d-1f327abe42fa 2024-04-24T08:20:53.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.9 up in weight 1 up_from 70 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.146:6826/2032154724,v1:172.21.15.146:6827/2032154724] [v2:172.21.15.146:6828/2032154724,v1:172.21.15.146:6829/2032154724] exists,up 18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f 2024-04-24T08:20:53.756 INFO:teuthology.orchestra.run.smithi146.stdout:osd.10 up in weight 1 up_from 75 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/679103416,v1:172.21.15.153:6825/679103416] [v2:172.21.15.153:6826/679103416,v1:172.21.15.153:6827/679103416] exists,up e7cb8a28-80cc-473d-91d1-dda23d2790af 2024-04-24T08:20:53.757 INFO:teuthology.orchestra.run.smithi146.stdout:osd.11 up in weight 1 up_from 83 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/2574161099,v1:172.21.15.177:6825/2574161099] [v2:172.21.15.177:6826/2574161099,v1:172.21.15.177:6827/2574161099] exists,up 3a48d77c-4350-46bf-8fdf-49827d3957c4 2024-04-24T08:20:53.757 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 2.4 [2,11] 2024-04-24T08:20:53.757 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 2.6 [8,11] 2024-04-24T08:20:53.757 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 2.1d [2,4] 2024-04-24T08:20:53.757 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 3.c [3,9] 2024-04-24T08:20:53.757 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 3.e [5,9] 2024-04-24T08:20:53.758 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 3.19 [3,7] 2024-04-24T08:20:53.758 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 3.1b [4,6] 2024-04-24T08:20:53.758 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 3.23 [4,8] 2024-04-24T08:20:53.758 INFO:teuthology.orchestra.run.smithi146.stdout:pg_upmap_items 3.2c [4,9] 2024-04-24T08:20:53.758 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:6801/888738583 expires 2024-04-25T08:10:56.306756+0000 2024-04-24T08:20:53.758 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:0/3118900567 expires 2024-04-25T08:10:56.306756+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:6800/520435616 expires 2024-04-25T08:10:22.202345+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:0/1190010075 expires 2024-04-25T08:10:56.306756+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:6800/888738583 expires 2024-04-25T08:10:56.306756+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:0/3978832813 expires 2024-04-25T08:10:56.306756+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:0/2983645773 expires 2024-04-25T08:10:22.202345+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:0/2776821726 expires 2024-04-25T08:10:22.202345+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:6801/520435616 expires 2024-04-25T08:10:22.202345+0000 2024-04-24T08:20:53.759 INFO:teuthology.orchestra.run.smithi146.stdout:blocklist 172.21.15.146:0/677277692 expires 2024-04-25T08:10:22.202345+0000 2024-04-24T08:20:53.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f57280a3c40 msgr2=0x7f57280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f57280a3c40 0x7f57280a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f572c007be0 tx=0x7f572c0073d0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5740109190 msgr2=0x7f574019f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5740109190 0x7f574019f2b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f573400e980 tx=0x7f573400ec90 comp rx=0 tx=0).stop 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 shutdown_connections 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f574010ad30 0x7f57401a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f57280a3c40 0x7f57280a6100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5740109190 0x7f574019f2b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 --2- 172.21.15.146:0/3177160112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574010a390 0x7f574019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 >> 172.21.15.146:0/3177160112 conn(0x7f5740078160 msgr2=0x7f574010d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 shutdown_connections 2024-04-24T08:20:53.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:53.754+0000 7f5748200700 1 -- 172.21.15.146:0/3177160112 wait complete. 2024-04-24T08:20:53.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:53 smithi153 ceph-mon[32746]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:54.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:53 smithi177 ceph-mon[33046]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:20:54.473 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-24T08:20:54.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:54.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 -- 172.21.15.146:0/347020691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724104260 msgr2=0x7f3724104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:54.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 --2- 172.21.15.146:0/347020691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724104260 0x7f3724104680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3714009900 tx=0x7f371402a400 comp rx=0 tx=0).stop 2024-04-24T08:20:54.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 -- 172.21.15.146:0/347020691 shutdown_connections 2024-04-24T08:20:54.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 --2- 172.21.15.146:0/347020691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3724105e10 0x7f372410c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 --2- 172.21.15.146:0/347020691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 0x7f37241058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 --2- 172.21.15.146:0/347020691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724104260 0x7f3724104680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 -- 172.21.15.146:0/347020691 >> 172.21.15.146:0/347020691 conn(0x7f37240ff7f0 msgr2=0x7f3724101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:54.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 -- 172.21.15.146:0/347020691 shutdown_connections 2024-04-24T08:20:54.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.702+0000 7f372b600640 1 -- 172.21.15.146:0/347020691 wait complete. 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.703+0000 7f372b600640 1 Processor -- start 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.703+0000 7f372b600640 1 -- start start 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f372b600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3724104260 0x7f372419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f372b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 0x7f372419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f372b600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724105e10 0x7f37241a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f372b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f372410ff40 con 0x7f3724105450 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f372b600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f372410fdc0 con 0x7f3724105e10 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f372b600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f37241100c0 con 0x7f3724104260 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 0x7f372419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:54.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 0x7f372419aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39154/0 (socket says 172.21.15.146:39154) 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 -- 172.21.15.146:0/3168278944 learned_addr learned my addr 172.21.15.146:0/3168278944 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f3728c00640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3724104260 0x7f372419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f3729600640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724105e10 0x7f37241a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 -- 172.21.15.146:0/3168278944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3724104260 msgr2=0x7f372419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3724104260 0x7f372419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 -- 172.21.15.146:0/3168278944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724105e10 msgr2=0x7f37241a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724105e10 0x7f37241a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f371be00640 1 -- 172.21.15.146:0/3168278944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37241a2190 con 0x7f3724105450 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.704+0000 7f3728c00640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3724104260 0x7f372419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.705+0000 7f371be00640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 0x7f372419aab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e9f0 tx=0x7f370c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.705+0000 7f3722000640 1 -- 172.21.15.146:0/3168278944 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c00ccc0 con 0x7f3724105450 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.705+0000 7f3722000640 1 -- 172.21.15.146:0/3168278944 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f370c010910 con 0x7f3724105450 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.705+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37241a2480 con 0x7f3724105450 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.705+0000 7f3722000640 1 -- 172.21.15.146:0/3168278944 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c0188a0 con 0x7f3724105450 2024-04-24T08:20:54.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.705+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37241a29d0 con 0x7f3724105450 2024-04-24T08:20:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.706+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f372410b600 con 0x7f3724105450 2024-04-24T08:20:54.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.708+0000 7f3722000640 1 -- 172.21.15.146:0/3168278944 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f370c010ac0 con 0x7f3724105450 2024-04-24T08:20:54.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.710+0000 7f3722000640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37080a3bb0 0x7f37080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:54.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.711+0000 7f3722000640 1 -- 172.21.15.146:0/3168278944 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f370c0d0870 con 0x7f3724105450 2024-04-24T08:20:54.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.711+0000 7f3728c00640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37080a3bb0 0x7f37080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:54.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.711+0000 7f3728c00640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37080a3bb0 0x7f37080a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f371402a9f0 tx=0x7f37140315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:54.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.715+0000 7f3722000640 1 -- 172.21.15.146:0/3168278944 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f370c09d210 con 0x7f3724105450 2024-04-24T08:20:54.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:54 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3177160112' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T08:20:54.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.919+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3724062c00 con 0x7f3724105450 2024-04-24T08:20:54.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.921+0000 7f3722000640 1 -- 172.21.15.146:0/3168278944 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7f370c09cb10 con 0x7f3724105450 2024-04-24T08:20:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:54.923 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:54.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37080a3bb0 msgr2=0x7f37080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:54.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37080a3bb0 0x7f37080a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f371402a9f0 tx=0x7f37140315b0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 msgr2=0x7f372419aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:54.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 0x7f372419aab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e9f0 tx=0x7f370c00edf0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 shutdown_connections 2024-04-24T08:20:54.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f37080a3bb0 0x7f37080a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3724105e10 0x7f37241a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3724105450 0x7f372419aab0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 --2- 172.21.15.146:0/3168278944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3724104260 0x7f372419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:54.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.924+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 >> 172.21.15.146:0/3168278944 conn(0x7f37240ff7f0 msgr2=0x7f37241087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:54.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.925+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 shutdown_connections 2024-04-24T08:20:54.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:54.925+0000 7f372b600640 1 -- 172.21.15.146:0/3168278944 wait complete. 2024-04-24T08:20:54.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:54.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:54.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:54 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3177160112' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T08:20:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:54 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3177160112' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 -- 172.21.15.146:0/2548505452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 msgr2=0x7f488c10c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 --2- 172.21.15.146:0/2548505452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c10c580 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4884009ca0 tx=0x7f488402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 -- 172.21.15.146:0/2548505452 shutdown_connections 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 --2- 172.21.15.146:0/2548505452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c10c580 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 --2- 172.21.15.146:0/2548505452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c105330 0x7f488c1057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 --2- 172.21.15.146:0/2548505452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 0x7f488c104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 -- 172.21.15.146:0/2548505452 >> 172.21.15.146:0/2548505452 conn(0x7f488c0ff6b0 msgr2=0x7f488c101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 -- 172.21.15.146:0/2548505452 shutdown_connections 2024-04-24T08:20:55.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.143+0000 7f488b200640 1 -- 172.21.15.146:0/2548505452 wait complete. 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 Processor -- start 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 -- start start 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 0x7f488c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c105330 0x7f488c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f488c10fdf0 con 0x7f488c105cf0 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f488c10fc70 con 0x7f488c105330 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.144+0000 7f488b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f488c10ff70 con 0x7f488c104140 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f488a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c1a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 0x7f488c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 0x7f488c19a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38636/0 (socket says 172.21.15.146:38636) 2024-04-24T08:20:55.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 -- 172.21.15.146:0/1948718628 learned_addr learned my addr 172.21.15.146:0/1948718628 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f488a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c1a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39166/0 (socket says 172.21.15.146:39166) 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4883e00640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c105330 0x7f488c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 -- 172.21.15.146:0/1948718628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c105330 msgr2=0x7f488c19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c105330 0x7f488c19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 -- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 msgr2=0x7f488c1a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c1a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 -- 172.21.15.146:0/1948718628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c1a2030 con 0x7f488c104140 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f488a800640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:55.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.145+0000 7f4889e00640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 0x7f488c19a410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f487c00e9c0 tx=0x7f487c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:55.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.146+0000 7f4882a00640 1 -- 172.21.15.146:0/1948718628 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c00cd80 con 0x7f488c104140 2024-04-24T08:20:55.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.146+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c1a2320 con 0x7f488c104140 2024-04-24T08:20:55.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.146+0000 7f4882a00640 1 -- 172.21.15.146:0/1948718628 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f487c0035f0 con 0x7f488c104140 2024-04-24T08:20:55.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.146+0000 7f4882a00640 1 -- 172.21.15.146:0/1948718628 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c010690 con 0x7f488c104140 2024-04-24T08:20:55.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.146+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c1a28c0 con 0x7f488c104140 2024-04-24T08:20:55.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.147+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f488c10b400 con 0x7f488c104140 2024-04-24T08:20:55.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.149+0000 7f4882a00640 1 -- 172.21.15.146:0/1948718628 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f487c0037a0 con 0x7f488c104140 2024-04-24T08:20:55.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.151+0000 7f4882a00640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f48700a3bb0 0x7f48700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.151+0000 7f4882a00640 1 -- 172.21.15.146:0/1948718628 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f487c0d0980 con 0x7f488c104140 2024-04-24T08:20:55.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.151+0000 7f4883e00640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f48700a3bb0 0x7f48700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.151+0000 7f4883e00640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f48700a3bb0 0x7f48700a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f488c19b920 tx=0x7f4874005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:55.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.153+0000 7f4882a00640 1 -- 172.21.15.146:0/1948718628 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f487c09d320 con 0x7f488c104140 2024-04-24T08:20:55.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.362+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f488c02d8d0 con 0x7f488c104140 2024-04-24T08:20:55.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.363+0000 7f4882a00640 1 -- 172.21.15.146:0/1948718628 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f487c0870a0 con 0x7f488c104140 2024-04-24T08:20:55.364 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:20:55.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f48700a3bb0 msgr2=0x7f48700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f48700a3bb0 0x7f48700a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f488c19b920 tx=0x7f4874005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:55.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 msgr2=0x7f488c19a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 0x7f488c19a410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f487c00e9c0 tx=0x7f487c00ee80 comp rx=0 tx=0).stop 2024-04-24T08:20:55.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 shutdown_connections 2024-04-24T08:20:55.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f48700a3bb0 0x7f48700a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f488c105cf0 0x7f488c1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f488c105330 0x7f488c19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 --2- 172.21.15.146:0/1948718628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f488c104140 0x7f488c19a410 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.366+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 >> 172.21.15.146:0/1948718628 conn(0x7f488c0ff6b0 msgr2=0x7f488c1085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:55.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.367+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 shutdown_connections 2024-04-24T08:20:55.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.367+0000 7f488b200640 1 -- 172.21.15.146:0/1948718628 wait complete. 2024-04-24T08:20:55.386 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:55.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 -- 172.21.15.146:0/4117746526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb96010a210 msgr2=0x7fb960110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 --2- 172.21.15.146:0/4117746526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb96010a210 0x7fb960110aa0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb958009ca0 tx=0x7fb95802a8b0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 -- 172.21.15.146:0/4117746526 shutdown_connections 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 --2- 172.21.15.146:0/4117746526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb96010a210 0x7fb960110aa0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 --2- 172.21.15.146:0/4117746526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb960109850 0x7fb960109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 --2- 172.21.15.146:0/4117746526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb960108660 0x7fb960108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 -- 172.21.15.146:0/4117746526 >> 172.21.15.146:0/4117746526 conn(0x7fb960077600 msgr2=0x7fb960079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 -- 172.21.15.146:0/4117746526 shutdown_connections 2024-04-24T08:20:55.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.591+0000 7fb968400640 1 -- 172.21.15.146:0/4117746526 wait complete. 2024-04-24T08:20:55.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.592+0000 7fb968400640 1 Processor -- start 2024-04-24T08:20:55.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.592+0000 7fb968400640 1 -- start start 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb968400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb960108660 0x7fb96019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb968400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb960109850 0x7fb96019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb968400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb96010a210 0x7fb9601a5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb968400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb960114310 con 0x7fb960108660 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb968400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb960114190 con 0x7fb96010a210 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb968400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb960114490 con 0x7fb960109850 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb965a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb960108660 0x7fb96019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb966400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb96010a210 0x7fb9601a5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb965a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb960108660 0x7fb96019e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39194/0 (socket says 172.21.15.146:39194) 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb965a00640 1 -- 172.21.15.146:0/2590877847 learned_addr learned my addr 172.21.15.146:0/2590877847 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb965000640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb960109850 0x7fb96019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb965a00640 1 -- 172.21.15.146:0/2590877847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb960109850 msgr2=0x7fb96019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb965a00640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb960109850 0x7fb96019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.593+0000 7fb965a00640 1 -- 172.21.15.146:0/2590877847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb96010a210 msgr2=0x7fb9601a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb965a00640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb96010a210 0x7fb9601a5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb965a00640 1 -- 172.21.15.146:0/2590877847 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9601a64b0 con 0x7fb960108660 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb966400640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb96010a210 0x7fb9601a5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb965a00640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb960108660 0x7fb96019e890 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb95400c970 tx=0x7fb95400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb95e000640 1 -- 172.21.15.146:0/2590877847 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954007c10 con 0x7fb960108660 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9601a67a0 con 0x7fb960108660 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb95e000640 1 -- 172.21.15.146:0/2590877847 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb954007db0 con 0x7fb960108660 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb95e000640 1 -- 172.21.15.146:0/2590877847 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95400f8c0 con 0x7fb960108660 2024-04-24T08:20:55.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.594+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9601a6cf0 con 0x7fb960108660 2024-04-24T08:20:55.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.595+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb96010f8b0 con 0x7fb960108660 2024-04-24T08:20:55.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.598+0000 7fb95e000640 1 -- 172.21.15.146:0/2590877847 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fb954018480 con 0x7fb960108660 2024-04-24T08:20:55.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.599+0000 7fb95e000640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9480a3bb0 0x7fb9480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.599+0000 7fb95e000640 1 -- 172.21.15.146:0/2590877847 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fb9540d06a0 con 0x7fb960108660 2024-04-24T08:20:55.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.599+0000 7fb965000640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9480a3bb0 0x7fb9480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.600+0000 7fb965000640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9480a3bb0 0x7fb9480a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb96019fda0 tx=0x7fb95000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:55.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.603+0000 7fb95e000640 1 -- 172.21.15.146:0/2590877847 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb95409d0d0 con 0x7fb960108660 2024-04-24T08:20:55.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:55 smithi146 ceph-mon[27805]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:55.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:55 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3168278944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:55.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:55 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1948718628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:55.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.807+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb960062c30 con 0x7fb960108660 2024-04-24T08:20:55.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.808+0000 7fb95e000640 1 -- 172.21.15.146:0/2590877847 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fb95409c9d0 con 0x7fb960108660 2024-04-24T08:20:55.808 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:20:55.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.810+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9480a3bb0 msgr2=0x7fb9480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.810+0000 7fb968400640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9480a3bb0 0x7fb9480a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb96019fda0 tx=0x7fb95000b390 comp rx=0 tx=0).stop 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.810+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb960108660 msgr2=0x7fb96019e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.810+0000 7fb968400640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb960108660 0x7fb96019e890 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb95400c970 tx=0x7fb95400ce30 comp rx=0 tx=0).stop 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 shutdown_connections 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb960109850 0x7fb96019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fb9480a3bb0 0x7fb9480a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb96010a210 0x7fb9601a5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 --2- 172.21.15.146:0/2590877847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb960108660 0x7fb96019e890 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 >> 172.21.15.146:0/2590877847 conn(0x7fb960077600 msgr2=0x7fb96010ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 shutdown_connections 2024-04-24T08:20:55.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:55.811+0000 7fb968400640 1 -- 172.21.15.146:0/2590877847 wait complete. 2024-04-24T08:20:55.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:55 smithi153 ceph-mon[32746]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:55 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3168278944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:55.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:55 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1948718628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:56.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 -- 172.21.15.146:0/1742128882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c104280 msgr2=0x7f820c1046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 --2- 172.21.15.146:0/1742128882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c104280 0x7f820c1046a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009ca0 tx=0x7f81fc02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 -- 172.21.15.146:0/1742128882 shutdown_connections 2024-04-24T08:20:56.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 --2- 172.21.15.146:0/1742128882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c105470 0x7f820c1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 --2- 172.21.15.146:0/1742128882 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f820c105e30 0x7f820c10c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 --2- 172.21.15.146:0/1742128882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c104280 0x7f820c1046a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 -- 172.21.15.146:0/1742128882 >> 172.21.15.146:0/1742128882 conn(0x7f820c0ff7f0 msgr2=0x7f820c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 -- 172.21.15.146:0/1742128882 shutdown_connections 2024-04-24T08:20:56.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.032+0000 7f8213c00640 1 -- 172.21.15.146:0/1742128882 wait complete. 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.033+0000 7f8213c00640 1 Processor -- start 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8213c00640 1 -- start start 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8213c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c104280 0x7f820c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8213c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c105470 0x7f820c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8213c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f820c105e30 0x7f820c1a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8213c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10ff60 con 0x7f820c105470 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8213c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10fde0 con 0x7f820c105e30 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8213c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f820c1100e0 con 0x7f820c104280 2024-04-24T08:20:56.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f820be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c105470 0x7f820c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8211200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c104280 0x7f820c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8211200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c104280 0x7f820c19a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38676/0 (socket says 172.21.15.146:38676) 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8211200640 1 -- 172.21.15.146:0/284264589 learned_addr learned my addr 172.21.15.146:0/284264589 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f820be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c105470 0x7f820c19ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39214/0 (socket says 172.21.15.146:39214) 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.034+0000 7f8211c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f820c105e30 0x7f820c1a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8211200640 1 -- 172.21.15.146:0/284264589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f820c105e30 msgr2=0x7f820c1a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8211200640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f820c105e30 0x7f820c1a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8211200640 1 -- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c105470 msgr2=0x7f820c19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8211200640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c105470 0x7f820c19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8211200640 1 -- 172.21.15.146:0/284264589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c1a2210 con 0x7f820c104280 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8211c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f820c105e30 0x7f820c1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8211200640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c104280 0x7f820c19a5f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f81fc000f80 tx=0x7f81fc0319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8209600640 1 -- 172.21.15.146:0/284264589 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc003b70 con 0x7f820c104280 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c1a24a0 con 0x7f820c104280 2024-04-24T08:20:56.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8209600640 1 -- 172.21.15.146:0/284264589 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81fc003d10 con 0x7f820c104280 2024-04-24T08:20:56.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8209600640 1 -- 172.21.15.146:0/284264589 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc02fc50 con 0x7f820c104280 2024-04-24T08:20:56.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.035+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c1a2a40 con 0x7f820c104280 2024-04-24T08:20:56.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.037+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f820c10b5a0 con 0x7f820c104280 2024-04-24T08:20:56.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.039+0000 7f8209600640 1 -- 172.21.15.146:0/284264589 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f81fc004040 con 0x7f820c104280 2024-04-24T08:20:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.040+0000 7f8209600640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f81f00a3c30 0x7f81f00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.040+0000 7f8209600640 1 -- 172.21.15.146:0/284264589 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f81fc0f17b0 con 0x7f820c104280 2024-04-24T08:20:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.040+0000 7f820be00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f81f00a3c30 0x7f81f00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.041+0000 7f820be00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f81f00a3c30 0x7f81f00a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f820c19bb00 tx=0x7f81f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.044+0000 7f8209600640 1 -- 172.21.15.146:0/284264589 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81fc0be150 con 0x7f820c104280 2024-04-24T08:20:56.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:55 smithi177 ceph-mon[33046]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:56.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:55 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3168278944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:56.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:55 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1948718628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:56.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.248+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f820c062c00 con 0x7f820c104280 2024-04-24T08:20:56.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.248+0000 7f8209600640 1 -- 172.21.15.146:0/284264589 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f81fc0bda50 con 0x7f820c104280 2024-04-24T08:20:56.249 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f81f00a3c30 msgr2=0x7f81f00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f81f00a3c30 0x7f81f00a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f820c19bb00 tx=0x7f81f4005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c104280 msgr2=0x7f820c19a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c104280 0x7f820c19a5f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f81fc000f80 tx=0x7f81fc0319f0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 shutdown_connections 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f81f00a3c30 0x7f81f00a60f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f820c105e30 0x7f820c1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f820c105470 0x7f820c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 --2- 172.21.15.146:0/284264589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c104280 0x7f820c19a5f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 >> 172.21.15.146:0/284264589 conn(0x7f820c0ff7f0 msgr2=0x7f820c108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 shutdown_connections 2024-04-24T08:20:56.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.251+0000 7f8213c00640 1 -- 172.21.15.146:0/284264589 wait complete. 2024-04-24T08:20:56.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:56.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.474+0000 7ff11b200640 1 -- 172.21.15.146:0/1234608380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c107b60 msgr2=0x7ff11c109fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.474+0000 7ff11b200640 1 --2- 172.21.15.146:0/1234608380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c107b60 0x7ff11c109fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff114009ca0 tx=0x7ff11402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.474+0000 7ff11b200640 1 -- 172.21.15.146:0/1234608380 shutdown_connections 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.474+0000 7ff11b200640 1 --2- 172.21.15.146:0/1234608380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c105230 0x7ff11c107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.474+0000 7ff11b200640 1 --2- 172.21.15.146:0/1234608380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c107b60 0x7ff11c109fb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.474+0000 7ff11b200640 1 --2- 172.21.15.146:0/1234608380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c102900 0x7ff11c104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.474+0000 7ff11b200640 1 -- 172.21.15.146:0/1234608380 >> 172.21.15.146:0/1234608380 conn(0x7ff11c0fc2d0 msgr2=0x7ff11c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.475+0000 7ff11b200640 1 -- 172.21.15.146:0/1234608380 shutdown_connections 2024-04-24T08:20:56.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.475+0000 7ff11b200640 1 -- 172.21.15.146:0/1234608380 wait complete. 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 Processor -- start 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 -- start start 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c102900 0x7ff11c198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c105230 0x7ff11c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c107b60 0x7ff11c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10db20 con 0x7ff11c107b60 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10d9a0 con 0x7ff11c105230 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10dca0 con 0x7ff11c102900 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.476+0000 7ff11a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c107b60 0x7ff11c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c107b60 0x7ff11c19f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39230/0 (socket says 172.21.15.146:39230) 2024-04-24T08:20:56.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 -- 172.21.15.146:0/2621983234 learned_addr learned my addr 172.21.15.146:0/2621983234 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff119400640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c105230 0x7ff11c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff119e00640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c102900 0x7ff11c198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 -- 172.21.15.146:0/2621983234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c102900 msgr2=0x7ff11c198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c102900 0x7ff11c198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 -- 172.21.15.146:0/2621983234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c105230 msgr2=0x7ff11c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c105230 0x7ff11c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 -- 172.21.15.146:0/2621983234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c19fd60 con 0x7ff11c107b60 2024-04-24T08:20:56.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.477+0000 7ff11a800640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c107b60 0x7ff11c19f6f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff114031ae0 tx=0x7ff114031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.478+0000 7ff112a00640 1 -- 172.21.15.146:0/2621983234 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114003b70 con 0x7ff11c107b60 2024-04-24T08:20:56.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.478+0000 7ff112a00640 1 -- 172.21.15.146:0/2621983234 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff114003d10 con 0x7ff11c107b60 2024-04-24T08:20:56.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.478+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c19fff0 con 0x7ff11c107b60 2024-04-24T08:20:56.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.478+0000 7ff112a00640 1 -- 172.21.15.146:0/2621983234 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11402fc40 con 0x7ff11c107b60 2024-04-24T08:20:56.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.478+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1a04b0 con 0x7ff11c107b60 2024-04-24T08:20:56.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.479+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c0fddc0 con 0x7ff11c107b60 2024-04-24T08:20:56.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.481+0000 7ff112a00640 1 -- 172.21.15.146:0/2621983234 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7ff114004050 con 0x7ff11c107b60 2024-04-24T08:20:56.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.482+0000 7ff112a00640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff1000a3bb0 0x7ff1000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.483+0000 7ff112a00640 1 -- 172.21.15.146:0/2621983234 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7ff1140f16b0 con 0x7ff11c107b60 2024-04-24T08:20:56.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.483+0000 7ff119e00640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff1000a3bb0 0x7ff1000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.483+0000 7ff119e00640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff1000a3bb0 0x7ff1000a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009bd0 tx=0x7ff10c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.487+0000 7ff112a00640 1 -- 172.21.15.146:0/2621983234 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1140be050 con 0x7ff11c107b60 2024-04-24T08:20:56.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.742+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff11c02d620 con 0x7ff11c107b60 2024-04-24T08:20:56.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.743+0000 7ff112a00640 1 -- 172.21.15.146:0/2621983234 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11891 (secure 0 0 0) 0x7ff1140bd950 con 0x7ff11c107b60 2024-04-24T08:20:56.744 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:56.744 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:18:52.518710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:20:56.745 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-24T08:20:56.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff1000a3bb0 msgr2=0x7ff1000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff1000a3bb0 0x7ff1000a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009bd0 tx=0x7ff10c005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:56.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c107b60 msgr2=0x7ff11c19f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c107b60 0x7ff11c19f6f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff114031ae0 tx=0x7ff114031ee0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 shutdown_connections 2024-04-24T08:20:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7ff1000a3bb0 0x7ff1000a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c107b60 0x7ff11c19f6f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff11c105230 0x7ff11c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 --2- 172.21.15.146:0/2621983234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c102900 0x7ff11c198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.746+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 >> 172.21.15.146:0/2621983234 conn(0x7ff11c0fc2d0 msgr2=0x7ff11c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.747+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 shutdown_connections 2024-04-24T08:20:56.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.747+0000 7ff11b200640 1 -- 172.21.15.146:0/2621983234 wait complete. 2024-04-24T08:20:56.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2590877847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/284264589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.971+0000 7f0dc3200640 1 -- 172.21.15.146:0/4055491119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4101e70 msgr2=0x7f0dc4102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.971+0000 7f0dc3200640 1 --2- 172.21.15.146:0/4055491119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4101e70 0x7f0dc4102290 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0dac009900 tx=0x7f0dac02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.972+0000 7f0dc3200640 1 -- 172.21.15.146:0/4055491119 shutdown_connections 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.972+0000 7f0dc3200640 1 --2- 172.21.15.146:0/4055491119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc4102fc0 0x7f0dc4103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.972+0000 7f0dc3200640 1 --2- 172.21.15.146:0/4055491119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4103980 0x7f0dc410a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.972+0000 7f0dc3200640 1 --2- 172.21.15.146:0/4055491119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4101e70 0x7f0dc4102290 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.972+0000 7f0dc3200640 1 -- 172.21.15.146:0/4055491119 >> 172.21.15.146:0/4055491119 conn(0x7f0dc40fd3e0 msgr2=0x7f0dc40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.972+0000 7f0dc3200640 1 -- 172.21.15.146:0/4055491119 shutdown_connections 2024-04-24T08:20:56.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.972+0000 7f0dc3200640 1 -- 172.21.15.146:0/4055491119 wait complete. 2024-04-24T08:20:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 Processor -- start 2024-04-24T08:20:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 -- start start 2024-04-24T08:20:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc4101e70 0x7f0dc419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4102fc0 0x7f0dc419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4103980 0x7f0dc41a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc410db60 con 0x7f0dc4102fc0 2024-04-24T08:20:56.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc410d9e0 con 0x7f0dc4103980 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.973+0000 7f0dc3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc410dce0 con 0x7f0dc4101e70 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4102fc0 0x7f0dc419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc4101e70 0x7f0dc419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4103980 0x7f0dc41a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4102fc0 0x7f0dc419a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39244/0 (socket says 172.21.15.146:39244) 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc1400640 1 -- 172.21.15.146:0/2058958371 learned_addr learned my addr 172.21.15.146:0/2058958371 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc2800640 1 -- 172.21.15.146:0/2058958371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc4101e70 msgr2=0x7f0dc419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc2800640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc4101e70 0x7f0dc419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc2800640 1 -- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4102fc0 msgr2=0x7f0dc419a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc2800640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4102fc0 0x7f0dc419a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc2800640 1 -- 172.21.15.146:0/2058958371 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc41a2000 con 0x7f0dc4103980 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.974+0000 7f0dc2800640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4103980 0x7f0dc41a1990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009fd0 tx=0x7f0dbc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.975+0000 7f0dbaa00640 1 -- 172.21.15.146:0/2058958371 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc00ce20 con 0x7f0dc4103980 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.975+0000 7f0dbaa00640 1 -- 172.21.15.146:0/2058958371 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dbc005020 con 0x7f0dc4103980 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.975+0000 7f0dbaa00640 1 -- 172.21.15.146:0/2058958371 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc010720 con 0x7f0dc4103980 2024-04-24T08:20:56.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.975+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc41a22f0 con 0x7f0dc4103980 2024-04-24T08:20:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.975+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc41a2840 con 0x7f0dc4103980 2024-04-24T08:20:56.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.976+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc41a2b00 con 0x7f0dc4103980 2024-04-24T08:20:56.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.978+0000 7f0dbaa00640 1 -- 172.21.15.146:0/2058958371 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f0dbc005540 con 0x7f0dc4103980 2024-04-24T08:20:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.979+0000 7f0dbaa00640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0da80a3c30 0x7f0da80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.979+0000 7f0dbaa00640 1 -- 172.21.15.146:0/2058958371 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0dbc0d0ef0 con 0x7f0dc4103980 2024-04-24T08:20:56.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.979+0000 7f0dc1e00640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0da80a3c30 0x7f0da80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.980+0000 7f0dc1e00640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0da80a3c30 0x7f0da80a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0dac031440 tx=0x7f0dac0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:56.982+0000 7f0dbaa00640 1 -- 172.21.15.146:0/2058958371 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dbc09d890 con 0x7f0dc4103980 2024-04-24T08:20:56.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2590877847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:56.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/284264589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:57.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2590877847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:57.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/284264589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:57.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.175+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0dc41a2cd0 con 0x7f0dc4103980 2024-04-24T08:20:57.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.177+0000 7f0dbaa00640 1 -- 172.21.15.146:0/2058958371 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7f0dbc09d190 con 0x7f0dc4103980 2024-04-24T08:20:57.178 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:57.178 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.179+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0da80a3c30 msgr2=0x7f0da80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.179+0000 7f0dc3200640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0da80a3c30 0x7f0da80a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0dac031440 tx=0x7f0dac0315b0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4103980 msgr2=0x7f0dc41a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4103980 0x7f0dc41a1990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009fd0 tx=0x7f0dbc00ef10 comp rx=0 tx=0).stop 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 shutdown_connections 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0da80a3c30 0x7f0da80a60f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dc4103980 0x7f0dc41a1990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dc4102fc0 0x7f0dc419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 --2- 172.21.15.146:0/2058958371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc4101e70 0x7f0dc419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 >> 172.21.15.146:0/2058958371 conn(0x7f0dc40fd3e0 msgr2=0x7f0dc4074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 shutdown_connections 2024-04-24T08:20:57.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.180+0000 7f0dc3200640 1 -- 172.21.15.146:0/2058958371 wait complete. 2024-04-24T08:20:57.200 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:57.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 -- 172.21.15.146:0/1724867942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 msgr2=0x7f14f8108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 --2- 172.21.15.146:0/1724867942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 0x7f14f8108a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009900 tx=0x7f14e802a410 comp rx=0 tx=0).stop 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 -- 172.21.15.146:0/1724867942 shutdown_connections 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 --2- 172.21.15.146:0/1724867942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 0x7f14f8110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 --2- 172.21.15.146:0/1724867942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f8109850 0x7f14f8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 --2- 172.21.15.146:0/1724867942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 0x7f14f8108a80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 -- 172.21.15.146:0/1724867942 >> 172.21.15.146:0/1724867942 conn(0x7f14f8077600 msgr2=0x7f14f8079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 -- 172.21.15.146:0/1724867942 shutdown_connections 2024-04-24T08:20:57.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.396+0000 7f14ff200640 1 -- 172.21.15.146:0/1724867942 wait complete. 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 Processor -- start 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 -- start start 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 0x7f14f819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f8109850 0x7f14f819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 0x7f14f81a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f14f8114300 con 0x7f14f810a210 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14f8114180 con 0x7f14f8109850 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14ff200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14f8114480 con 0x7f14f8108660 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.397+0000 7f14fd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 0x7f14f81a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14fd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 0x7f14f81a5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39250/0 (socket says 172.21.15.146:39250) 2024-04-24T08:20:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14fd200640 1 -- 172.21.15.146:0/3751944577 learned_addr learned my addr 172.21.15.146:0/3751944577 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14f7e00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 0x7f14f819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14efe00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f8109850 0x7f14f819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14f7e00640 1 -- 172.21.15.146:0/3751944577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f8109850 msgr2=0x7f14f819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14f7e00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f8109850 0x7f14f819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14f7e00640 1 -- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 msgr2=0x7f14f81a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14f7e00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 0x7f14f81a5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14f7e00640 1 -- 172.21.15.146:0/3751944577 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f81a65c0 con 0x7f14f8108660 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14fd200640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 0x7f14f81a5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.398+0000 7f14efe00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f8109850 0x7f14f819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.399+0000 7f14f7e00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 0x7f14f819e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009230 tx=0x7f14e8032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.399+0000 7f14f5600640 1 -- 172.21.15.146:0/3751944577 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e803f030 con 0x7f14f8108660 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.399+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f81a6850 con 0x7f14f8108660 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.399+0000 7f14f5600640 1 -- 172.21.15.146:0/3751944577 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14e8003ce0 con 0x7f14f8108660 2024-04-24T08:20:57.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.399+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f81a6d60 con 0x7f14f8108660 2024-04-24T08:20:57.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.399+0000 7f14f5600640 1 -- 172.21.15.146:0/3751944577 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8049060 con 0x7f14f8108660 2024-04-24T08:20:57.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.400+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f810f980 con 0x7f14f8108660 2024-04-24T08:20:57.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.402+0000 7f14f5600640 1 -- 172.21.15.146:0/3751944577 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f14e8042030 con 0x7f14f8108660 2024-04-24T08:20:57.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.403+0000 7f14f5600640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f14dc0a3c30 0x7f14dc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.404+0000 7f14f5600640 1 -- 172.21.15.146:0/3751944577 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f14e80f0410 con 0x7f14f8108660 2024-04-24T08:20:57.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.404+0000 7f14efe00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f14dc0a3c30 0x7f14dc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.404+0000 7f14efe00640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f14dc0a3c30 0x7f14dc0a60f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f14e00095f0 tx=0x7f14e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:57.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.408+0000 7f14f5600640 1 -- 172.21.15.146:0/3751944577 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14e80bc760 con 0x7f14f8108660 2024-04-24T08:20:57.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.611+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f14f8062c00 con 0x7f14f8108660 2024-04-24T08:20:57.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.611+0000 7f14f5600640 1 -- 172.21.15.146:0/3751944577 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f14e80c0a60 con 0x7f14f8108660 2024-04-24T08:20:57.612 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f14dc0a3c30 msgr2=0x7f14dc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f14dc0a3c30 0x7f14dc0a60f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f14e00095f0 tx=0x7f14e0005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 msgr2=0x7f14f819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 0x7f14f819e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009230 tx=0x7f14e8032ce0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 shutdown_connections 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f14dc0a3c30 0x7f14dc0a60f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f810a210 0x7f14f81a5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14f8109850 0x7f14f819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 --2- 172.21.15.146:0/3751944577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14f8108660 0x7f14f819e9f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 >> 172.21.15.146:0/3751944577 conn(0x7f14f8077600 msgr2=0x7f14f810cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:57.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 shutdown_connections 2024-04-24T08:20:57.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.614+0000 7f14ff200640 1 -- 172.21.15.146:0/3751944577 wait complete. 2024-04-24T08:20:57.634 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:57.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 -- 172.21.15.146:0/1560404742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc108540 msgr2=0x7f97dc108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 --2- 172.21.15.146:0/1560404742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc108540 0x7f97dc108960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009900 tx=0x7f97d002a410 comp rx=0 tx=0).stop 2024-04-24T08:20:57.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 -- 172.21.15.146:0/1560404742 shutdown_connections 2024-04-24T08:20:57.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 --2- 172.21.15.146:0/1560404742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97dc10a0f0 0x7f97dc110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 --2- 172.21.15.146:0/1560404742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc109730 0x7f97dc109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 --2- 172.21.15.146:0/1560404742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc108540 0x7f97dc108960 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 -- 172.21.15.146:0/1560404742 >> 172.21.15.146:0/1560404742 conn(0x7f97dc0774d0 msgr2=0x7f97dc079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:57.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 -- 172.21.15.146:0/1560404742 shutdown_connections 2024-04-24T08:20:57.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.834+0000 7f97e2800640 1 -- 172.21.15.146:0/1560404742 wait complete. 2024-04-24T08:20:57.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.835+0000 7f97e2800640 1 Processor -- start 2024-04-24T08:20:57.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.835+0000 7f97e2800640 1 -- start start 2024-04-24T08:20:57.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc108540 0x7f97dc19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc109730 0x7f97dc19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97dc10a0f0 0x7f97dc1a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e2800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc114220 con 0x7f97dc109730 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e2800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1140a0 con 0x7f97dc108540 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e2800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1143a0 con 0x7f97dc10a0f0 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc108540 0x7f97dc19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc108540 0x7f97dc19e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:35886/0 (socket says 172.21.15.146:35886) 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 -- 172.21.15.146:0/1168940726 learned_addr learned my addr 172.21.15.146:0/1168940726 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e0a00640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc109730 0x7f97dc19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1e00640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97dc10a0f0 0x7f97dc1a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 -- 172.21.15.146:0/1168940726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97dc10a0f0 msgr2=0x7f97dc1a5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97dc10a0f0 0x7f97dc1a5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 -- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc109730 msgr2=0x7f97dc19ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:57.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc109730 0x7f97dc19ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.836+0000 7f97e1400640 1 -- 172.21.15.146:0/1168940726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97dc1a6410 con 0x7f97dc108540 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.837+0000 7f97e1400640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc108540 0x7f97dc19e7f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f97d0031440 tx=0x7f97d0032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.837+0000 7f97da000640 1 -- 172.21.15.146:0/1168940726 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d003f030 con 0x7f97dc108540 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.837+0000 7f97da000640 1 -- 172.21.15.146:0/1168940726 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d0003ce0 con 0x7f97dc108540 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.837+0000 7f97da000640 1 -- 172.21.15.146:0/1168940726 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0035070 con 0x7f97dc108540 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.837+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97dc1a66a0 con 0x7f97dc108540 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.837+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97dc1a6b40 con 0x7f97dc108540 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.838+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97dc10f800 con 0x7f97dc108540 2024-04-24T08:20:57.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.840+0000 7f97da000640 1 -- 172.21.15.146:0/1168940726 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f97d0042030 con 0x7f97dc108540 2024-04-24T08:20:57.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.841+0000 7f97da000640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97c40a3c30 0x7f97c40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:57.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.841+0000 7f97da000640 1 -- 172.21.15.146:0/1168940726 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f97d00f0b90 con 0x7f97dc108540 2024-04-24T08:20:57.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.842+0000 7f97e0a00640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97c40a3c30 0x7f97c40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:57.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.842+0000 7f97e0a00640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97c40a3c30 0x7f97c40a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0095f0 tx=0x7f97cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:57.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:57.844+0000 7f97da000640 1 -- 172.21.15.146:0/1168940726 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97d00bd530 con 0x7f97dc108540 2024-04-24T08:20:57.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:57 smithi153 ceph-mon[32746]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:57.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:57 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2621983234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:57.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:57 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2058958371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:57.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:57 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3751944577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:57.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:57 smithi146 ceph-mon[27805]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:57.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:57 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2621983234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:57.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:57 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2058958371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:57.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:57 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3751944577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:58.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.047+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97dc062c00 con 0x7f97dc108540 2024-04-24T08:20:58.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.047+0000 7f97da000640 1 -- 172.21.15.146:0/1168940726 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f97d00bce30 con 0x7f97dc108540 2024-04-24T08:20:58.048 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:20:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97c40a3c30 msgr2=0x7f97c40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97c40a3c30 0x7f97c40a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0095f0 tx=0x7f97cc005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc108540 msgr2=0x7f97dc19e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc108540 0x7f97dc19e7f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f97d0031440 tx=0x7f97d0032ce0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 shutdown_connections 2024-04-24T08:20:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97dc10a0f0 0x7f97dc1a5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f97c40a3c30 0x7f97c40a60f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97dc109730 0x7f97dc19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 --2- 172.21.15.146:0/1168940726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97dc108540 0x7f97dc19e7f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 >> 172.21.15.146:0/1168940726 conn(0x7f97dc0774d0 msgr2=0x7f97dc10c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 shutdown_connections 2024-04-24T08:20:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.050+0000 7f97e2800640 1 -- 172.21.15.146:0/1168940726 wait complete. 2024-04-24T08:20:58.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:57 smithi177 ceph-mon[33046]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:58.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:57 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2621983234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:58.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:57 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2058958371' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:58.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:57 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3751944577' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:20:58.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 -- 172.21.15.146:0/3685715343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 msgr2=0x7f0bb810c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 --2- 172.21.15.146:0/3685715343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 0x7f0bb810c6a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009be0 tx=0x7f0bb002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 -- 172.21.15.146:0/3685715343 shutdown_connections 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 --2- 172.21.15.146:0/3685715343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb8105450 0x7f0bb81058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 --2- 172.21.15.146:0/3685715343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 0x7f0bb810c6a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 --2- 172.21.15.146:0/3685715343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb8104260 0x7f0bb8104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 -- 172.21.15.146:0/3685715343 >> 172.21.15.146:0/3685715343 conn(0x7f0bb80ff7f0 msgr2=0x7f0bb8101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:58.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 -- 172.21.15.146:0/3685715343 shutdown_connections 2024-04-24T08:20:58.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.275+0000 7f0bc0200640 1 -- 172.21.15.146:0/3685715343 wait complete. 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 Processor -- start 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 -- start start 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb8104260 0x7f0bb819a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb8105450 0x7f0bb819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 0x7f0bb81a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb810fe70 con 0x7f0bb8105450 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb810fcf0 con 0x7f0bb8105e10 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.276+0000 7f0bc0200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb810fff0 con 0x7f0bb8104260 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb8105450 0x7f0bb819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbe200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 0x7f0bb81a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb8105450 0x7f0bb819aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39302/0 (socket says 172.21.15.146:39302) 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbce00640 1 -- 172.21.15.146:0/4294701961 learned_addr learned my addr 172.21.15.146:0/4294701961 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:58.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbd800640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb8104260 0x7f0bb819a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbe200640 1 -- 172.21.15.146:0/4294701961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb8104260 msgr2=0x7f0bb819a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbe200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb8104260 0x7f0bb819a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbe200640 1 -- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb8105450 msgr2=0x7f0bb819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbe200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb8105450 0x7f0bb819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbe200640 1 -- 172.21.15.146:0/4294701961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb81a2130 con 0x7f0bb8105e10 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.277+0000 7f0bbe200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 0x7f0bb81a1ac0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0034040 tx=0x7f0bb00305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.278+0000 7f0bb6000640 1 -- 172.21.15.146:0/4294701961 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0041020 con 0x7f0bb8105e10 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.278+0000 7f0bb6000640 1 -- 172.21.15.146:0/4294701961 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bb0037040 con 0x7f0bb8105e10 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.278+0000 7f0bb6000640 1 -- 172.21.15.146:0/4294701961 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0036070 con 0x7f0bb8105e10 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.278+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb81a23c0 con 0x7f0bb8105e10 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.278+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb81a2900 con 0x7f0bb8105e10 2024-04-24T08:20:58.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.279+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb810b600 con 0x7f0bb8105e10 2024-04-24T08:20:58.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.281+0000 7f0bb6000640 1 -- 172.21.15.146:0/4294701961 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f0bb0044020 con 0x7f0bb8105e10 2024-04-24T08:20:58.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.282+0000 7f0bb6000640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ba00a3c30 0x7f0ba00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.283+0000 7f0bb6000640 1 -- 172.21.15.146:0/4294701961 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f0bb00f12b0 con 0x7f0bb8105e10 2024-04-24T08:20:58.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.283+0000 7f0bbd800640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ba00a3c30 0x7f0ba00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.283+0000 7f0bbd800640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ba00a3c30 0x7f0ba00a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0bac005fd0 tx=0x7f0bac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:58.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.285+0000 7f0bb6000640 1 -- 172.21.15.146:0/4294701961 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0bb004b020 con 0x7f0bb8105e10 2024-04-24T08:20:58.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.487+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0bb802d7b0 con 0x7f0bb8105e10 2024-04-24T08:20:58.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.487+0000 7f0bb6000640 1 -- 172.21.15.146:0/4294701961 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f0bb00bdc50 con 0x7f0bb8105e10 2024-04-24T08:20:58.488 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:20:58.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ba00a3c30 msgr2=0x7f0ba00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ba00a3c30 0x7f0ba00a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0bac005fd0 tx=0x7f0bac00b410 comp rx=0 tx=0).stop 2024-04-24T08:20:58.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 msgr2=0x7f0bb81a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 0x7f0bb81a1ac0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0034040 tx=0x7f0bb00305d0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 shutdown_connections 2024-04-24T08:20:58.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f0ba00a3c30 0x7f0ba00a60f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0bb8105e10 0x7f0bb81a1ac0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb8105450 0x7f0bb819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 --2- 172.21.15.146:0/4294701961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bb8104260 0x7f0bb819a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 >> 172.21.15.146:0/4294701961 conn(0x7f0bb80ff7f0 msgr2=0x7f0bb81087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:58.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 shutdown_connections 2024-04-24T08:20:58.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.490+0000 7f0bc0200640 1 -- 172.21.15.146:0/4294701961 wait complete. 2024-04-24T08:20:58.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 -- 172.21.15.146:0/3110859562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105e10 msgr2=0x7fa11010c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 --2- 172.21.15.146:0/3110859562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105e10 0x7fa11010c6a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa108009ca0 tx=0x7fa10802a8b0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 -- 172.21.15.146:0/3110859562 shutdown_connections 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 --2- 172.21.15.146:0/3110859562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105e10 0x7fa11010c6a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 --2- 172.21.15.146:0/3110859562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa110105450 0x7fa1101058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 --2- 172.21.15.146:0/3110859562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110104260 0x7fa110104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 -- 172.21.15.146:0/3110859562 >> 172.21.15.146:0/3110859562 conn(0x7fa1100ff7f0 msgr2=0x7fa110101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 -- 172.21.15.146:0/3110859562 shutdown_connections 2024-04-24T08:20:58.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.715+0000 7fa115800640 1 -- 172.21.15.146:0/3110859562 wait complete. 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.716+0000 7fa115800640 1 Processor -- start 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.716+0000 7fa115800640 1 -- start start 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa115800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110104260 0x7fa11019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa115800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105450 0x7fa11019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa115800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa110105e10 0x7fa1101a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa115800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa11010ff10 con 0x7fa110105e10 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa115800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa11010fd90 con 0x7fa110104260 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa115800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa110110090 con 0x7fa110105450 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa10e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105450 0x7fa11019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa10f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa110105e10 0x7fa1101a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa10e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105450 0x7fa11019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38768/0 (socket says 172.21.15.146:38768) 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa10e000640 1 -- 172.21.15.146:0/246698663 learned_addr learned my addr 172.21.15.146:0/246698663 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:58.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.717+0000 7fa10ea00640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110104260 0x7fa11019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa10e000640 1 -- 172.21.15.146:0/246698663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110104260 msgr2=0x7fa11019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa10e000640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110104260 0x7fa11019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa10e000640 1 -- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa110105e10 msgr2=0x7fa1101a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa10e000640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa110105e10 0x7fa1101a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa10e000640 1 -- 172.21.15.146:0/246698663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1101a2140 con 0x7fa110105450 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa10e000640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105450 0x7fa11019aa60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00d950 tx=0x7fa0fc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa107400640 1 -- 172.21.15.146:0/246698663 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc017070 con 0x7fa110105450 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1101a2430 con 0x7fa110105450 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa107400640 1 -- 172.21.15.146:0/246698663 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0fc009c30 con 0x7fa110105450 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa107400640 1 -- 172.21.15.146:0/246698663 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00f6b0 con 0x7fa110105450 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.718+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1101a2980 con 0x7fa110105450 2024-04-24T08:20:58.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.719+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11010b520 con 0x7fa110105450 2024-04-24T08:20:58.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.722+0000 7fa107400640 1 -- 172.21.15.146:0/246698663 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fa0fc0109d0 con 0x7fa110105450 2024-04-24T08:20:58.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.723+0000 7fa107400640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0f40a3bb0 0x7fa0f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:58.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.723+0000 7fa107400640 1 -- 172.21.15.146:0/246698663 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fa0fc0cff70 con 0x7fa110105450 2024-04-24T08:20:58.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.724+0000 7fa10ea00640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0f40a3bb0 0x7fa0f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:58.724 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.724+0000 7fa10ea00640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0f40a3bb0 0x7fa0f40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa100005fd0 tx=0x7fa10000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:58.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.728+0000 7fa107400640 1 -- 172.21.15.146:0/246698663 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0fc009de0 con 0x7fa110105450 2024-04-24T08:20:58.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:58 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1168940726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:58.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:58 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4294701961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:58.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:58 smithi153 ceph-mon[32746]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:58.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.973+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa11002d620 con 0x7fa110105450 2024-04-24T08:20:58.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.974+0000 7fa107400640 1 -- 172.21.15.146:0/246698663 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11891 (secure 0 0 0) 0x7fa0fc09c2c0 con 0x7fa110105450 2024-04-24T08:20:58.975 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:58.976 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:18:52.518710+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:20:58.976 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 23 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0f40a3bb0 msgr2=0x7fa0f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0f40a3bb0 0x7fa0f40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa100005fd0 tx=0x7fa10000b410 comp rx=0 tx=0).stop 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105450 msgr2=0x7fa11019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105450 0x7fa11019aa60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00d950 tx=0x7fa0fc00de10 comp rx=0 tx=0).stop 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 shutdown_connections 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa110105450 0x7fa11019aa60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fa0f40a3bb0 0x7fa0f40a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa110105e10 0x7fa1101a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 --2- 172.21.15.146:0/246698663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa110104260 0x7fa11019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:58.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.977+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 >> 172.21.15.146:0/246698663 conn(0x7fa1100ff7f0 msgr2=0x7fa1101086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:58.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.978+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 shutdown_connections 2024-04-24T08:20:58.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:58.978+0000 7fa115800640 1 -- 172.21.15.146:0/246698663 wait complete. 2024-04-24T08:20:58.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:58 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1168940726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:58.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:58 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4294701961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:58.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:58 smithi146 ceph-mon[27805]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:58.997 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:58 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1168940726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:20:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:58 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4294701961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:20:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:58 smithi177 ceph-mon[33046]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 -- 172.21.15.146:0/2472639094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0109850 msgr2=0x7f03f0109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 --2- 172.21.15.146:0/2472639094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0109850 0x7f03f0109cd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009900 tx=0x7f03dc02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 -- 172.21.15.146:0/2472639094 shutdown_connections 2024-04-24T08:20:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 --2- 172.21.15.146:0/2472639094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03f010a210 0x7f03f0110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 --2- 172.21.15.146:0/2472639094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0109850 0x7f03f0109cd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 --2- 172.21.15.146:0/2472639094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0108660 0x7f03f0108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 -- 172.21.15.146:0/2472639094 >> 172.21.15.146:0/2472639094 conn(0x7f03f0077620 msgr2=0x7f03f0079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 -- 172.21.15.146:0/2472639094 shutdown_connections 2024-04-24T08:20:59.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.193+0000 7f03f6200640 1 -- 172.21.15.146:0/2472639094 wait complete. 2024-04-24T08:20:59.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.194+0000 7f03f6200640 1 Processor -- start 2024-04-24T08:20:59.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03f6200640 1 -- start start 2024-04-24T08:20:59.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03f6200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0108660 0x7f03f019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03f6200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0109850 0x7f03f019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03f6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03f010a210 0x7f03f01a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03f6200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03f01142f0 con 0x7f03f0109850 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03f6200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03f0114170 con 0x7f03f0108660 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03f6200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03f0114470 con 0x7f03f010a210 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03eea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0109850 0x7f03f019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03eea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0109850 0x7f03f019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39338/0 (socket says 172.21.15.146:39338) 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03eea00640 1 -- 172.21.15.146:0/2157507670 learned_addr learned my addr 172.21.15.146:0/2157507670 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03ef400640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0108660 0x7f03f019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.195+0000 7f03efe00640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03f010a210 0x7f03f01a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03eea00640 1 -- 172.21.15.146:0/2157507670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03f010a210 msgr2=0x7f03f01a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03eea00640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03f010a210 0x7f03f01a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03eea00640 1 -- 172.21.15.146:0/2157507670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0108660 msgr2=0x7f03f019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03eea00640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0108660 0x7f03f019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03eea00640 1 -- 172.21.15.146:0/2157507670 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f01a6580 con 0x7f03f0109850 2024-04-24T08:20:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03eea00640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0109850 0x7f03f019eea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009230 tx=0x7f03dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03e7e00640 1 -- 172.21.15.146:0/2157507670 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc03f030 con 0x7f03f0109850 2024-04-24T08:20:59.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03e7e00640 1 -- 172.21.15.146:0/2157507670 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03dc003ce0 con 0x7f03f0109850 2024-04-24T08:20:59.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f01a6810 con 0x7f03f0109850 2024-04-24T08:20:59.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03e7e00640 1 -- 172.21.15.146:0/2157507670 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc035070 con 0x7f03f0109850 2024-04-24T08:20:59.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.196+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f01a6d50 con 0x7f03f0109850 2024-04-24T08:20:59.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.198+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f010f980 con 0x7f03f0109850 2024-04-24T08:20:59.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.200+0000 7f03e7e00640 1 -- 172.21.15.146:0/2157507670 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f03dc042030 con 0x7f03f0109850 2024-04-24T08:20:59.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.201+0000 7f03e7e00640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d40a3c30 0x7f03d40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.202+0000 7f03ef400640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d40a3c30 0x7f03d40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.202+0000 7f03e7e00640 1 -- 172.21.15.146:0/2157507670 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f03dc0f04d0 con 0x7f03f0109850 2024-04-24T08:20:59.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.202+0000 7f03ef400640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d40a3c30 0x7f03d40a60f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f03f01096b0 tx=0x7f03e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.205+0000 7f03e7e00640 1 -- 172.21.15.146:0/2157507670 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03dc0bc820 con 0x7f03f0109850 2024-04-24T08:20:59.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.409+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03f0062c00 con 0x7f03f0109850 2024-04-24T08:20:59.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.411+0000 7f03e7e00640 1 -- 172.21.15.146:0/2157507670 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22685 (secure 0 0 0) 0x7f03dc0c0b20 con 0x7f03f0109850 2024-04-24T08:20:59.412 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:20:59.413 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":90,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:18:57.557751+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d40a3c30 msgr2=0x7f03d40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d40a3c30 0x7f03d40a60f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f03f01096b0 tx=0x7f03e0005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0109850 msgr2=0x7f03f019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0109850 0x7f03f019eea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f03dc009230 tx=0x7f03dc0042f0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 shutdown_connections 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03f010a210 0x7f03f01a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f03d40a3c30 0x7f03d40a60f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03f0109850 0x7f03f019eea0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 --2- 172.21.15.146:0/2157507670 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03f0108660 0x7f03f019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 >> 172.21.15.146:0/2157507670 conn(0x7f03f0077620 msgr2=0x7f03f010cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 shutdown_connections 2024-04-24T08:20:59.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.414+0000 7f03f6200640 1 -- 172.21.15.146:0/2157507670 wait complete. 2024-04-24T08:20:59.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-24T08:18:46.894792+0000', 'modified': '2024-04-24T08:18:52.518710+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24569, 'mds_1': 24568, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6837/2495345664', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 2495345664}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 2495345664}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24518': {'gid': 24518, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6837/565470680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6836', 'nonce': 565470680}, {'type': 'v1', 'addr': '172.21.15.153:6837', 'nonce': 565470680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6839/1533073551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 1533073551}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 1533073551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.146:6839/2207966302', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6838', 'nonce': 2207966302}, {'type': 'v1', 'addr': '172.21.15.146:6839', 'nonce': 2207966302}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24568': {'gid': 24568, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.146:6841/1590374458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6840', 'nonce': 1590374458}, {'type': 'v1', 'addr': '172.21.15.146:6841', 'nonce': 1590374458}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.153:6839/2023799727', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 2023799727}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 2023799727}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24569, 'qdb_cluster': [24553, 24568, 24569]} 2024-04-24T08:20:59.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-24T08:20:59.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-24T08:20:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 -- 172.21.15.146:0/4094105462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c104280 msgr2=0x7f5e7c1046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 --2- 172.21.15.146:0/4094105462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c104280 0x7f5e7c1046a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009be0 tx=0x7f5e7402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 -- 172.21.15.146:0/4094105462 shutdown_connections 2024-04-24T08:20:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 --2- 172.21.15.146:0/4094105462 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c105e30 0x7f5e7c10c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 --2- 172.21.15.146:0/4094105462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7c105470 0x7f5e7c1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 --2- 172.21.15.146:0/4094105462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c104280 0x7f5e7c1046a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 -- 172.21.15.146:0/4094105462 >> 172.21.15.146:0/4094105462 conn(0x7f5e7c0ff7f0 msgr2=0x7f5e7c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 -- 172.21.15.146:0/4094105462 shutdown_connections 2024-04-24T08:20:59.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.644+0000 7f5e81800640 1 -- 172.21.15.146:0/4094105462 wait complete. 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.645+0000 7f5e81800640 1 Processor -- start 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.645+0000 7f5e81800640 1 -- start start 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e81800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c104280 0x7f5e7c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e81800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c105470 0x7f5e7c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e81800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7c105e30 0x7f5e7c1a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e81800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c10ff10 con 0x7f5e7c105e30 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e81800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c10fd90 con 0x7f5e7c104280 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e81800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c110090 con 0x7f5e7c105470 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7c105e30 0x7f5e7c1a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c104280 0x7f5e7c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c104280 0x7f5e7c19a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:35944/0 (socket says 172.21.15.146:35944) 2024-04-24T08:20:59.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2965407625 learned_addr learned my addr 172.21.15.146:0/2965407625 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7a000640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c105470 0x7f5e7c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2965407625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c105470 msgr2=0x7f5e7c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c105470 0x7f5e7c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7c105e30 msgr2=0x7f5e7c1a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7c105e30 0x7f5e7c1a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.646+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2965407625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7c1a21d0 con 0x7f5e7c104280 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.647+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c104280 0x7f5e7c19a5b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402acb0 tx=0x7f5e740305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.647+0000 7f5e73400640 1 -- 172.21.15.146:0/2965407625 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74041020 con 0x7f5e7c104280 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.647+0000 7f5e73400640 1 -- 172.21.15.146:0/2965407625 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e74031920 con 0x7f5e7c104280 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.647+0000 7f5e73400640 1 -- 172.21.15.146:0/2965407625 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74036040 con 0x7f5e7c104280 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.647+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e7c1a2460 con 0x7f5e7c104280 2024-04-24T08:20:59.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.647+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e7c1a2900 con 0x7f5e7c104280 2024-04-24T08:20:59.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.650+0000 7f5e84614740 1 Processor -- start 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.650+0000 7f5e73400640 1 -- 172.21.15.146:0/2965407625 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f5e74044020 con 0x7f5e7c104280 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.650+0000 7f5e84614740 1 -- start start 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e84614740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504090 0x55d6035044b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e84614740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d603509580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e84614740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d60350c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e84614740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d603469b10 con 0x55d603504090 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e84614740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d6034102b0 con 0x55d603504a80 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e84614740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d603465ad0 con 0x55d603509bf0 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504090 0x55d6035044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d603509580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e7a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d60350c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d603509580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:35946/0 (socket says 172.21.15.146:35946) 2024-04-24T08:20:59.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.651+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2583470373 learned_addr learned my addr 172.21.15.146:0/2583470373 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2583470373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 msgr2=0x55d60350c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d60350c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2583470373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504090 msgr2=0x55d6035044b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504090 0x55d6035044b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e7aa00640 1 -- 172.21.15.146:0/2583470373 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d60317c2d0 con 0x55d603504a80 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e7a000640 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d60350c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d603509580 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e740f9190 tx=0x7f5e740f9280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2029e0b6ba18fc8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e6be00640 1 -- 172.21.15.146:0/2583470373 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740f9cb0 con 0x55d603504a80 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e73400640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5e600a3c30 0x7f5e600a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e6be00640 1 -- 172.21.15.146:0/2583470373 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e740fe030 con 0x55d603504a80 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e6be00640 1 -- 172.21.15.146:0/2583470373 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740b9370 con 0x55d603504a80 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.652+0000 7f5e73400640 1 -- 172.21.15.146:0/2965407625 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f5e740f0fe0 con 0x7f5e7c104280 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e7b400640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5e600a3c30 0x7f5e600a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 -- 172.21.15.146:0/2583470373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 msgr2=0x55d603509580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d603509580 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e740f9190 tx=0x7f5e740f9280 comp rx=0 tx=0).stop 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 -- 172.21.15.146:0/2583470373 shutdown_connections 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d60350c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d603509580 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 --2- 172.21.15.146:0/2583470373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504090 0x55d6035044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 -- 172.21.15.146:0/2583470373 >> 172.21.15.146:0/2583470373 conn(0x55d6034ff700 msgr2=0x55d6034ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 -- 172.21.15.146:0/2583470373 shutdown_connections 2024-04-24T08:20:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e7b400640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5e600a3c30 0x7f5e600a60f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55d6035052e0 tx=0x7f5e6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.653+0000 7f5e84614740 1 -- 172.21.15.146:0/2583470373 wait complete. 2024-04-24T08:20:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 -- start start 2024-04-24T08:20:59.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 Processor -- start 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504090 0x55d60350e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d60350e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d603515720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d60350c960 con 0x55d603509bf0 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d603423590 con 0x55d603504a80 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d60350cae0 con 0x55d603504090 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.654+0000 7f5e84614740 10 client.0 _ll_register_callbacks cb 0x55d6034063a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d60350e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d603515720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d60350e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:35956/0 (socket says 172.21.15.146:35956) 2024-04-24T08:20:59.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 -- 172.21.15.146:0/2063935065 learned_addr learned my addr 172.21.15.146:0/2063935065 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504090 0x55d60350e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 -- 172.21.15.146:0/2063935065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504090 msgr2=0x55d60350e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504090 0x55d60350e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 -- 172.21.15.146:0/2063935065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 msgr2=0x55d603515720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d603515720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 -- 172.21.15.146:0/2063935065 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6032466b0 con 0x55d603504a80 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e7a000640 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d60350e6e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5e64011760 tx=0x7f5e64011c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71f13c61e1a35773 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.655+0000 7f5e68c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e68c00640 1 -- 172.21.15.146:0/2063935065 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e64004300 con 0x55d603504a80 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e68c00640 1 -- 172.21.15.146:0/2063935065 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f5e6400f8c0 con 0x55d603504a80 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e740fe030 con 0x55d603504a80 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e68c00640 1 -- 172.21.15.146:0/2063935065 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5e640044a0 con 0x55d603504a80 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e84614740 10 client.25136 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d603518b00 con 0x55d603504a80 2024-04-24T08:20:59.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e68c00640 1 -- 172.21.15.146:0/2063935065 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e740fe030 con 0x55d603504a80 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.656+0000 7f5e68c00640 1 -- 172.21.15.146:0/2063935065 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5e64019030 con 0x55d603504a80 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e84614740 10 client.25136 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e84614740 10 client.25136 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e84614740 20 client.25136 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e84614740 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d60351b190 0x55d60351c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e84614740 4 client.25136 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d603477530 con 0x55d60351b190 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d60351b190 0x55d60351c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d60351b190 0x55d60351c600 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=676bb4710279adfe server_cookie=d6340f088ec11e2a in_seq=0 out_seq=0 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.657+0000 7f5e68c00640 10 client.25136 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e68c00640 1 -- 172.21.15.146:0/2063935065 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d603477530 con 0x55d60351b190 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e68c00640 10 client.25136 handle_command_reply: tid=0 2024-04-24T08:20:59.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 client.25136 shutdown 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 msgr2=0x55d60350e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d60350e6e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5e64011760 tx=0x7f5e64011c20 comp rx=0 tx=0).stop 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 shutdown_connections 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d60351b190 0x55d60351c600 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d603515720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e68c00640 0 client.25136 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504a80 0x55d60350e6e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.658+0000 7f5e84614740 1 --2- 172.21.15.146:0/2063935065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504090 0x55d60350e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.659+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 >> 172.21.15.146:0/2063935065 conn(0x55d6034ff700 msgr2=0x55d6034fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.659+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 shutdown_connections 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.659+0000 7f5e84614740 1 -- 172.21.15.146:0/2063935065 wait complete. 2024-04-24T08:20:59.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.659+0000 7f5e84614740 20 client.25136 trim_cache size 0 max 16384 2024-04-24T08:20:59.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.678+0000 7f5e84614740 1 Processor -- start 2024-04-24T08:20:59.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.678+0000 7f5e84614740 1 -- start start 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.678+0000 7f5e84614740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603509430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e84614740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504a80 0x55d6035156e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e84614740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d6035023e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e84614740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d60351a940 con 0x55d603504a80 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e84614740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d60351b010 con 0x55d603504090 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e84614740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d60351b6e0 con 0x55d603509bf0 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504a80 0x55d6035156e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603509430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504a80 0x55d6035156e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39388/0 (socket says 172.21.15.146:39388) 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7a000640 1 -- 172.21.15.146:0/3677826533 learned_addr learned my addr 172.21.15.146:0/3677826533 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7b400640 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d6035023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7aa00640 1 -- 172.21.15.146:0/3677826533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 msgr2=0x55d6035023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d6035023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7aa00640 1 -- 172.21.15.146:0/3677826533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504a80 msgr2=0x55d6035156e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504a80 0x55d6035156e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7aa00640 1 -- 172.21.15.146:0/3677826533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d60350f320 con 0x55d603504090 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.679+0000 7f5e7a000640 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504a80 0x55d6035156e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603509430 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55d603504e80 tx=0x7f5e74004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7296e7f2588ff05 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e6b400640 1 -- 172.21.15.146:0/3677826533 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740bc6f0 con 0x55d603504090 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e6b400640 1 -- 172.21.15.146:0/3677826533 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e740fe040 con 0x55d603504090 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e6b400640 1 -- 172.21.15.146:0/3677826533 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740bca20 con 0x55d603504090 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e84614740 1 -- 172.21.15.146:0/3677826533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 msgr2=0x55d603509430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e84614740 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603509430 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55d603504e80 tx=0x7f5e74004a40 comp rx=0 tx=0).stop 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e84614740 1 -- 172.21.15.146:0/3677826533 shutdown_connections 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e84614740 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603509bf0 0x55d6035023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e84614740 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603504a80 0x55d6035156e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e84614740 1 --2- 172.21.15.146:0/3677826533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603509430 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.680+0000 7f5e84614740 1 -- 172.21.15.146:0/3677826533 >> 172.21.15.146:0/3677826533 conn(0x55d6034ff700 msgr2=0x55d6034ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.681+0000 7f5e84614740 1 -- 172.21.15.146:0/3677826533 shutdown_connections 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.681+0000 7f5e84614740 1 -- 172.21.15.146:0/3677826533 wait complete. 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.681+0000 7f5e84614740 1 -- start start 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.681+0000 7f5e84614740 1 Processor -- start 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.681+0000 7f5e84614740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603508880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.681+0000 7f5e84614740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504a80 0x55d603515640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.681+0000 7f5e84614740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d60357f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e84614740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55d60351c480 con 0x55d603509bf0 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e84614740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55d60351bdb0 con 0x55d603504090 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e84614740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d60351cb50 con 0x55d603504a80 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e84614740 10 client.0 _ll_register_callbacks cb 0x7f5e64009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:59.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d60357f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d60357f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39392/0 (socket says 172.21.15.146:39392) 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504a80 0x55d603515640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 -- 172.21.15.146:0/2529931244 learned_addr learned my addr 172.21.15.146:0/2529931244 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603508880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 -- 172.21.15.146:0/2529931244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504a80 msgr2=0x55d603515640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504a80 0x55d603515640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 -- 172.21.15.146:0/2529931244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 msgr2=0x55d603508880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603508880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.682+0000 7f5e7b400640 1 -- 172.21.15.146:0/2529931244 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d60357f9c0 con 0x55d603509bf0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e7b400640 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d60357f190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c003730 tx=0x7f5e6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c399b8e0341f5d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e68c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e68c00640 1 -- 172.21.15.146:0/2529931244 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c005020 con 0x55d603509bf0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e68c00640 1 -- 172.21.15.146:0/2529931244 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f5e6c0051c0 con 0x55d603509bf0 2024-04-24T08:20:59.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e740fe040 con 0x55d603509bf0 2024-04-24T08:20:59.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e68c00640 1 -- 172.21.15.146:0/2529931244 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5e6c0053c0 con 0x55d603509bf0 2024-04-24T08:20:59.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e84614740 10 client.15030 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:59.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d603580d90 con 0x55d603509bf0 2024-04-24T08:20:59.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.683+0000 7f5e68c00640 1 -- 172.21.15.146:0/2529931244 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e740fe040 con 0x55d603509bf0 2024-04-24T08:20:59.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e68c00640 1 -- 172.21.15.146:0/2529931244 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5e6c004bb0 con 0x55d603509bf0 2024-04-24T08:20:59.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e84614740 10 client.15030 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:59.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e84614740 10 client.15030 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e84614740 20 client.15030 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e84614740 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d603585a20 0x55d603584730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e84614740 4 client.15030 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "status"}) -- 0x55d6034713a0 con 0x55d603585a20 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d603585a20 0x55d603584730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e7aa00640 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d603585a20 0x55d603584730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9b7e675ca19cbcd server_cookie=e6e6a659c5cfa15a in_seq=0 out_seq=0 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.684+0000 7f5e68c00640 10 client.15030 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e68c00640 1 -- 172.21.15.146:0/2529931244 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d6034713a0 con 0x55d603585a20 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e68c00640 10 client.15030 handle_command_reply: tid=0 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 client.15030 shutdown 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 msgr2=0x55d60357f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d60357f190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c003730 tx=0x7f5e6c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 shutdown_connections 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55d603585a20 0x55d603584730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d603504a80 0x55d603515640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55d603509bf0 0x55d60357f190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 --2- 172.21.15.146:0/2529931244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55d603504090 0x55d603508880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e68c00640 0 client.15030 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 >> 172.21.15.146:0/2529931244 conn(0x55d6034ff700 msgr2=0x55d603509000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 shutdown_connections 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 1 -- 172.21.15.146:0/2529931244 wait complete. 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.685+0000 7f5e84614740 20 client.15030 trim_cache size 0 max 16384 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:20:59.690 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 2, 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 24553, 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 128.17615100699999, 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 23, 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 90, 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 88, 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 144.44129207399999 2024-04-24T08:20:59.691 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5e600a3c30 msgr2=0x7f5e600a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5e600a3c30 0x7f5e600a60f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55d6035052e0 tx=0x7f5e6c005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c104280 msgr2=0x7f5e7c19a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c104280 0x7f5e7c19a5b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402acb0 tx=0x7f5e740305d0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 shutdown_connections 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e7c105470 0x7f5e7c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f5e600a3c30 0x7f5e600a60f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7c105e30 0x7f5e7c1a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 --2- 172.21.15.146:0/2965407625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e7c104280 0x7f5e7c19a5b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 >> 172.21.15.146:0/2965407625 conn(0x7f5e7c0ff7f0 msgr2=0x7f5e7c108840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 shutdown_connections 2024-04-24T08:20:59.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.686+0000 7f5e81800640 1 -- 172.21.15.146:0/2965407625 wait complete. 2024-04-24T08:20:59.702 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:59 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/246698663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:59.702 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:20:59 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2157507670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:59.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T08:20:59.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 -- 172.21.15.146:0/2095182090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 msgr2=0x7f1ae4109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/2095182090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae4109bb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1acc009ca0 tx=0x7f1acc02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 -- 172.21.15.146:0/2095182090 shutdown_connections 2024-04-24T08:20:59.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/2095182090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae410a0f0 0x7f1ae4110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/2095182090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae4109bb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/2095182090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ae4108540 0x7f1ae4108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 -- 172.21.15.146:0/2095182090 >> 172.21.15.146:0/2095182090 conn(0x7f1ae40774d0 msgr2=0x7f1ae4079910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 -- 172.21.15.146:0/2095182090 shutdown_connections 2024-04-24T08:20:59.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.903+0000 7f1ae9a00640 1 -- 172.21.15.146:0/2095182090 wait complete. 2024-04-24T08:20:59.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.904+0000 7f1ae9a00640 1 Processor -- start 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.904+0000 7f1ae9a00640 1 -- start start 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ae4108540 0x7f1ae419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae9a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae410a0f0 0x7f1ae41a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae4114220 con 0x7f1ae410a0f0 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae9a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41140a0 con 0x7f1ae4109730 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae9a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae41143a0 con 0x7f1ae4108540 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae410a0f0 0x7f1ae41a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:36004/0 (socket says 172.21.15.146:36004) 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 -- 172.21.15.146:0/1943251821 learned_addr learned my addr 172.21.15.146:0/1943251821 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ae4108540 0x7f1ae419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 -- 172.21.15.146:0/1943251821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ae4108540 msgr2=0x7f1ae419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ae4108540 0x7f1ae419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 -- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae410a0f0 msgr2=0x7f1ae41a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae410a0f0 0x7f1ae41a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1adb400640 1 -- 172.21.15.146:0/1943251821 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae41a63f0 con 0x7f1ae4109730 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.905+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ae4108540 0x7f1ae419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.906+0000 7f1adb400640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae419ee50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1acc000f80 tx=0x7f1acc0319f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.906+0000 7f1ae1600640 1 -- 172.21.15.146:0/1943251821 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc003b70 con 0x7f1ae4109730 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.906+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae41a6680 con 0x7f1ae4109730 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.906+0000 7f1ae1600640 1 -- 172.21.15.146:0/1943251821 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1acc003d10 con 0x7f1ae4109730 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.906+0000 7f1ae1600640 1 -- 172.21.15.146:0/1943251821 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc02fc00 con 0x7f1ae4109730 2024-04-24T08:20:59.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.906+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae41a6c20 con 0x7f1ae4109730 2024-04-24T08:20:59.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1ae1600640 1 -- 172.21.15.146:0/1943251821 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f1acc004040 con 0x7f1ae4109730 2024-04-24T08:20:59.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 Processor -- start 2024-04-24T08:20:59.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 -- start start 2024-04-24T08:20:59.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9a4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9a9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e10b909b10 con 0x55e10b9a4260 2024-04-24T08:20:59.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e10b8b02b0 con 0x55e10b9a4c50 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e10b905ad0 con 0x55e10b9a9dc0 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1ae9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1ae9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39408/0 (socket says 172.21.15.146:39408) 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9a9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.910+0000 7f1ae9000640 1 -- 172.21.15.146:0/2663504573 learned_addr learned my addr 172.21.15.146:0/2663504573 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1adb400640 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ae3e00640 1 -- 172.21.15.146:0/2663504573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 msgr2=0x55e10b9ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9ac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ae3e00640 1 -- 172.21.15.146:0/2663504573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 msgr2=0x55e10b9a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ae3e00640 1 -- 172.21.15.146:0/2663504573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10b61c2d0 con 0x55e10b9a4c50 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9a9750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009fa0 tx=0x7f1ad40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ecf8920db2a4a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ae1600640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1ac80a3bb0 0x7f1ac80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.911+0000 7f1ad9600640 1 -- 172.21.15.146:0/2663504573 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4004030 con 0x55e10b9a4c50 2024-04-24T08:20:59.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1ad9600640 1 -- 172.21.15.146:0/2663504573 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad40041d0 con 0x55e10b9a4c50 2024-04-24T08:20:59.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1ad9600640 1 -- 172.21.15.146:0/2663504573 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad400f8d0 con 0x55e10b9a4c50 2024-04-24T08:20:59.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1ae1600640 1 -- 172.21.15.146:0/1943251821 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1acc0f19e0 con 0x7f1ae4109730 2024-04-24T08:20:59.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1ae9000640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1ac80a3bb0 0x7f1ac80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2663504573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 msgr2=0x55e10b9a9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9a9750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4009fa0 tx=0x7f1ad40118d0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2663504573 shutdown_connections 2024-04-24T08:20:59.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9ac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9a9750 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2663504573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2663504573 >> 172.21.15.146:0/2663504573 conn(0x55e10b99f8d0 msgr2=0x55e10b99fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1ae9000640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1ac80a3bb0 0x7f1ac80a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55e10b9a54b0 tx=0x7f1adc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2663504573 shutdown_connections 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.912+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2663504573 wait complete. 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.913+0000 7f1aeddcd740 1 -- start start 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.913+0000 7f1aeddcd740 1 Processor -- start 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.913+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.913+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9acd70 con 0x55e10b9a4260 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e10b8c3590 con 0x55e10b9a4c50 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9acef0 con 0x55e10b9a9dc0 2024-04-24T08:20:59.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1aeddcd740 10 client.0 _ll_register_callbacks cb 0x55e10b8a63a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39420/0 (socket says 172.21.15.146:39420) 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 -- 172.21.15.146:0/2577798246 learned_addr learned my addr 172.21.15.146:0/2577798246 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae9000640 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1adb400640 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 -- 172.21.15.146:0/2577798246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 msgr2=0x55e10b9b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 -- 172.21.15.146:0/2577798246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 msgr2=0x55e10b9aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 -- 172.21.15.146:0/2577798246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10b6e66b0 con 0x55e10b9a4260 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ae6b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1ad401b040 tx=0x7f1ad4003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c976d99f0dbaa41a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.914+0000 7f1ad2000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.915+0000 7f1ad2000640 1 -- 172.21.15.146:0/2577798246 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4027070 con 0x55e10b9a4260 2024-04-24T08:20:59.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.915+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad40041d0 con 0x55e10b9a4260 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.915+0000 7f1ad2000640 1 -- 172.21.15.146:0/2577798246 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1ad4005040 con 0x55e10b9a4260 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.915+0000 7f1ad2000640 1 -- 172.21.15.146:0/2577798246 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1ad4033050 con 0x55e10b9a4260 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.915+0000 7f1aeddcd740 10 client.15042 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.915+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e10b9b8ff0 con 0x55e10b9a4260 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.915+0000 7f1ad2000640 1 -- 172.21.15.146:0/2577798246 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad40041d0 con 0x55e10b9a4260 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1ad2000640 1 -- 172.21.15.146:0/2577798246 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1ad4004c50 con 0x55e10b9a4260 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1aeddcd740 10 client.15042 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1aeddcd740 10 client.15042 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1aeddcd740 20 client.15042 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10b9bb3d0 0x55e10b9bc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1aeddcd740 4 client.15042 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e10b917530 con 0x55e10b9bb3d0 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1adb400640 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10b9bb3d0 0x55e10b9bc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1adb400640 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10b9bb3d0 0x55e10b9bc840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fb71428f23d08009 server_cookie=8812260d1eb9fde0 in_seq=0 out_seq=0 2024-04-24T08:20:59.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.916+0000 7f1ad2000640 10 client.15042 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.917+0000 7f1ad2000640 1 -- 172.21.15.146:0/2577798246 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e10b917530 con 0x55e10b9bb3d0 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.917+0000 7f1ad2000640 10 client.15042 handle_command_reply: tid=0 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.917+0000 7f1aeddcd740 1 client.15042 shutdown 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 msgr2=0x55e10b9ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ae6b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1ad401b040 tx=0x7f1ad4003c60 comp rx=0 tx=0).stop 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 shutdown_connections 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10b9bb3d0 0x55e10b9bc840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b9b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1ad2000640 0 client.15042 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2577798246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ae6b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 >> 172.21.15.146:0/2577798246 conn(0x55e10b99f8d0 msgr2=0x55e10b99e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 shutdown_connections 2024-04-24T08:20:59.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2577798246 wait complete. 2024-04-24T08:20:59.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.918+0000 7f1aeddcd740 20 client.15042 trim_cache size 0 max 16384 2024-04-24T08:20:59.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 Processor -- start 2024-04-24T08:20:59.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 -- start start 2024-04-24T08:20:59.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b99aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9bab80 con 0x55e10b9a4260 2024-04-24T08:20:59.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9bb250 con 0x55e10b9a4c50 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.938+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9bb920 con 0x55e10b9a9dc0 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39432/0 (socket says 172.21.15.146:39432) 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 -- 172.21.15.146:0/3530778834 learned_addr learned my addr 172.21.15.146:0/3530778834 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1adb400640 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae9000640 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b99aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 -- 172.21.15.146:0/3530778834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 msgr2=0x55e10b99aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b99aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 -- 172.21.15.146:0/3530778834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 msgr2=0x55e10b9b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9b5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 -- 172.21.15.146:0/3530778834 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e10b9a9330 con 0x55e10b9a4260 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1adb400640 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.939+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ac1b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400f780 tx=0x7f1ad400f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02479a0d4cbf188 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1ad8c00640 1 -- 172.21.15.146:0/3530778834 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4027040 con 0x55e10b9a4260 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1ad8c00640 1 -- 172.21.15.146:0/3530778834 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad4005520 con 0x55e10b9a4260 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1ad8c00640 1 -- 172.21.15.146:0/3530778834 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4004590 con 0x55e10b9a4260 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 -- 172.21.15.146:0/3530778834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 msgr2=0x55e10b9ac1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ac1b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400f780 tx=0x7f1ad400f5a0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 -- 172.21.15.146:0/3530778834 shutdown_connections 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a9dc0 0x55e10b99aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a4c50 0x55e10b9b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/3530778834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4260 0x55e10b9ac1b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 -- 172.21.15.146:0/3530778834 >> 172.21.15.146:0/3530778834 conn(0x55e10b99f8d0 msgr2=0x55e10b99e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 -- 172.21.15.146:0/3530778834 shutdown_connections 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.940+0000 7f1aeddcd740 1 -- 172.21.15.146:0/3530778834 wait complete. 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 -- start start 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 Processor -- start 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a4260 0x55e10b9b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4c50 0x55e10ba1d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a9dc0 0x55e10ba1f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9bc6c0 con 0x55e10b9a4c50 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9bbff0 con 0x55e10b9a9dc0 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e10b9bcd90 con 0x55e10b9a4260 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.941+0000 7f1aeddcd740 10 client.0 _ll_register_callbacks cb 0x7f1ad4009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4c50 0x55e10ba1d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4c50 0x55e10ba1d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39438/0 (socket says 172.21.15.146:39438) 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 -- 172.21.15.146:0/2999411614 learned_addr learned my addr 172.21.15.146:0/2999411614 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a4260 0x55e10b9b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1ae9000640 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a9dc0 0x55e10ba1f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 -- 172.21.15.146:0/2999411614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a4260 msgr2=0x55e10b9b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a4260 0x55e10b9b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 -- 172.21.15.146:0/2999411614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a9dc0 msgr2=0x55e10ba1f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a9dc0 0x55e10ba1f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 -- 172.21.15.146:0/2999411614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e10ba1fe40 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1adb400640 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4c50 0x55e10ba1d360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0bdc10 tx=0x7f1acc0f7480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e93b5c86ccd8db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1ad2000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.942+0000 7f1ad2000640 1 -- 172.21.15.146:0/2999411614 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc0bd5e0 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1ad2000640 1 -- 172.21.15.146:0/2999411614 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f1acc02fda0 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ad4005520 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1ad2000640 1 -- 172.21.15.146:0/2999411614 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1acc0be380 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 10 client.15048 fetch_fsmap learned FSMap version 23 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e10ba21140 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1ad2000640 1 -- 172.21.15.146:0/2999411614 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad4005520 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1ad2000640 1 -- 172.21.15.146:0/2999411614 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1acc0ba2f0 con 0x55e10b9a4c50 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 10 client.15048 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:20:59.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 10 client.15048 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 20 client.15048 populate_metadata read hostname 'smithi146' 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10ba25e40 0x55e10ba24b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 4 client.15048 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.943+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "status"}) -- 0x55e10b9113a0 con 0x55e10ba25e40 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.944+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10ba25e40 0x55e10ba24b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.944+0000 7f1ae3e00640 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10ba25e40 0x55e10ba24b50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b8d85c2e25cd7e0 server_cookie=9591b8364d658838 in_seq=0 out_seq=0 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.944+0000 7f1ad2000640 10 client.15048 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.944+0000 7f1ad2000640 1 -- 172.21.15.146:0/2999411614 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55e10b9113a0 con 0x55e10ba25e40 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.944+0000 7f1ad2000640 10 client.15048 handle_command_reply: tid=0 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.944+0000 7f1aeddcd740 1 client.15048 shutdown 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4c50 msgr2=0x55e10ba1d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4c50 0x55e10ba1d360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0bdc10 tx=0x7f1acc0f7480 comp rx=0 tx=0).stop 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 shutdown_connections 2024-04-24T08:20:59.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e10ba25e40 0x55e10ba24b50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e10b9a9dc0 0x55e10ba1f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e10b9a4c50 0x55e10ba1d360 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 --2- 172.21.15.146:0/2999411614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e10b9a4260 0x55e10b9b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1ad2000640 0 client.15048 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 >> 172.21.15.146:0/2999411614 conn(0x55e10b99f8d0 msgr2=0x55e10b99e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 shutdown_connections 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 1 -- 172.21.15.146:0/2999411614 wait complete. 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.945+0000 7f1aeddcd740 20 client.15048 trim_cache size 0 max 16384 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:20:59.953 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 1, 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 24568, 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 129.450702514, 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 23, 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 90, 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 88, 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 140.509938401 2024-04-24T08:20:59.954 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:20:59.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1ac80a3bb0 msgr2=0x7f1ac80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1ac80a3bb0 0x7f1ac80a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55e10b9a54b0 tx=0x7f1adc005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:59.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 msgr2=0x7f1ae419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:59.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae419ee50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1acc000f80 tx=0x7f1acc0319f0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 shutdown_connections 2024-04-24T08:20:59.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f1ac80a3bb0 0x7f1ac80a6070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae410a0f0 0x7f1ae41a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ae4109730 0x7f1ae419ee50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 --2- 172.21.15.146:0/1943251821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ae4108540 0x7f1ae419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:59.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 >> 172.21.15.146:0/1943251821 conn(0x7f1ae40774d0 msgr2=0x7f1ae410c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:59.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 shutdown_connections 2024-04-24T08:20:59.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:20:59.946+0000 7f1ae9a00640 1 -- 172.21.15.146:0/1943251821 wait complete. 2024-04-24T08:20:59.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:59 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/246698663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:59.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:20:59 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2157507670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:20:59.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T08:21:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:59 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/246698663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:20:59 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2157507670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.162+0000 7fdbcfe00640 1 -- 172.21.15.146:0/49614410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd01007f0 msgr2=0x7fdbd0100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.162+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/49614410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd01007f0 0x7fdbd0100c10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009900 tx=0x7fdbc002a480 comp rx=0 tx=0).stop 2024-04-24T08:21:00.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.162+0000 7fdbcfe00640 1 -- 172.21.15.146:0/49614410 shutdown_connections 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.162+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/49614410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd0101150 0x7fdbd01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.162+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/49614410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbd0101af0 0x7fdbd0101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.163+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/49614410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd01007f0 0x7fdbd0100c10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.163+0000 7fdbcfe00640 1 -- 172.21.15.146:0/49614410 >> 172.21.15.146:0/49614410 conn(0x7fdbd00fc370 msgr2=0x7fdbd00fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.163+0000 7fdbcfe00640 1 -- 172.21.15.146:0/49614410 shutdown_connections 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.163+0000 7fdbcfe00640 1 -- 172.21.15.146:0/49614410 wait complete. 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.164+0000 7fdbcfe00640 1 Processor -- start 2024-04-24T08:21:00.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.164+0000 7fdbcfe00640 1 -- start start 2024-04-24T08:21:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd01007f0 0x7fdbd019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbd0101150 0x7fdbd019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd0101af0 0x7fdbd01a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd010fde0 con 0x7fdbd0101150 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcfe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd010fc60 con 0x7fdbd0101af0 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcfe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd010ff60 con 0x7fdbd01007f0 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd01007f0 0x7fdbd019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd01007f0 0x7fdbd019a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38936/0 (socket says 172.21.15.146:38936) 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 -- 172.21.15.146:0/3216847098 learned_addr learned my addr 172.21.15.146:0/3216847098 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbce000640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbd0101150 0x7fdbd019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcf400640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd0101af0 0x7fdbd01a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 -- 172.21.15.146:0/3216847098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd0101af0 msgr2=0x7fdbd01a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd0101af0 0x7fdbd01a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 -- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbd0101150 msgr2=0x7fdbd019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbd0101150 0x7fdbd019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.165+0000 7fdbcea00640 1 -- 172.21.15.146:0/3216847098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd01a2050 con 0x7fdbd01007f0 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.166+0000 7fdbcea00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd01007f0 0x7fdbd019a430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0031440 tx=0x7fdbc0032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.166+0000 7fdbc7400640 1 -- 172.21.15.146:0/3216847098 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc003f030 con 0x7fdbd01007f0 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.166+0000 7fdbc7400640 1 -- 172.21.15.146:0/3216847098 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc0003ce0 con 0x7fdbd01007f0 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.166+0000 7fdbc7400640 1 -- 172.21.15.146:0/3216847098 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0035070 con 0x7fdbd01007f0 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.166+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd01a22e0 con 0x7fdbd01007f0 2024-04-24T08:21:00.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.166+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd01a2780 con 0x7fdbd01007f0 2024-04-24T08:21:00.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.169+0000 7fdbd87a5740 1 Processor -- start 2024-04-24T08:21:00.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.169+0000 7fdbc7400640 1 -- 172.21.15.146:0/3216847098 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fdbc0042030 con 0x7fdbd01007f0 2024-04-24T08:21:00.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.169+0000 7fdbd87a5740 1 -- start start 2024-04-24T08:21:00.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51260 0x55e43dd51680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51c50 0x55e43dd56750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd56dc0 0x55e43dd59250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e43dcb6b10 con 0x55e43dd56dc0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e43dc5d2b0 con 0x55e43dd51260 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e43dcb2ad0 con 0x55e43dd51c50 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51260 0x55e43dd51680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbce000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd56dc0 0x55e43dd59250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51260 0x55e43dd51680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:36064/0 (socket says 172.21.15.146:36064) 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 -- 172.21.15.146:0/222001666 learned_addr learned my addr 172.21.15.146:0/222001666 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcea00640 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51c50 0x55e43dd56750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 -- 172.21.15.146:0/222001666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51c50 msgr2=0x55e43dd56750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51c50 0x55e43dd56750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 -- 172.21.15.146:0/222001666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd56dc0 msgr2=0x55e43dd59250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd56dc0 0x55e43dd59250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.170+0000 7fdbcf400640 1 -- 172.21.15.146:0/222001666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e43d9c92d0 con 0x55e43dd51260 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.171+0000 7fdbcf400640 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51260 0x55e43dd51680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009fd0 tx=0x7fdbc80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c1b262fa46408c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.171+0000 7fdbbbe00640 1 -- 172.21.15.146:0/222001666 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8004d20 con 0x55e43dd51260 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.171+0000 7fdbbbe00640 1 -- 172.21.15.146:0/222001666 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc8004050 con 0x55e43dd51260 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.171+0000 7fdbbbe00640 1 -- 172.21.15.146:0/222001666 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8005260 con 0x55e43dd51260 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.171+0000 7fdbc7400640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdbb40a3bb0 0x7fdbb40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.171+0000 7fdbcea00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdbb40a3bb0 0x7fdbb40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.171+0000 7fdbc7400640 1 -- 172.21.15.146:0/3216847098 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdbc00f08e0 con 0x7fdbd01007f0 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 -- 172.21.15.146:0/222001666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51260 msgr2=0x55e43dd51680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51260 0x55e43dd51680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009fd0 tx=0x7fdbc80118d0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 -- 172.21.15.146:0/222001666 shutdown_connections 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51c50 0x55e43dd56750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd56dc0 0x55e43dd59250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/222001666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51260 0x55e43dd51680 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 -- 172.21.15.146:0/222001666 >> 172.21.15.146:0/222001666 conn(0x55e43dd4c8d0 msgr2=0x55e43dd4ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbcea00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdbb40a3bb0 0x7fdbb40a6070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0032fd0 tx=0x7fdbc00dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 -- 172.21.15.146:0/222001666 shutdown_connections 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 -- 172.21.15.146:0/222001666 wait complete. 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.172+0000 7fdbd87a5740 1 -- start start 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 1 Processor -- start 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51260 0x55e43dd5b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51c50 0x55e43dd5bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd56dc0 0x55e43dd62c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd59d70 con 0x55e43dd51260 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e43dc70590 con 0x55e43dd51c50 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd59ef0 con 0x55e43dd56dc0 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbd87a5740 10 client.0 _ll_register_callbacks cb 0x55e43dc533a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51260 0x55e43dd5b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51260 0x55e43dd5b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39480/0 (socket says 172.21.15.146:39480) 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbcf400640 1 -- 172.21.15.146:0/3361491072 learned_addr learned my addr 172.21.15.146:0/3361491072 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.173+0000 7fdbce000640 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51c50 0x55e43dd5bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbcf400640 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd56dc0 0x55e43dd62c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbcf400640 1 -- 172.21.15.146:0/3361491072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd56dc0 msgr2=0x55e43dd62c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbcf400640 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd56dc0 0x55e43dd62c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbcf400640 1 -- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51c50 msgr2=0x55e43dd5bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbcf400640 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51c50 0x55e43dd5bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbcf400640 1 -- 172.21.15.146:0/3361491072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43da936b0 con 0x55e43dd51260 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbcf400640 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51260 0x55e43dd5b6d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8011dd0 tx=0x7fdbc8009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50866fe5121fe064 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbb8c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbb8c00640 1 -- 172.21.15.146:0/3361491072 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8005040 con 0x55e43dd51260 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbb8c00640 1 -- 172.21.15.146:0/3361491072 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdbc80051e0 con 0x55e43dd51260 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.174+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc8004050 con 0x55e43dd51260 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbb8c00640 1 -- 172.21.15.146:0/3361491072 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdbc80053e0 con 0x55e43dd51260 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 10 client.15060 fetch_fsmap learned FSMap version 23 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e43dd65fc0 con 0x55e43dd51260 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbb8c00640 1 -- 172.21.15.146:0/3361491072 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc8004050 con 0x55e43dd51260 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbb8c00640 1 -- 172.21.15.146:0/3361491072 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdbc800faa0 con 0x55e43dd51260 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 10 client.15060 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 10 client.15060 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 20 client.15060 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43dd68410 0x55e43dd69880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 4 client.15060 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.175+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e43dcc4530 con 0x55e43dd68410 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.176+0000 7fdbce000640 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43dd68410 0x55e43dd69880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.176+0000 7fdbce000640 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43dd68410 0x55e43dd69880 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=776c48e9ee9e9462 server_cookie=e71f6edc64d4a073 in_seq=0 out_seq=0 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.176+0000 7fdbb8c00640 10 client.15060 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbb8c00640 1 -- 172.21.15.146:0/3361491072 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e43dcc4530 con 0x55e43dd68410 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbb8c00640 10 client.15060 handle_command_reply: tid=0 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 client.15060 shutdown 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51260 msgr2=0x55e43dd5b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51260 0x55e43dd5b6d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8011dd0 tx=0x7fdbc8009c80 comp rx=0 tx=0).stop 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 shutdown_connections 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43dd68410 0x55e43dd69880 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd56dc0 0x55e43dd62c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd51c50 0x55e43dd5bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/3361491072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51260 0x55e43dd5b6d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbb8c00640 0 client.15060 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.177+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 >> 172.21.15.146:0/3361491072 conn(0x55e43dd4c8d0 msgr2=0x55e43dd4b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.178+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 shutdown_connections 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.178+0000 7fdbd87a5740 1 -- 172.21.15.146:0/3361491072 wait complete. 2024-04-24T08:21:00.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.178+0000 7fdbd87a5740 20 client.15060 trim_cache size 0 max 16384 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 Processor -- start 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 -- start start 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd53150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd53690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd62be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd67bc0 con 0x55e43dd51c50 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd68290 con 0x55e43dd56dc0 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd68960 con 0x55e43dd51260 2024-04-24T08:21:00.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd53150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbcf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd53150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38962/0 (socket says 172.21.15.146:38962) 2024-04-24T08:21:00.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbcf400640 1 -- 172.21.15.146:0/386993320 learned_addr learned my addr 172.21.15.146:0/386993320 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:00.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbce000640 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd53690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.198+0000 7fdbcf400640 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd62be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbcf400640 1 -- 172.21.15.146:0/386993320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 msgr2=0x55e43dd53150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbcf400640 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd53150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbcf400640 1 -- 172.21.15.146:0/386993320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 msgr2=0x55e43dd53690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbcf400640 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd53690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbcf400640 1 -- 172.21.15.146:0/386993320 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e43dd4c700 con 0x55e43dd56dc0 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbce000640 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd53690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbcf400640 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd62be0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800ff10 tx=0x7fdbc80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab302983212d490 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.199+0000 7fdbbb400640 1 -- 172.21.15.146:0/386993320 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8005040 con 0x55e43dd56dc0 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbbb400640 1 -- 172.21.15.146:0/386993320 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc802c070 con 0x55e43dd56dc0 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbbb400640 1 -- 172.21.15.146:0/386993320 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8019780 con 0x55e43dd56dc0 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 -- 172.21.15.146:0/386993320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 msgr2=0x55e43dd62be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd62be0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800ff10 tx=0x7fdbc80118d0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 -- 172.21.15.146:0/386993320 shutdown_connections 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd62be0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd53690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/386993320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd53150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 -- 172.21.15.146:0/386993320 >> 172.21.15.146:0/386993320 conn(0x55e43dd4c8d0 msgr2=0x55e43dd47800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 -- 172.21.15.146:0/386993320 shutdown_connections 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.200+0000 7fdbd87a5740 1 -- 172.21.15.146:0/386993320 wait complete. 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 -- start start 2024-04-24T08:21:00.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 Processor -- start 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd60db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd61870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd69700 con 0x55e43dd51c50 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd69030 con 0x55e43dd56dc0 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.201+0000 7fdbd87a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e43dd69dd0 con 0x55e43dd51260 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbd87a5740 10 client.0 _ll_register_callbacks cb 0x7fdbc8004eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39510/0 (socket says 172.21.15.146:39510) 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 -- 172.21.15.146:0/1429560973 learned_addr learned my addr 172.21.15.146:0/1429560973 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbcf400640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd61870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbcf400640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd60db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 -- 172.21.15.146:0/1429560973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 msgr2=0x55e43dd60db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd60db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 -- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 msgr2=0x55e43dd61870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd61870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 -- 172.21.15.146:0/1429560973 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43dd620a0 con 0x55e43dd51c50 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbcf400640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd61870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbce000640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd612f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc005d00 tx=0x7fdbbc004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f57a60ee9500f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.202+0000 7fdbb8c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbb8c00640 1 -- 172.21.15.146:0/1429560973 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc024030 con 0x55e43dd51c50 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbb8c00640 1 -- 172.21.15.146:0/1429560973 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fdbbc005020 con 0x55e43dd51c50 2024-04-24T08:21:00.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc802c070 con 0x55e43dd51c50 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbb8c00640 1 -- 172.21.15.146:0/1429560973 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdbbc005220 con 0x55e43dd51c50 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbd87a5740 10 client.15066 fetch_fsmap learned FSMap version 23 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e43ddce610 con 0x55e43dd51c50 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbb8c00640 1 -- 172.21.15.146:0/1429560973 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbc802c070 con 0x55e43dd51c50 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbb8c00640 1 -- 172.21.15.146:0/1429560973 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdbbc003680 con 0x55e43dd51c50 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.203+0000 7fdbd87a5740 10 client.15066 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbd87a5740 10 client.15066 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:21:00.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbd87a5740 20 client.15066 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43ddd3200 0x55e43ddd1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbd87a5740 4 client.15066 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "status"}) -- 0x55e43dcbe3a0 con 0x55e43ddd3200 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbcf400640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43ddd3200 0x55e43ddd1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbb8c00640 10 client.15066 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.204+0000 7fdbcf400640 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43ddd3200 0x55e43ddd1f10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a94c593f01d8e5a server_cookie=ba560fc1e6f3a067 in_seq=0 out_seq=0 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbb8c00640 1 -- 172.21.15.146:0/1429560973 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e43dcbe3a0 con 0x55e43ddd3200 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbb8c00640 10 client.15066 handle_command_reply: tid=0 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 client.15066 shutdown 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 msgr2=0x55e43dd612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd612f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc005d00 tx=0x7fdbbc004260 comp rx=0 tx=0).stop 2024-04-24T08:21:00.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 shutdown_connections 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e43ddd3200 0x55e43ddd1f10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e43dd56dc0 0x55e43dd61870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbb8c00640 0 client.15066 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e43dd51c50 0x55e43dd612f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 --2- 172.21.15.146:0/1429560973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e43dd51260 0x55e43dd60db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 >> 172.21.15.146:0/1429560973 conn(0x55e43dd4c8d0 msgr2=0x55e43dd53150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 shutdown_connections 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 1 -- 172.21.15.146:0/1429560973 wait complete. 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.205+0000 7fdbd87a5740 20 client.15066 trim_cache size 0 max 16384 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:21:00.211 INFO:teuthology.orchestra.run.smithi146.stdout: "cluster_fsid": "cca14746-0211-11ef-bc93-c7b262605968", 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "whoami": 0, 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 24569, 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "want_state": "up:active", 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "state": "up:active", 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs", 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "rank_uptime": 132.75108047800001, 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "mdsmap_epoch": 23, 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch": 90, 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_epoch_barrier": 88, 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout: "uptime": 138.00414191799999 2024-04-24T08:21:00.212 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:21:00.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdbb40a3bb0 msgr2=0x7fdbb40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdbb40a3bb0 0x7fdbb40a6070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0032fd0 tx=0x7fdbc00dd040 comp rx=0 tx=0).stop 2024-04-24T08:21:00.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd01007f0 msgr2=0x7fdbd019a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd01007f0 0x7fdbd019a430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0031440 tx=0x7fdbc0032ce0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 shutdown_connections 2024-04-24T08:21:00.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fdbb40a3bb0 0x7fdbb40a6070 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbd0101af0 0x7fdbd01a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbd0101150 0x7fdbd019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 --2- 172.21.15.146:0/3216847098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbd01007f0 0x7fdbd019a430 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 >> 172.21.15.146:0/3216847098 conn(0x7fdbd00fc370 msgr2=0x7fdbd00fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 shutdown_connections 2024-04-24T08:21:00.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.206+0000 7fdbcfe00640 1 -- 172.21.15.146:0/3216847098 wait complete. 2024-04-24T08:21:00.224 INFO:teuthology.run_tasks:Running task sequential... 2024-04-24T08:21:00.234 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-24T08:21:00.234 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-24T08:21:00.235 INFO:teuthology.task.print:Enabling mgr modules 2024-04-24T08:21:00.235 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-24T08:21:00.235 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:21:00.235 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:21:00.236 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-24T08:21:00.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 -- 172.21.15.146:0/428451659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c105c70 msgr2=0x7f244c10c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 --2- 172.21.15.146:0/428451659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c105c70 0x7f244c10c500 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2444009ca0 tx=0x7f244402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 -- 172.21.15.146:0/428451659 shutdown_connections 2024-04-24T08:21:00.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 --2- 172.21.15.146:0/428451659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c105c70 0x7f244c10c500 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 --2- 172.21.15.146:0/428451659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f244c1052b0 0x7f244c105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 --2- 172.21.15.146:0/428451659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c1040c0 0x7f244c1044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 -- 172.21.15.146:0/428451659 >> 172.21.15.146:0/428451659 conn(0x7f244c0ff670 msgr2=0x7f244c101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 -- 172.21.15.146:0/428451659 shutdown_connections 2024-04-24T08:21:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.420+0000 7f2451a00640 1 -- 172.21.15.146:0/428451659 wait complete. 2024-04-24T08:21:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.421+0000 7f2451a00640 1 Processor -- start 2024-04-24T08:21:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.421+0000 7f2451a00640 1 -- start start 2024-04-24T08:21:00.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c1040c0 0x7f244c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f244c1052b0 0x7f244c19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c105c70 0x7f244c1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f244c10fd50 con 0x7f244c105c70 2024-04-24T08:21:00.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f244c10fbd0 con 0x7f244c1052b0 2024-04-24T08:21:00.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f244c10fed0 con 0x7f244c1040c0 2024-04-24T08:21:00.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c105c70 0x7f244c1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c105c70 0x7f244c1a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39524/0 (socket says 172.21.15.146:39524) 2024-04-24T08:21:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 -- 172.21.15.146:0/2336647293 learned_addr learned my addr 172.21.15.146:0/2336647293 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f244be00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c1040c0 0x7f244c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f244b400640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f244c1052b0 0x7f244c19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 -- 172.21.15.146:0/2336647293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c1040c0 msgr2=0x7f244c19a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c1040c0 0x7f244c19a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 -- 172.21.15.146:0/2336647293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f244c1052b0 msgr2=0x7f244c19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f244c1052b0 0x7f244c19a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f2451000640 1 -- 172.21.15.146:0/2336647293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c1a1f90 con 0x7f244c105c70 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f244b400640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f244c1052b0 0x7f244c19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.422+0000 7f244be00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c1040c0 0x7f244c19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.423+0000 7f2451000640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c105c70 0x7f244c1a1920 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2444031ae0 tx=0x7f2444031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.423+0000 7f2448c00640 1 -- 172.21.15.146:0/2336647293 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444003c30 con 0x7f244c105c70 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.423+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c1a2220 con 0x7f244c105c70 2024-04-24T08:21:00.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.423+0000 7f2448c00640 1 -- 172.21.15.146:0/2336647293 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2444003dd0 con 0x7f244c105c70 2024-04-24T08:21:00.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.423+0000 7f2448c00640 1 -- 172.21.15.146:0/2336647293 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244402fcb0 con 0x7f244c105c70 2024-04-24T08:21:00.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.423+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c1a2770 con 0x7f244c105c70 2024-04-24T08:21:00.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.424+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f244c10b380 con 0x7f244c105c70 2024-04-24T08:21:00.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.426+0000 7f2448c00640 1 -- 172.21.15.146:0/2336647293 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f2444004050 con 0x7f244c105c70 2024-04-24T08:21:00.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.428+0000 7f2448c00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f24300a3ba0 0x7f24300a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.428+0000 7f2448c00640 1 -- 172.21.15.146:0/2336647293 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f24440f1640 con 0x7f244c105c70 2024-04-24T08:21:00.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.428+0000 7f244be00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f24300a3ba0 0x7f24300a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.428+0000 7f244be00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f24300a3ba0 0x7f24300a6060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f243c005fd0 tx=0x7f243c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.433+0000 7f2448c00640 1 -- 172.21.15.146:0/2336647293 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24440bdfe0 con 0x7f244c105c70 2024-04-24T08:21:00.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:00.695+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f244c062c00 con 0x7f244c105c70 2024-04-24T08:21:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:00 smithi153 ceph-mon[32746]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:00.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:00 smithi146 ceph-mon[27805]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:00 smithi177 ceph-mon[33046]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:01.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.650+0000 7f2448c00640 1 -- 172.21.15.146:0/2336647293 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f24440bd8e0 con 0x7f244c105c70 2024-04-24T08:21:01.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.654+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f24300a3ba0 msgr2=0x7f24300a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.654+0000 7f2451a00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f24300a3ba0 0x7f24300a6060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f243c005fd0 tx=0x7f243c00b410 comp rx=0 tx=0).stop 2024-04-24T08:21:01.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.654+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c105c70 msgr2=0x7f244c1a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.654+0000 7f2451a00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c105c70 0x7f244c1a1920 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2444031ae0 tx=0x7f2444031ee0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.655+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 shutdown_connections 2024-04-24T08:21:01.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.655+0000 7f2451a00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f24300a3ba0 0x7f24300a6060 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.655+0000 7f2451a00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f244c105c70 0x7f244c1a1920 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.655+0000 7f2451a00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f244c1052b0 0x7f244c19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.655+0000 7f2451a00640 1 --2- 172.21.15.146:0/2336647293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f244c1040c0 0x7f244c19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.655+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 >> 172.21.15.146:0/2336647293 conn(0x7f244c0ff670 msgr2=0x7f244c108550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.658+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 shutdown_connections 2024-04-24T08:21:01.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.658+0000 7f2451a00640 1 -- 172.21.15.146:0/2336647293 wait complete. 2024-04-24T08:21:01.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-24T08:21:01.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 -- 172.21.15.146:0/4176932649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc105e30 msgr2=0x7f9ebc10c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/4176932649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc105e30 0x7f9ebc10c6c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009be0 tx=0x7f9eb402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 -- 172.21.15.146:0/4176932649 shutdown_connections 2024-04-24T08:21:01.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/4176932649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc105e30 0x7f9ebc10c6c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/4176932649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ebc105470 0x7f9ebc1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/4176932649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc104280 0x7f9ebc1046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 -- 172.21.15.146:0/4176932649 >> 172.21.15.146:0/4176932649 conn(0x7f9ebc0ff7f0 msgr2=0x7f9ebc101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 -- 172.21.15.146:0/4176932649 shutdown_connections 2024-04-24T08:21:01.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.863+0000 7f9ec0c00640 1 -- 172.21.15.146:0/4176932649 wait complete. 2024-04-24T08:21:01.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.864+0000 7f9ec0c00640 1 Processor -- start 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.864+0000 7f9ec0c00640 1 -- start start 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9ec0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc104280 0x7f9ebc19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9ec0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ebc105470 0x7f9ebc19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9ec0c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc105e30 0x7f9ebc1a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9ec0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc10ff10 con 0x7f9ebc105470 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9ec0c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc10fd90 con 0x7f9ebc105e30 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9ec0c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ebc110090 con 0x7f9ebc104280 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc104280 0x7f9ebc19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eba800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc105e30 0x7f9ebc1a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ebc105470 0x7f9ebc19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc104280 0x7f9ebc19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38746/0 (socket says 172.21.15.146:38746) 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 -- 172.21.15.146:0/3699985675 learned_addr learned my addr 172.21.15.146:0/3699985675 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:01.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 -- 172.21.15.146:0/3699985675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc105e30 msgr2=0x7f9ebc1a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc105e30 0x7f9ebc1a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 -- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ebc105470 msgr2=0x7f9ebc19a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ebc105470 0x7f9ebc19a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.865+0000 7f9eb9e00640 1 -- 172.21.15.146:0/3699985675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ebc1a2040 con 0x7f9ebc104280 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.866+0000 7f9eb9e00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc104280 0x7f9ebc19a420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00e9f0 tx=0x7f9eac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.866+0000 7f9eb2a00640 1 -- 172.21.15.146:0/3699985675 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac00ccc0 con 0x7f9ebc104280 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.866+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ebc1a2330 con 0x7f9ebc104280 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.866+0000 7f9eb2a00640 1 -- 172.21.15.146:0/3699985675 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac0035f0 con 0x7f9ebc104280 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.866+0000 7f9eb2a00640 1 -- 172.21.15.146:0/3699985675 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac00ce60 con 0x7f9ebc104280 2024-04-24T08:21:01.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.866+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ebc1a2830 con 0x7f9ebc104280 2024-04-24T08:21:01.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.867+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ebc10b540 con 0x7f9ebc104280 2024-04-24T08:21:01.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.869+0000 7f9eb2a00640 1 -- 172.21.15.146:0/3699985675 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f9eac003b10 con 0x7f9ebc104280 2024-04-24T08:21:01.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.870+0000 7f9eb2a00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 0x7f9ea00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.871+0000 7f9eb9400640 1 -- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 msgr2=0x7f9ea00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:01.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.871+0000 7f9eb2a00640 1 -- 172.21.15.146:0/3699985675 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f9eac0d08c0 con 0x7f9ebc104280 2024-04-24T08:21:01.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.871+0000 7f9eb9400640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 0x7f9ea00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:21:01.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:01.877+0000 7f9eb2a00640 1 -- 172.21.15.146:0/3699985675 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9eac09d260 con 0x7f9ebc104280 2024-04-24T08:21:01.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:01 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2336647293' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-24T08:21:01.961 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: ignoring --setuser ceph since I am not root 2024-04-24T08:21:01.961 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: ignoring --setgroup ceph since I am not root 2024-04-24T08:21:01.961 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:01 smithi153 ceph-mgr[34133]: -- 172.21.15.153:0/1719278471 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56396e71e5a0 con 0x56396e6f2800 2024-04-24T08:21:01.961 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:01.836+0000 7f4b9abe3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:21:01.983 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:01 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2336647293' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-24T08:21:01.984 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: ignoring --setuser ceph since I am not root 2024-04-24T08:21:01.984 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: ignoring --setgroup ceph since I am not root 2024-04-24T08:21:02.024 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:01 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2336647293' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-24T08:21:02.024 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: ignoring --setuser ceph since I am not root 2024-04-24T08:21:02.024 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: ignoring --setgroup ceph since I am not root 2024-04-24T08:21:02.024 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:01 smithi177 ceph-mgr[34910]: -- 172.21.15.177:0/2198019353 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b9c6a605a0 con 0x55b9c6a34c00 2024-04-24T08:21:02.024 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:01.835+0000 7ff81c3b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:21:02.024 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:02.023+0000 7ff81c3b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:21:02.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.072+0000 7f9eb9400640 1 -- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 msgr2=0x7f9ea00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:02.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.072+0000 7f9eb9400640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 0x7f9ea00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:21:02.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.083+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f9ebc02d620 con 0x7f9ebc104280 2024-04-24T08:21:02.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.092+0000 7f9eb2a00640 1 -- 172.21.15.146:0/3699985675 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f9eac09cb60 con 0x7f9ebc104280 2024-04-24T08:21:02.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 msgr2=0x7f9ea00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:21:02.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 0x7f9ea00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc104280 msgr2=0x7f9ebc19a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc104280 0x7f9ebc19a420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00e9f0 tx=0x7f9eac00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 shutdown_connections 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f9ea00a3ca0 0x7f9ea00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ebc105e30 0x7f9ebc1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ebc105470 0x7f9ebc19a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 --2- 172.21.15.146:0/3699985675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ebc104280 0x7f9ebc19a420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 >> 172.21.15.146:0/3699985675 conn(0x7f9ebc0ff7f0 msgr2=0x7f9ebc108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 shutdown_connections 2024-04-24T08:21:02.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.095+0000 7f9ec0c00640 1 -- 172.21.15.146:0/3699985675 wait complete. 2024-04-24T08:21:02.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-24T08:21:02.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:01.982+0000 7f16c2dfc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:21:02.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 -- 172.21.15.146:0/3983781177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c076040 msgr2=0x7fcd8c074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 --2- 172.21.15.146:0/3983781177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c076040 0x7fcd8c074490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c009900 tx=0x7fcd7c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 -- 172.21.15.146:0/3983781177 shutdown_connections 2024-04-24T08:21:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 --2- 172.21.15.146:0/3983781177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c109d50 0x7fcd8c10c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 --2- 172.21.15.146:0/3983781177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c074a60 0x7fcd8c074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 --2- 172.21.15.146:0/3983781177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c076040 0x7fcd8c074490 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 -- 172.21.15.146:0/3983781177 >> 172.21.15.146:0/3983781177 conn(0x7fcd8c0fd400 msgr2=0x7fcd8c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 -- 172.21.15.146:0/3983781177 shutdown_connections 2024-04-24T08:21:02.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.296+0000 7fcd91400640 1 -- 172.21.15.146:0/3983781177 wait complete. 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 Processor -- start 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 -- start start 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c074a60 0x7fcd8c19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c076040 0x7fcd8c19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c109d50 0x7fcd8c1a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c10fd20 con 0x7fcd8c076040 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c10fba0 con 0x7fcd8c109d50 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd91400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c10fea0 con 0x7fcd8c074a60 2024-04-24T08:21:02.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c074a60 0x7fcd8c19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c074a60 0x7fcd8c19a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38770/0 (socket says 172.21.15.146:38770) 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd8be00640 1 -- 172.21.15.146:0/3724444944 learned_addr learned my addr 172.21.15.146:0/3724444944 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd83e00640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c076040 0x7fcd8c19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.297+0000 7fcd90a00640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c109d50 0x7fcd8c1a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd8be00640 1 -- 172.21.15.146:0/3724444944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c109d50 msgr2=0x7fcd8c1a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd8be00640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c109d50 0x7fcd8c1a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd8be00640 1 -- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c076040 msgr2=0x7fcd8c19a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd8be00640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c076040 0x7fcd8c19a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd8be00640 1 -- 172.21.15.146:0/3724444944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8c1a2000 con 0x7fcd8c074a60 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd8be00640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c074a60 0x7fcd8c19a3e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c031440 tx=0x7fcd7c032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd89600640 1 -- 172.21.15.146:0/3724444944 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c03f030 con 0x7fcd8c074a60 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd89600640 1 -- 172.21.15.146:0/3724444944 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd7c003ce0 con 0x7fcd8c074a60 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8c1a2290 con 0x7fcd8c074a60 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd89600640 1 -- 172.21.15.146:0/3724444944 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c035070 con 0x7fcd8c074a60 2024-04-24T08:21:02.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.298+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd8c1a2780 con 0x7fcd8c074a60 2024-04-24T08:21:02.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.299+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd8c108a80 con 0x7fcd8c074a60 2024-04-24T08:21:02.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.301+0000 7fcd89600640 1 -- 172.21.15.146:0/3724444944 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7fcd7c042030 con 0x7fcd8c074a60 2024-04-24T08:21:02.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.302+0000 7fcd89600640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 0x7fcd700a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.303+0000 7fcd83e00640 1 -- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 msgr2=0x7fcd700a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:02.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.303+0000 7fcd89600640 1 -- 172.21.15.146:0/3724444944 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fcd7c0f0920 con 0x7fcd8c074a60 2024-04-24T08:21:02.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.303+0000 7fcd83e00640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 0x7fcd700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:21:02.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.307+0000 7fcd89600640 1 -- 172.21.15.146:0/3724444944 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd7c0bd2c0 con 0x7fcd8c074a60 2024-04-24T08:21:02.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:02.128+0000 7ff81c3b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:21:02.438 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:02.021+0000 7f4b9abe3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:21:02.438 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:02.125+0000 7f4b9abe3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:21:02.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.504+0000 7fcd83e00640 1 -- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 msgr2=0x7fcd700a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:02.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.504+0000 7fcd83e00640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 0x7fcd700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:21:02.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.511+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fcd8c02d620 con 0x7fcd8c074a60 2024-04-24T08:21:02.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.520+0000 7fcd89600640 1 -- 172.21.15.146:0/3724444944 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7fcd7c0bcbc0 con 0x7fcd8c074a60 2024-04-24T08:21:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 msgr2=0x7fcd700a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:21:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 0x7fcd700a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c074a60 msgr2=0x7fcd8c19a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c074a60 0x7fcd8c19a3e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c031440 tx=0x7fcd7c032ce0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 shutdown_connections 2024-04-24T08:21:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fcd700a3bf0 0x7fcd700a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd8c109d50 0x7fcd8c1a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c076040 0x7fcd8c19a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 --2- 172.21.15.146:0/3724444944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd8c074a60 0x7fcd8c19a3e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 >> 172.21.15.146:0/3724444944 conn(0x7fcd8c0fd400 msgr2=0x7fcd8c105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.522+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 shutdown_connections 2024-04-24T08:21:02.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:02.523+0000 7fcd91400640 1 -- 172.21.15.146:0/3724444944 wait complete. 2024-04-24T08:21:02.537 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:02.241+0000 7f16c2dfc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:21:02.537 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:02.379+0000 7f16c2dfc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:21:02.541 INFO:teuthology.run_tasks:Running task kclient... 2024-04-24T08:21:02.566 INFO:tasks.kclient:Mounting kernel clients... 2024-04-24T08:21:02.566 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-24T08:21:02.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:02.567 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-24T08:21:02.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:02.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-04-24T08:21:02.702 INFO:teuthology.orchestra.run.smithi146.stderr:Cannot find device "ceph-brx" 2024-04-24T08:21:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:21:02.707 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-24T08:21:02.707 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T08:21:02.707 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:02.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-24T08:21:02.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2336647293' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-24T08:21:02.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-mon[32746]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-24T08:21:02.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' 2024-04-24T08:21:02.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' 2024-04-24T08:21:02.711 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:02.437+0000 7f4b9abe3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:21:02.711 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:02.527+0000 7f4b9abe3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:21:02.801 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2336647293' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-24T08:21:02.801 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-mon[27805]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-24T08:21:02.801 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' 2024-04-24T08:21:02.801 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' 2024-04-24T08:21:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2336647293' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-24T08:21:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-mon[33046]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-24T08:21:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' 2024-04-24T08:21:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' 2024-04-24T08:21:02.815 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:02.438+0000 7ff81c3b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:21:02.815 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:02.525+0000 7ff81c3b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:21:03.143 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:21:03.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:02.798+0000 7f16c2dfc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:21:03.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:02.914+0000 7f16c2dfc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:21:04.061 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:03 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:03.719+0000 7ff81c3b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:21:04.061 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:03 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:03.889+0000 7ff81c3b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:21:04.066 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:03.722+0000 7f4b9abe3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:21:04.066 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:03.894+0000 7f4b9abe3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:21:04.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:04.059+0000 7ff81c3b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:21:04.358 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:04.066+0000 7f4b9abe3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:21:04.646 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:04.347+0000 7ff81c3b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:21:04.646 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:04.440+0000 7ff81c3b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:21:04.646 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:04.644+0000 7ff81c3b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:21:04.656 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:04.357+0000 7f4b9abe3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:21:04.656 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:04.449+0000 7f4b9abe3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:21:04.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:04.505+0000 7f16c2dfc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:21:04.960 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:04.655+0000 7f4b9abe3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:21:04.960 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:04.851+0000 7f4b9abe3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:21:05.012 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:04.741+0000 7f16c2dfc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:21:05.013 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:04.964+0000 7f16c2dfc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:21:05.065 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:04.839+0000 7ff81c3b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 -- 172.21.15.146:0/556327502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8108b90 msgr2=0x7fbbf8108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 --2- 172.21.15.146:0/556327502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8108b90 0x7fbbf8108f70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009990 tx=0x7fbbe8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 -- 172.21.15.146:0/556327502 shutdown_connections 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 --2- 172.21.15.146:0/556327502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8103500 0x7fbbf810f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 --2- 172.21.15.146:0/556327502 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8108b90 0x7fbbf8108f70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 --2- 172.21.15.146:0/556327502 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbf8102b40 0x7fbbf8102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 -- 172.21.15.146:0/556327502 >> 172.21.15.146:0/556327502 conn(0x7fbbf8079c50 msgr2=0x7fbbf807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 -- 172.21.15.146:0/556327502 shutdown_connections 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.271+0000 7fbbfd800700 1 -- 172.21.15.146:0/556327502 wait complete. 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.273+0000 7fbbfd800700 1 Processor -- start 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.273+0000 7fbbfd800700 1 -- start start 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.273+0000 7fbbfd800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8102b40 0x7fbbf819b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.273+0000 7fbbfd800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8103500 0x7fbbf819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbfd800700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbf8108b90 0x7fbbf819fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbfd800700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8112e60 con 0x7fbbf8108b90 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbfd800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8112ce0 con 0x7fbbf8102b40 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbfd800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8112fe0 con 0x7fbbf8103500 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8102b40 0x7fbbf819b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8102b40 0x7fbbf819b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43256/0 (socket says 172.21.15.146:43256) 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 -- 172.21.15.146:0/1849060871 learned_addr learned my addr 172.21.15.146:0/1849060871 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6000700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8103500 0x7fbbf819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf7400700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbf8108b90 0x7fbbf819fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 -- 172.21.15.146:0/1849060871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8103500 msgr2=0x7fbbf819ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8103500 0x7fbbf819ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 -- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbf8108b90 msgr2=0x7fbbf819fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbf8108b90 0x7fbbf819fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6a00700 1 -- 172.21.15.146:0/1849060871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf81a0540 con 0x7fbbf8102b40 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.274+0000 7fbbf6000700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8103500 0x7fbbf819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.275+0000 7fbbf6a00700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8102b40 0x7fbbf819b530 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe800bb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.275+0000 7fbbef400700 1 -- 172.21.15.146:0/1849060871 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8027070 con 0x7fbbf8102b40 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.275+0000 7fbbef400700 1 -- 172.21.15.146:0/1849060871 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbe8003d70 con 0x7fbbf8102b40 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.275+0000 7fbbef400700 1 -- 172.21.15.146:0/1849060871 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe800f460 con 0x7fbbf8102b40 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.275+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf81a07d0 con 0x7fbbf8102b40 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.275+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf81a80b0 con 0x7fbbf8102b40 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.276+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf81041f0 con 0x7fbbf8102b40 2024-04-24T08:21:05.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.278+0000 7fbbef400700 1 -- 172.21.15.146:0/1849060871 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7fbbe802a070 con 0x7fbbf8102b40 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.279+0000 7fbbef400700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 0x7fbbdc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.279+0000 7fbbf6000700 1 -- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 msgr2=0x7fbbdc0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.279+0000 7fbbef400700 1 -- 172.21.15.146:0/1849060871 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7fbbe80d11d0 con 0x7fbbf8102b40 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.279+0000 7fbbf6000700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 0x7fbbdc0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.283+0000 7fbbef400700 1 -- 172.21.15.146:0/1849060871 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbbe8013070 con 0x7fbbf8102b40 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.480+0000 7fbbf6000700 1 -- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 msgr2=0x7fbbdc0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.480+0000 7fbbf6000700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 0x7fbbdc0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.555+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbbf81a8280 con 0x7fbbf8102b40 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.556+0000 7fbbef400700 1 -- 172.21.15.146:0/1849060871 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7fbbe809dc00 con 0x7fbbf8102b40 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi146.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:21:05.558 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:05.365+0000 7f16c2dfc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:21:05.558 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:05.492+0000 7f16c2dfc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 msgr2=0x7fbbdc0a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 0x7fbbdc0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8102b40 msgr2=0x7fbbf819b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8102b40 0x7fbbf819b530 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe800bb60 comp rx=0 tx=0).stop 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 shutdown_connections 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7fbbdc0a3d10 0x7fbbdc0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbf8108b90 0x7fbbf819fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbbf8102b40 0x7fbbf819b530 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 --2- 172.21.15.146:0/1849060871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8103500 0x7fbbf819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 >> 172.21.15.146:0/1849060871 conn(0x7fbbf8079c50 msgr2=0x7fbbf8074d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 shutdown_connections 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:05.559+0000 7fbbfd800700 1 -- 172.21.15.146:0/1849060871 wait complete. 2024-04-24T08:21:05.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:05 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1849060871' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T08:21:05.711 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:05.454+0000 7f4b9abe3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:21:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:05 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1849060871' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T08:21:05.815 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:05.436+0000 7ff81c3b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:21:05.921 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:05.761+0000 7f16c2dfc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:21:05.921 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:05 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1849060871' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T08:21:06.155 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:05.857+0000 7ff81c3b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:21:06.155 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:05.953+0000 7ff81c3b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:21:06.184 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:05.882+0000 7f4b9abe3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:21:06.184 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:05.980+0000 7f4b9abe3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:21:06.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:06.023+0000 7f16c2dfc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:21:06.316 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:06.317 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 41955sec preferred_lft 41955sec 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:06.360 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T08:21:06.360 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:21:06.360 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-24T08:21:06.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-04-24T08:21:06.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-04-24T08:21:06.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-04-24T08:21:06.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T08:21:06.361 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-24T08:21:06.461 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:06.183+0000 7f4b9abe3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:21:06.565 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:06.154+0000 7ff81c3b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:06.860 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:06.926 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:21:06.984 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:21:06.984 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-24T08:21:06.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:21:06.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:21:06.984 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:21:06.984 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-24T08:21:07.013 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:06.678+0000 7ff81c3b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:21:07.013 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:06.770+0000 7ff81c3b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:21:07.050 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:06.713+0000 7f4b9abe3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:21:07.051 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:06.807+0000 7f4b9abe3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:21:07.101 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:06.891+0000 7f16c2dfc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:21:07.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:07.012+0000 7ff81c3b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:21:07.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:07.144+0000 7ff81c3b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:21:07.315 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:07.239+0000 7ff81c3b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:21:07.398 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:07.049+0000 7f4b9abe3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:21:07.398 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:07.183+0000 7f4b9abe3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:21:07.398 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:07.280+0000 7f4b9abe3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:21:07.627 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:07.486+0000 7f16c2dfc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:21:07.660 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:07.354+0000 7ff81c3b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:21:07.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:07.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:07.691 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-24T08:21:07.703 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:07.397+0000 7f4b9abe3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:21:07.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:07.732 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-04-24T08:21:07.814 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:21:07.814 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-24T08:21:07.814 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:21:07.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-24T08:21:07.815 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-24T08:21:07.917 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:07.625+0000 7f16c2dfc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:21:07.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:21:07.960 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:07.702+0000 7f4b9abe3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:21:08.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:08.023 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-24T08:21:08.023 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:21:08.023 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-24T08:21:08.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-24T08:21:08.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T08:21:08.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-24T08:21:08.023 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-24T08:21:08.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-24T08:21:08.024 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-24T08:21:08.065 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:07.658+0000 7ff81c3b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:21:08.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:21:08.186 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:07.915+0000 7f16c2dfc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:21:08.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:08.295 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:21:08.295 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-04-24T08:21:08.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-04-24T08:21:08.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T08:21:08.295 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-04-24T08:21:08.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:21:08.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:08.478 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:21:08.478 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:08.519 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:21:08.636 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:21:08.636 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:08.637 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:08.256+0000 7ff81c3b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:21:08.637 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z[34906]: 2024-04-24T08:21:08.473+0000 7ff81c3b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:21:08.678 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-24T08:21:08.678 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-24T08:21:08.678 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/dbf0f2a2-34a9-43a0-b34c-6ccd54f4bf23 2024-04-24T08:21:08.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:08.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/dbf0f2a2-34a9-43a0-b34c-6ccd54f4bf23 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-24T08:21:08.710 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:08.312+0000 7f4b9abe3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:21:08.711 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y[34129]: 2024-04-24T08:21:08.531+0000 7f4b9abe3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:21:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-mon[32746]: Standby manager daemon z restarted 2024-04-24T08:21:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-mon[32746]: Standby manager daemon z started 2024-04-24T08:21:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:21:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:21:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:21:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:08 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:21:08.726 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-mon[27805]: Standby manager daemon z restarted 2024-04-24T08:21:08.726 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-mon[27805]: Standby manager daemon z started 2024-04-24T08:21:08.726 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:21:08.727 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:21:08.727 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:21:08.727 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:21:08.727 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:08.618+0000 7f16c2dfc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:21:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-mon[33046]: Standby manager daemon z restarted 2024-04-24T08:21:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-mon[33046]: Standby manager daemon z started 2024-04-24T08:21:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:21:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:21:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:21:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:08 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.177:0/268964220' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:21:08.833 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-24T08:21:08.833 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-24T08:21:08.987 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:08 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:08.754+0000 7f16c2dfc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:21:09.282 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:09.094+0000 7f16c2dfc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:21:09.554 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:09.279+0000 7f16c2dfc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:21:09.554 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:09.412+0000 7f16c2dfc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:21:09.672 INFO:teuthology.orchestra.run.smithi146.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-24T08:21:09.672 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-24T08:21:09.672 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: resolved to: "172.21.15.146:3300,172.21.15.153:3300,172.21.15.177:3300" 2024-04-24T08:21:09.672 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: trying mount with new device syntax: 0@cca14746-0211-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/dbf0f2a2-34a9-43a0-b34c-6ccd54f4bf23 2024-04-24T08:21:09.672 INFO:teuthology.orchestra.run.smithi146.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.146:3300/172.21.15.153:3300/172.21.15.177:3300" will pass to kernel 2024-04-24T08:21:09.675 INFO:teuthology.orchestra.run.smithi146.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-24T08:21:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: You just mounted a file system that supports labels which does not 2024-04-24T08:21:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-24T08:21:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: applications will generate AVC messages and not be allowed access to 2024-04-24T08:21:09.675 INFO:teuthology.orchestra.run.smithi146.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-24T08:21:09.678 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-24T08:21:09.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:09.679 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:09.746 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T08:21:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:09 smithi177 ceph-mon[33046]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-04-24T08:21:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:09 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:21:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:09 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:21:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:09 smithi177 ceph-mon[33046]: Standby manager daemon y restarted 2024-04-24T08:21:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:09 smithi177 ceph-mon[33046]: Standby manager daemon y started 2024-04-24T08:21:09.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:09 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:21:09.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:09 smithi177 ceph-mon[33046]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:21:09.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T08:21:09.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:09.863 INFO:teuthology.orchestra.run.smithi146.stdout:client25220 2024-04-24T08:21:09.863 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:21:09.863 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:09.913 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T08:21:09.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:09 smithi153 ceph-mon[32746]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-04-24T08:21:09.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:09 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:21:09.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:09 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:21:09.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:09 smithi153 ceph-mon[32746]: Standby manager daemon y restarted 2024-04-24T08:21:09.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:09 smithi153 ceph-mon[32746]: Standby manager daemon y started 2024-04-24T08:21:09.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:09 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:21:09.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:09 smithi153 ceph-mon[32746]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:21:09.977 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:09.575+0000 7f16c2dfc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:09.975+0000 7f16c2dfc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-mon[27805]: mgrmap e20: x(active, since 10m), standbys: y, z 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-mon[27805]: Standby manager daemon y restarted 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-mon[27805]: Standby manager daemon y started 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:21:09.978 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:09 smithi146 ceph-mon[27805]: from='mgr.? 172.21.15.153:0/3101964075' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:21:10.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T08:21:10.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:10.025 INFO:teuthology.orchestra.run.smithi146.stdout:client25220 2024-04-24T08:21:10.025 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T08:21:10.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T08:21:10.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:10.143 INFO:teuthology.orchestra.run.smithi146.stdout:client25220 2024-04-24T08:21:10.143 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/cca14746-0211-11ef-bc93-c7b262605968.client25220/status') 2024-04-24T08:21:10.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:21:10.271 INFO:teuthology.orchestra.run.smithi146.stdout:instance: client.25220 (3)172.21.15.146:0/4063491484 2024-04-24T08:21:10.271 INFO:teuthology.orchestra.run.smithi146.stdout:blocklisted: false 2024-04-24T08:21:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records in 2024-04-24T08:21:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:0+1 records out 2024-04-24T08:21:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:72 bytes copied, 8.704e-05 s, 827 kB/s 2024-04-24T08:21:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:10.284 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T08:21:10.294 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:21:10.294 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:21:10.295 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-24T08:21:10.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.478+0000 7f646a600640 1 -- 172.21.15.146:0/408139906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 msgr2=0x7f6464074430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.478+0000 7f646a600640 1 --2- 172.21.15.146:0/408139906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 0x7f6464074430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f645c009900 tx=0x7f645c02a480 comp rx=0 tx=0).stop 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.479+0000 7f646a600640 1 -- 172.21.15.146:0/408139906 shutdown_connections 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.479+0000 7f646a600640 1 --2- 172.21.15.146:0/408139906 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6464109db0 0x7f646410c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.479+0000 7f646a600640 1 --2- 172.21.15.146:0/408139906 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464074a00 0x7f6464074e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.479+0000 7f646a600640 1 --2- 172.21.15.146:0/408139906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 0x7f6464074430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.479+0000 7f646a600640 1 -- 172.21.15.146:0/408139906 >> 172.21.15.146:0/408139906 conn(0x7f64640fd490 msgr2=0x7f64640ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.479+0000 7f646a600640 1 -- 172.21.15.146:0/408139906 shutdown_connections 2024-04-24T08:21:10.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.479+0000 7f646a600640 1 -- 172.21.15.146:0/408139906 wait complete. 2024-04-24T08:21:10.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 Processor -- start 2024-04-24T08:21:10.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 -- start start 2024-04-24T08:21:10.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464074a00 0x7f646419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 0x7f646419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6464109db0 0x7f64641a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f646410fd90 con 0x7f6464074a00 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f646410fc10 con 0x7f6464109db0 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f646a600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f646410ff10 con 0x7f6464075fe0 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f6469200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464074a00 0x7f646419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f6463e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 0x7f646419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f6463e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 0x7f646419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38796/0 (socket says 172.21.15.146:38796) 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f6463e00640 1 -- 172.21.15.146:0/1422858093 learned_addr learned my addr 172.21.15.146:0/1422858093 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.480+0000 7f6469c00640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6464109db0 0x7f64641a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6469200640 1 -- 172.21.15.146:0/1422858093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 msgr2=0x7f646419a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6469200640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 0x7f646419a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6469200640 1 -- 172.21.15.146:0/1422858093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6464109db0 msgr2=0x7f64641a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6469200640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6464109db0 0x7f64641a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6469200640 1 -- 172.21.15.146:0/1422858093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64641a1f80 con 0x7f6464074a00 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6469c00640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6464109db0 0x7f64641a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6469200640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464074a00 0x7f646419a360 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f645c02a9a0 tx=0x7f645c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.481+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c03f030 con 0x7f6464074a00 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.482+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f645c003ce0 con 0x7f6464074a00 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.482+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c035070 con 0x7f6464074a00 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.482+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64641a2210 con 0x7f6464074a00 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.482+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64641a2680 con 0x7f6464074a00 2024-04-24T08:21:10.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.483+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6464108b70 con 0x7f6464074a00 2024-04-24T08:21:10.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.485+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 21) ==== 138664+0+0 (secure 0 0 0) 0x7f645c042030 con 0x7f6464074a00 2024-04-24T08:21:10.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.487+0000 7f6461600640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 0x7f64480a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.487+0000 7f6463e00640 1 -- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 msgr2=0x7f64480a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:10.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.487+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8240+0+0 (secure 0 0 0) 0x7f645c0f0870 con 0x7f6464074a00 2024-04-24T08:21:10.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.487+0000 7f6463e00640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 0x7f64480a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:21:10.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.492+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f645c0bcbc0 con 0x7f6464074a00 2024-04-24T08:21:10.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.688+0000 7f6463e00640 1 -- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 msgr2=0x7f64480a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:10.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.688+0000 7f6463e00640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 0x7f64480a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:21:10.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:10.739+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f646402d880 con 0x7f6464074a00 2024-04-24T08:21:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:10 smithi177 ceph-mon[33046]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-04-24T08:21:10.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:10 smithi153 ceph-mon[32746]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-04-24T08:21:10.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:10 smithi146 ceph-mon[27805]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-04-24T08:21:10.992 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:10 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:10.783+0000 7f16c2dfc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:21:11.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.089+0000 7f6463e00640 1 -- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 msgr2=0x7f64480a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.146:6800/1475219332 2024-04-24T08:21:11.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.089+0000 7f6463e00640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 0x7f64480a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T08:21:11.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.094+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f645c03bd00 con 0x7f6464074a00 2024-04-24T08:21:11.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.094+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 msgr2=0x7f64480a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:21:11.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.094+0000 7f6461600640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 0x7f64480a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.373 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.073+0000 7f16c2dfc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:21:11.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.574+0000 7f6461600640 1 -- 172.21.15.146:0/1422858093 <== mon.0 v2:172.21.15.146:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f645c0b3c90 con 0x7f6464074a00 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464074a00 msgr2=0x7f646419a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464074a00 0x7f646419a360 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f645c02a9a0 tx=0x7f645c032ce0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 shutdown_connections 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6464075fe0 0x7f646419a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:6800/1475219332,v1:172.21.15.146:6801/1475219332] conn(0x7f64480a3c70 0x7f64480a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6464109db0 0x7f64641a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 --2- 172.21.15.146:0/1422858093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464074a00 0x7f646419a360 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 >> 172.21.15.146:0/1422858093 conn(0x7f64640fd490 msgr2=0x7f6464105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 shutdown_connections 2024-04-24T08:21:11.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.579+0000 7f646a600640 1 -- 172.21.15.146:0/1422858093 wait complete. 2024-04-24T08:21:11.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-24T08:21:11.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.399+0000 7f1614200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.399+0000 7f1614200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.399+0000 7f1614200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.399+0000 7f1614200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.399+0000 7f1614200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.402+0000 7f1619200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.402+0000 7f1619200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.402+0000 7f1619200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.402+0000 7f1619200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.660 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:11.402+0000 7f1619200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1422858093' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: Active manager daemon x restarted 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: Activating manager daemon x 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: mgrmap e22: x(active, starting, since 0.0150582s), standbys: y, z 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:21:11.661 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:21:11.662 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:21:11.663 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:21:11.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: Manager daemon x is now available 2024-04-24T08:21:11.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:11.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:21:11.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:21:11.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:21:11.664 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:11 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 -- 172.21.15.146:0/1790076405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 msgr2=0x7f6610074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 --2- 172.21.15.146:0/1790076405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f6610074490 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6600009930 tx=0x7f660002a420 comp rx=0 tx=0).stop 2024-04-24T08:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 -- 172.21.15.146:0/1790076405 shutdown_connections 2024-04-24T08:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 --2- 172.21.15.146:0/1790076405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f661010e460 0x7f6610110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 --2- 172.21.15.146:0/1790076405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6610074a60 0x7f6610074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 --2- 172.21.15.146:0/1790076405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f6610074490 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 -- 172.21.15.146:0/1790076405 >> 172.21.15.146:0/1790076405 conn(0x7f66100fd400 msgr2=0x7f66100ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 -- 172.21.15.146:0/1790076405 shutdown_connections 2024-04-24T08:21:11.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.776+0000 7f660f200640 1 -- 172.21.15.146:0/1790076405 wait complete. 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.777+0000 7f660f200640 1 Processor -- start 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.777+0000 7f660f200640 1 -- start start 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6610074a60 0x7f661019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f661019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f661010e460 0x7f66101a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6610114130 con 0x7f661010e460 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6610113fb0 con 0x7f6610076040 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f66101142b0 con 0x7f6610074a60 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f661019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6610074a60 0x7f661019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f661019ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:50912/0 (socket says 172.21.15.146:50912) 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 -- 172.21.15.146:0/1999055170 learned_addr learned my addr 172.21.15.146:0/1999055170 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660e800640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f661010e460 0x7f66101a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 -- 172.21.15.146:0/1999055170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6610074a60 msgr2=0x7f661019e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6610074a60 0x7f661019e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 -- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f661010e460 msgr2=0x7f66101a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f661010e460 0x7f66101a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660d400640 1 -- 172.21.15.146:0/1999055170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66101a63a0 con 0x7f6610076040 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.778+0000 7f660de00640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6610074a60 0x7f661019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.779+0000 7f660d400640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f661019ecc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00e9f0 tx=0x7f65fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.779+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc00cd80 con 0x7f6610076040 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.779+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65fc010910 con 0x7f6610076040 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.779+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc018990 con 0x7f6610076040 2024-04-24T08:21:11.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.779+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66101a6690 con 0x7f6610076040 2024-04-24T08:21:11.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.779+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66101a6c30 con 0x7f6610076040 2024-04-24T08:21:11.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.781+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6610104630 con 0x7f6610076040 2024-04-24T08:21:11.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.782+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f65fc010ac0 con 0x7f6610076040 2024-04-24T08:21:11.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.784+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f65fc0d0270 con 0x7f6610076040 2024-04-24T08:21:11.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:11.788+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65fc09c890 con 0x7f6610076040 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1422858093' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: Active manager daemon x restarted 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: Activating manager daemon x 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: mgrmap e22: x(active, starting, since 0.0150582s), standbys: y, z 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:21:11.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:21:11.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:21:11.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: Manager daemon x is now available 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:21:11.819 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:11 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:21:11.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1422858093' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-24T08:21:11.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: Active manager daemon x restarted 2024-04-24T08:21:11.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: Activating manager daemon x 2024-04-24T08:21:11.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T08:21:11.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: mgrmap e22: x(active, starting, since 0.0150582s), standbys: y, z 2024-04-24T08:21:11.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:21:11.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:21:11.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: Manager daemon x is now available 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:21:11.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:11 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:21:12.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.107+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f65fc0985e0 con 0x7f6610076040 2024-04-24T08:21:12.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.109+0000 7f6606a00640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f65f40aa9a0 0x7f65f40acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.109+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f65f401dcc0 con 0x7f65f40aa9a0 2024-04-24T08:21:12.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.112+0000 7f660de00640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f65f40aa9a0 0x7f65f40acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.120+0000 7f660de00640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f65f40aa9a0 0x7f65f40acd90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6600009900 tx=0x7f66000315b0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.146+0000 7f6606a00640 1 -- 172.21.15.146:0/1999055170 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f65f401dcc0 con 0x7f65f40aa9a0 2024-04-24T08:21:12.148 INFO:teuthology.orchestra.run.smithi146.stdout:{} 2024-04-24T08:21:12.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f65f40aa9a0 msgr2=0x7f65f40acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f65f40aa9a0 0x7f65f40acd90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6600009900 tx=0x7f66000315b0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 msgr2=0x7f661019ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f661019ecc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00e9f0 tx=0x7f65fc00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 shutdown_connections 2024-04-24T08:21:12.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f65f40aa9a0 0x7f65f40acd90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f661010e460 0x7f66101a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6610076040 0x7f661019ecc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 --2- 172.21.15.146:0/1999055170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6610074a60 0x7f661019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.149+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 >> 172.21.15.146:0/1999055170 conn(0x7f66100fd400 msgr2=0x7f6610101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.150+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 shutdown_connections 2024-04-24T08:21:12.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.150+0000 7f660f200640 1 -- 172.21.15.146:0/1999055170 wait complete. 2024-04-24T08:21:12.175 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-24T08:21:12.186 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-24T08:21:12.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 -- 172.21.15.146:0/3814918543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4101ea0 msgr2=0x7f74e41022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 --2- 172.21.15.146:0/3814918543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4101ea0 0x7f74e41022c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009900 tx=0x7f74d802a410 comp rx=0 tx=0).stop 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 -- 172.21.15.146:0/3814918543 shutdown_connections 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 --2- 172.21.15.146:0/3814918543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4102ff0 0x7f74e4103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 --2- 172.21.15.146:0/3814918543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74e41039b0 0x7f74e410a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 --2- 172.21.15.146:0/3814918543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4101ea0 0x7f74e41022c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 -- 172.21.15.146:0/3814918543 >> 172.21.15.146:0/3814918543 conn(0x7f74e40fd410 msgr2=0x7f74e40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 -- 172.21.15.146:0/3814918543 shutdown_connections 2024-04-24T08:21:12.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.376+0000 7f74eac00640 1 -- 172.21.15.146:0/3814918543 wait complete. 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 Processor -- start 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 -- start start 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4101ea0 0x7f74e419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4102ff0 0x7f74e419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74e41039b0 0x7f74e41a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f74e410db70 con 0x7f74e41039b0 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74e410d9f0 con 0x7f74e4102ff0 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74eac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74e410dcf0 con 0x7f74e4101ea0 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.377+0000 7f74e9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4101ea0 0x7f74e419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e8e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4102ff0 0x7f74e419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74ea200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74e41039b0 0x7f74e41a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4101ea0 0x7f74e419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44410/0 (socket says 172.21.15.146:44410) 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e9800640 1 -- 172.21.15.146:0/1091492889 learned_addr learned my addr 172.21.15.146:0/1091492889 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e8e00640 1 -- 172.21.15.146:0/1091492889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4101ea0 msgr2=0x7f74e419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e8e00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4101ea0 0x7f74e419a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e8e00640 1 -- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74e41039b0 msgr2=0x7f74e41a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e8e00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74e41039b0 0x7f74e41a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e8e00640 1 -- 172.21.15.146:0/1091492889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e41a2050 con 0x7f74e4102ff0 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e9800640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4101ea0 0x7f74e419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e8e00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4102ff0 0x7f74e419a9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f74d000e9c0 tx=0x7f74d000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e2000640 1 -- 172.21.15.146:0/1091492889 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d000cd80 con 0x7f74e4102ff0 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e2000640 1 -- 172.21.15.146:0/1091492889 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d0010910 con 0x7f74e4102ff0 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.378+0000 7f74e2000640 1 -- 172.21.15.146:0/1091492889 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0018980 con 0x7f74e4102ff0 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.379+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e41a2340 con 0x7f74e4102ff0 2024-04-24T08:21:12.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.379+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e41a2840 con 0x7f74e4102ff0 2024-04-24T08:21:12.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.380+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e41a2aa0 con 0x7f74e4102ff0 2024-04-24T08:21:12.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.382+0000 7f74e2000640 1 -- 172.21.15.146:0/1091492889 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f74d001f020 con 0x7f74e4102ff0 2024-04-24T08:21:12.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.384+0000 7f74e2000640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f74cc0a3a30 0x7f74cc0a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.384+0000 7f74e2000640 1 -- 172.21.15.146:0/1091492889 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f74d00d0060 con 0x7f74e4102ff0 2024-04-24T08:21:12.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.384+0000 7f74e9800640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f74cc0a3a30 0x7f74cc0a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.385+0000 7f74e9800640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f74cc0a3a30 0x7f74cc0a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74d80096c0 tx=0x7f74d80315b0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.388+0000 7f74e2000640 1 -- 172.21.15.146:0/1091492889 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74d009cbe0 con 0x7f74e4102ff0 2024-04-24T08:21:12.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:12.131+0000 7f1611a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:12.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:12.131+0000 7f1611a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:12.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:12.131+0000 7f1611a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:12.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:12.131+0000 7f1611a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:12.493 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:21:12.131+0000 7f1611a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:21:12.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.606+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74e41a3170 con 0x7f74e4102ff0 2024-04-24T08:21:12.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.608+0000 7f74e2000640 1 -- 172.21.15.146:0/1091492889 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23068 (secure 0 0 0) 0x7f74d009c4e0 con 0x7f74e4102ff0 2024-04-24T08:21:12.610 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:12.610 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":91,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:21:11.077519+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:0/4151754658":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2355630587":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6800/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/393585939":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/4023745007":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f74cc0a3a30 msgr2=0x7f74cc0a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f74cc0a3a30 0x7f74cc0a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74d80096c0 tx=0x7f74d80315b0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4102ff0 msgr2=0x7f74e419a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4102ff0 0x7f74e419a9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f74d000e9c0 tx=0x7f74d000ee80 comp rx=0 tx=0).stop 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 shutdown_connections 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f74cc0a3a30 0x7f74cc0a5ef0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74e41039b0 0x7f74e41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74e4102ff0 0x7f74e419a9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 --2- 172.21.15.146:0/1091492889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74e4101ea0 0x7f74e419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 >> 172.21.15.146:0/1091492889 conn(0x7f74e40fd410 msgr2=0x7f74e4074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 shutdown_connections 2024-04-24T08:21:12.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.611+0000 7f74eac00640 1 -- 172.21.15.146:0/1091492889 wait complete. 2024-04-24T08:21:12.632 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:12.632 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:12 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1422858093' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-24T08:21:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:12 smithi153 ceph-mon[32746]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:21:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:12 smithi153 ceph-mon[32746]: mgrmap e23: x(active, since 1.02274s), standbys: y, z 2024-04-24T08:21:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:12 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:12 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:12.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 -- 172.21.15.146:0/1088664020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105e30 msgr2=0x7f1ec410c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 --2- 172.21.15.146:0/1088664020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105e30 0x7f1ec410c6c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc009ca0 tx=0x7f1ebc02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 -- 172.21.15.146:0/1088664020 shutdown_connections 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 --2- 172.21.15.146:0/1088664020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105e30 0x7f1ec410c6c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 --2- 172.21.15.146:0/1088664020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4105470 0x7f1ec41058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 --2- 172.21.15.146:0/1088664020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4104280 0x7f1ec41046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 -- 172.21.15.146:0/1088664020 >> 172.21.15.146:0/1088664020 conn(0x7f1ec40ff7f0 msgr2=0x7f1ec4101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 -- 172.21.15.146:0/1088664020 shutdown_connections 2024-04-24T08:21:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.806+0000 7f1ec9800640 1 -- 172.21.15.146:0/1088664020 wait complete. 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 Processor -- start 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 -- start start 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4104280 0x7f1ec419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105470 0x7f1ec419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4105e30 0x7f1ec41a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec410ff10 con 0x7f1ec4105e30 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec410fd90 con 0x7f1ec4105470 2024-04-24T08:21:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.807+0000 7f1ec9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4110090 con 0x7f1ec4104280 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105470 0x7f1ec419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105470 0x7f1ec419aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:50966/0 (socket says 172.21.15.146:50966) 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 -- 172.21.15.146:0/2754322925 learned_addr learned my addr 172.21.15.146:0/2754322925 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2a00640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4104280 0x7f1ec419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec3400640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4105e30 0x7f1ec41a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 -- 172.21.15.146:0/2754322925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4104280 msgr2=0x7f1ec419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4104280 0x7f1ec419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 -- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4105e30 msgr2=0x7f1ec41a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4105e30 0x7f1ec41a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 -- 172.21.15.146:0/2754322925 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec41a2150 con 0x7f1ec4105470 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec3400640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4105e30 0x7f1ec41a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec2000640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105470 0x7f1ec419aa70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e9f0 tx=0x7f1eb000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ebb400640 1 -- 172.21.15.146:0/2754322925 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb000cd80 con 0x7f1ec4105470 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec41a2440 con 0x7f1ec4105470 2024-04-24T08:21:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.808+0000 7f1ebb400640 1 -- 172.21.15.146:0/2754322925 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb0010910 con 0x7f1ec4105470 2024-04-24T08:21:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.809+0000 7f1ebb400640 1 -- 172.21.15.146:0/2754322925 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0018990 con 0x7f1ec4105470 2024-04-24T08:21:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.809+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec41a2990 con 0x7f1ec4105470 2024-04-24T08:21:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.810+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec410b5a0 con 0x7f1ec4105470 2024-04-24T08:21:12.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.812+0000 7f1ebb400640 1 -- 172.21.15.146:0/2754322925 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f1eb0010ac0 con 0x7f1ec4105470 2024-04-24T08:21:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:12 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1422858093' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-24T08:21:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:12 smithi177 ceph-mon[33046]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:21:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:12 smithi177 ceph-mon[33046]: mgrmap e23: x(active, since 1.02274s), standbys: y, z 2024-04-24T08:21:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:12 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:12 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.814+0000 7f1ebb400640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ea80a39b0 0x7f1ea80a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.814+0000 7f1ebb400640 1 -- 172.21.15.146:0/2754322925 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1eb00d0520 con 0x7f1ec4105470 2024-04-24T08:21:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.814+0000 7f1ec2a00640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ea80a39b0 0x7f1ea80a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.815+0000 7f1ec2a00640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ea80a39b0 0x7f1ea80a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1ec41052d0 tx=0x7f1eb4005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:12.818+0000 7f1ebb400640 1 -- 172.21.15.146:0/2754322925 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1eb009d0a0 con 0x7f1ec4105470 2024-04-24T08:21:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1422858093' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-24T08:21:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-mon[27805]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:21:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-mon[27805]: mgrmap e23: x(active, since 1.02274s), standbys: y, z 2024-04-24T08:21:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:12 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.032+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ec4062c30 con 0x7f1ec4105470 2024-04-24T08:21:13.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.033+0000 7f1ebb400640 1 -- 172.21.15.146:0/2754322925 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f1eb009c9a0 con 0x7f1ec4105470 2024-04-24T08:21:13.034 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ea80a39b0 msgr2=0x7f1ea80a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ea80a39b0 0x7f1ea80a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1ec41052d0 tx=0x7f1eb4005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105470 msgr2=0x7f1ec419aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105470 0x7f1ec419aa70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e9f0 tx=0x7f1eb000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 shutdown_connections 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ea80a39b0 0x7f1ea80a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec4105e30 0x7f1ec41a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ec4105470 0x7f1ec419aa70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 --2- 172.21.15.146:0/2754322925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ec4104280 0x7f1ec419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 >> 172.21.15.146:0/2754322925 conn(0x7f1ec40ff7f0 msgr2=0x7f1ec4108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 shutdown_connections 2024-04-24T08:21:13.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.035+0000 7f1ec9800640 1 -- 172.21.15.146:0/2754322925 wait complete. 2024-04-24T08:21:13.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 -- 172.21.15.146:0/1066026375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105cf0 msgr2=0x7f168810c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 --2- 172.21.15.146:0/1066026375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105cf0 0x7f168810c580 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1680009be0 tx=0x7f168002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 -- 172.21.15.146:0/1066026375 shutdown_connections 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 --2- 172.21.15.146:0/1066026375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1688105330 0x7f16881057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 --2- 172.21.15.146:0/1066026375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105cf0 0x7f168810c580 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 --2- 172.21.15.146:0/1066026375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1688104140 0x7f1688104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 -- 172.21.15.146:0/1066026375 >> 172.21.15.146:0/1066026375 conn(0x7f16880ff6b0 msgr2=0x7f1688101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 -- 172.21.15.146:0/1066026375 shutdown_connections 2024-04-24T08:21:13.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.237+0000 7f168d800640 1 -- 172.21.15.146:0/1066026375 wait complete. 2024-04-24T08:21:13.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 Processor -- start 2024-04-24T08:21:13.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 -- start start 2024-04-24T08:21:13.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1688104140 0x7f168819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105330 0x7f168819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1688105cf0 0x7f16881a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f168810fd40 con 0x7f1688104140 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f168810fbc0 con 0x7f1688105330 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.238+0000 7f168d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f168810fec0 con 0x7f1688105cf0 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1688104140 0x7f168819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105330 0x7f168819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105330 0x7f168819a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:51008/0 (socket says 172.21.15.146:51008) 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 -- 172.21.15.146:0/4234416430 learned_addr learned my addr 172.21.15.146:0/4234416430 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f168ce00640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1688105cf0 0x7f16881a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 -- 172.21.15.146:0/4234416430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1688105cf0 msgr2=0x7f16881a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1688105cf0 0x7f16881a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 -- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1688104140 msgr2=0x7f168819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1688104140 0x7f168819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 -- 172.21.15.146:0/4234416430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16881a1fa0 con 0x7f1688105330 2024-04-24T08:21:13.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1687400640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105330 0x7f168819a8c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f167400e9f0 tx=0x7f167400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.239+0000 7f1684c00640 1 -- 172.21.15.146:0/4234416430 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167400cd80 con 0x7f1688105330 2024-04-24T08:21:13.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.240+0000 7f1684c00640 1 -- 172.21.15.146:0/4234416430 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1674010910 con 0x7f1688105330 2024-04-24T08:21:13.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.240+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16881a2290 con 0x7f1688105330 2024-04-24T08:21:13.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.240+0000 7f1684c00640 1 -- 172.21.15.146:0/4234416430 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1674018980 con 0x7f1688105330 2024-04-24T08:21:13.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.240+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16881a27e0 con 0x7f1688105330 2024-04-24T08:21:13.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.241+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f168810b460 con 0x7f1688105330 2024-04-24T08:21:13.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.243+0000 7f1684c00640 1 -- 172.21.15.146:0/4234416430 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f1674010ac0 con 0x7f1688105330 2024-04-24T08:21:13.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.244+0000 7f1684c00640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f166c0a39b0 0x7f166c0a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.245+0000 7f1684c00640 1 -- 172.21.15.146:0/4234416430 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f16740d0470 con 0x7f1688105330 2024-04-24T08:21:13.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.245+0000 7f1687e00640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f166c0a39b0 0x7f166c0a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.246+0000 7f1687e00640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f166c0a39b0 0x7f166c0a5e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1678009530 tx=0x7f1678009260 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.249+0000 7f1684c00640 1 -- 172.21.15.146:0/4234416430 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f167409c9a0 con 0x7f1688105330 2024-04-24T08:21:13.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.461+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1688062c00 con 0x7f1688105330 2024-04-24T08:21:13.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.462+0000 7f1684c00640 1 -- 172.21.15.146:0/4234416430 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f16740a0ca0 con 0x7f1688105330 2024-04-24T08:21:13.464 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:21:13.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f166c0a39b0 msgr2=0x7f166c0a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f166c0a39b0 0x7f166c0a5e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1678009530 tx=0x7f1678009260 comp rx=0 tx=0).stop 2024-04-24T08:21:13.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105330 msgr2=0x7f168819a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105330 0x7f168819a8c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f167400e9f0 tx=0x7f167400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 shutdown_connections 2024-04-24T08:21:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f166c0a39b0 0x7f166c0a5e70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1688105cf0 0x7f16881a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1688105330 0x7f168819a8c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 --2- 172.21.15.146:0/4234416430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1688104140 0x7f168819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.466+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 >> 172.21.15.146:0/4234416430 conn(0x7f16880ff6b0 msgr2=0x7f1688108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.467+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 shutdown_connections 2024-04-24T08:21:13.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.467+0000 7f168d800640 1 -- 172.21.15.146:0/4234416430 wait complete. 2024-04-24T08:21:13.486 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 -- 172.21.15.146:0/2965917312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c104280 msgr2=0x7fab2c1046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 --2- 172.21.15.146:0/2965917312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c104280 0x7fab2c1046a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab1c009900 tx=0x7fab1c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 -- 172.21.15.146:0/2965917312 shutdown_connections 2024-04-24T08:21:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 --2- 172.21.15.146:0/2965917312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab2c105e30 0x7fab2c10c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 --2- 172.21.15.146:0/2965917312 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 0x7fab2c1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 --2- 172.21.15.146:0/2965917312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c104280 0x7fab2c1046a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 -- 172.21.15.146:0/2965917312 >> 172.21.15.146:0/2965917312 conn(0x7fab2c0ff7f0 msgr2=0x7fab2c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 -- 172.21.15.146:0/2965917312 shutdown_connections 2024-04-24T08:21:13.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.689+0000 7fab32400640 1 -- 172.21.15.146:0/2965917312 wait complete. 2024-04-24T08:21:13.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.690+0000 7fab32400640 1 Processor -- start 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.690+0000 7fab32400640 1 -- start start 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab32400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab2c104280 0x7fab2c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab32400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 0x7fab2c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab32400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c1a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab32400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c10ff60 con 0x7fab2c104280 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab32400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c10fde0 con 0x7fab2c105470 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab32400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1100e0 con 0x7fab2c105e30 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 0x7fab2c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab2c104280 0x7fab2c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 0x7fab2c19ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:51034/0 (socket says 172.21.15.146:51034) 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 -- 172.21.15.146:0/1869966322 learned_addr learned my addr 172.21.15.146:0/1869966322 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2be00640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c1a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 -- 172.21.15.146:0/1869966322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 msgr2=0x7fab2c1a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c1a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 -- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab2c104280 msgr2=0x7fab2c19a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab2c104280 0x7fab2c19a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.691+0000 7fab2aa00640 1 -- 172.21.15.146:0/1869966322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c1a2210 con 0x7fab2c105470 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.692+0000 7fab2aa00640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 0x7fab2c19ab30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9f0 tx=0x7fab1800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.692+0000 7fab23e00640 1 -- 172.21.15.146:0/1869966322 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1800cd80 con 0x7fab2c105470 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.692+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c1a2500 con 0x7fab2c105470 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.692+0000 7fab23e00640 1 -- 172.21.15.146:0/1869966322 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab18005500 con 0x7fab2c105470 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.692+0000 7fab23e00640 1 -- 172.21.15.146:0/1869966322 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab18010690 con 0x7fab2c105470 2024-04-24T08:21:13.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.692+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c1a2a00 con 0x7fab2c105470 2024-04-24T08:21:13.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.694+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2c10b5a0 con 0x7fab2c105470 2024-04-24T08:21:13.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.696+0000 7fab23e00640 1 -- 172.21.15.146:0/1869966322 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) ==== 138818+0+0 (secure 0 0 0) 0x7fab180046c0 con 0x7fab2c105470 2024-04-24T08:21:13.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.697+0000 7fab23e00640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fab100a3db0 0x7fab100a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.697+0000 7fab23e00640 1 -- 172.21.15.146:0/1869966322 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fab180d0e00 con 0x7fab2c105470 2024-04-24T08:21:13.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.698+0000 7fab2b400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fab100a3db0 0x7fab100a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.698+0000 7fab2b400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fab100a3db0 0x7fab100a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02a9e0 tx=0x7fab1c0315b0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.702+0000 7fab23e00640 1 -- 172.21.15.146:0/1869966322 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab1809d2e0 con 0x7fab2c105470 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: [24/Apr/2024:08:21:12] ENGINE Bus STARTING 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1091492889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: [24/Apr/2024:08:21:12] ENGINE Serving on https://172.21.15.146:7150 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: [24/Apr/2024:08:21:12] ENGINE Serving on http://172.21.15.146:8765 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: [24/Apr/2024:08:21:12] ENGINE Bus STARTED 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2754322925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.739 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.740 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:13 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4234416430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:13.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.907+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fab2c02d7d0 con 0x7fab2c105470 2024-04-24T08:21:13.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.908+0000 7fab23e00640 1 -- 172.21.15.146:0/1869966322 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fab1809cbe0 con 0x7fab2c105470 2024-04-24T08:21:13.910 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fab100a3db0 msgr2=0x7fab100a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fab100a3db0 0x7fab100a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02a9e0 tx=0x7fab1c0315b0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 msgr2=0x7fab2c19ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 0x7fab2c19ab30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9f0 tx=0x7fab1800edf0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 shutdown_connections 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fab100a3db0 0x7fab100a6270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab2c105470 0x7fab2c19ab30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 --2- 172.21.15.146:0/1869966322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab2c104280 0x7fab2c19a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 >> 172.21.15.146:0/1869966322 conn(0x7fab2c0ff7f0 msgr2=0x7fab2c108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 shutdown_connections 2024-04-24T08:21:13.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:13.911+0000 7fab32400640 1 -- 172.21.15.146:0/1869966322 wait complete. 2024-04-24T08:21:13.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:13.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: [24/Apr/2024:08:21:12] ENGINE Bus STARTING 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1091492889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: [24/Apr/2024:08:21:12] ENGINE Serving on https://172.21.15.146:7150 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: [24/Apr/2024:08:21:12] ENGINE Serving on http://172.21.15.146:8765 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: [24/Apr/2024:08:21:12] ENGINE Bus STARTED 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2754322925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:13 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4234416430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:13.984 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: [24/Apr/2024:08:21:12] ENGINE Bus STARTING 2024-04-24T08:21:13.984 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1091492889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: [24/Apr/2024:08:21:12] ENGINE Serving on https://172.21.15.146:7150 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: [24/Apr/2024:08:21:12] ENGINE Serving on http://172.21.15.146:8765 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: [24/Apr/2024:08:21:12] ENGINE Bus STARTED 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2754322925' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:13.985 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:13 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4234416430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:14.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 -- 172.21.15.146:0/3169873838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250104280 msgr2=0x7f12501046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 --2- 172.21.15.146:0/3169873838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250104280 0x7f12501046a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1240009900 tx=0x7f124002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 -- 172.21.15.146:0/3169873838 shutdown_connections 2024-04-24T08:21:14.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 --2- 172.21.15.146:0/3169873838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105e30 0x7f125010c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 --2- 172.21.15.146:0/3169873838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1250105470 0x7f12501058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 --2- 172.21.15.146:0/3169873838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250104280 0x7f12501046a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 -- 172.21.15.146:0/3169873838 >> 172.21.15.146:0/3169873838 conn(0x7f12500ff7f0 msgr2=0x7f1250101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:14.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 -- 172.21.15.146:0/3169873838 shutdown_connections 2024-04-24T08:21:14.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.126+0000 7f1255c00640 1 -- 172.21.15.146:0/3169873838 wait complete. 2024-04-24T08:21:14.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.127+0000 7f1255c00640 1 Processor -- start 2024-04-24T08:21:14.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.127+0000 7f1255c00640 1 -- start start 2024-04-24T08:21:14.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f1255c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1250104280 0x7f125019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f1255c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105470 0x7f125019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f1255c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250105e30 0x7f12501a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f1255c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f125010ff60 con 0x7f1250104280 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f1255c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f125010fde0 con 0x7f1250105e30 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f1255c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12501100e0 con 0x7f1250105470 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f124ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1250104280 0x7f125019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f124e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105470 0x7f125019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f124e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105470 0x7f125019aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44516/0 (socket says 172.21.15.146:44516) 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f124e000640 1 -- 172.21.15.146:0/217157962 learned_addr learned my addr 172.21.15.146:0/217157962 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f124ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1250104280 0x7f125019a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39076/0 (socket says 172.21.15.146:39076) 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.128+0000 7f124f400640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250105e30 0x7f12501a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f124e000640 1 -- 172.21.15.146:0/217157962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250105e30 msgr2=0x7f12501a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f124e000640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250105e30 0x7f12501a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f124e000640 1 -- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1250104280 msgr2=0x7f125019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f124e000640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1250104280 0x7f125019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f124e000640 1 -- 172.21.15.146:0/217157962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12501a21b0 con 0x7f1250105470 2024-04-24T08:21:14.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f124f400640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250105e30 0x7f12501a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:14.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f124e000640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105470 0x7f125019aad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f123c00d950 tx=0x7f123c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:14.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f1247400640 1 -- 172.21.15.146:0/217157962 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123c017070 con 0x7f1250105470 2024-04-24T08:21:14.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12501a24a0 con 0x7f1250105470 2024-04-24T08:21:14.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f1247400640 1 -- 172.21.15.146:0/217157962 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f123c009c30 con 0x7f1250105470 2024-04-24T08:21:14.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f1247400640 1 -- 172.21.15.146:0/217157962 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123c00f6b0 con 0x7f1250105470 2024-04-24T08:21:14.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.129+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12501a2a40 con 0x7f1250105470 2024-04-24T08:21:14.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.131+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125010b5a0 con 0x7f1250105470 2024-04-24T08:21:14.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.134+0000 7f1247400640 1 -- 172.21.15.146:0/217157962 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138818+0+0 (secure 0 0 0) 0x7f123c0109d0 con 0x7f1250105470 2024-04-24T08:21:14.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.135+0000 7f1247400640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f12340a3db0 0x7f12340a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.135+0000 7f1247400640 1 -- 172.21.15.146:0/217157962 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f123c0d0560 con 0x7f1250105470 2024-04-24T08:21:14.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.135+0000 7f124ea00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f12340a3db0 0x7f12340a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.136+0000 7f124ea00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f12340a3db0 0x7f12340a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f124002a9e0 tx=0x7f12400315b0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:14.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.141+0000 7f1247400640 1 -- 172.21.15.146:0/217157962 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f123c009de0 con 0x7f1250105470 2024-04-24T08:21:14.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.374+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f125002d820 con 0x7f1250105470 2024-04-24T08:21:14.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.377+0000 7f1247400640 1 -- 172.21.15.146:0/217157962 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23068 (secure 0 0 0) 0x7f123c09c450 con 0x7f1250105470 2024-04-24T08:21:14.379 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:14.379 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":91,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:21:11.077519+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:0/4151754658":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2355630587":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6800/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/393585939":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/4023745007":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f12340a3db0 msgr2=0x7f12340a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f12340a3db0 0x7f12340a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f124002a9e0 tx=0x7f12400315b0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105470 msgr2=0x7f125019aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105470 0x7f125019aad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f123c00d950 tx=0x7f123c00de10 comp rx=0 tx=0).stop 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 shutdown_connections 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f12340a3db0 0x7f12340a6270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1250105470 0x7f125019aad0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1250105e30 0x7f12501a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 --2- 172.21.15.146:0/217157962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1250104280 0x7f125019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 >> 172.21.15.146:0/217157962 conn(0x7f12500ff7f0 msgr2=0x7f1250108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 shutdown_connections 2024-04-24T08:21:14.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.380+0000 7f1255c00640 1 -- 172.21.15.146:0/217157962 wait complete. 2024-04-24T08:21:14.400 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:14.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 -- 172.21.15.146:0/2270729770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40105e30 msgr2=0x7ffb4010c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 --2- 172.21.15.146:0/2270729770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40105e30 0x7ffb4010c6c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009ca0 tx=0x7ffb3802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 -- 172.21.15.146:0/2270729770 shutdown_connections 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 --2- 172.21.15.146:0/2270729770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40105e30 0x7ffb4010c6c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 --2- 172.21.15.146:0/2270729770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb40105470 0x7ffb401058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 --2- 172.21.15.146:0/2270729770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb40104280 0x7ffb401046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 -- 172.21.15.146:0/2270729770 >> 172.21.15.146:0/2270729770 conn(0x7ffb400ff7f0 msgr2=0x7ffb40101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:14.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 -- 172.21.15.146:0/2270729770 shutdown_connections 2024-04-24T08:21:14.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.595+0000 7ffb48400640 1 -- 172.21.15.146:0/2270729770 wait complete. 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.596+0000 7ffb48400640 1 Processor -- start 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.596+0000 7ffb48400640 1 -- start start 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb48400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40104280 0x7ffb4019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb48400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb40105470 0x7ffb4019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb48400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb40105e30 0x7ffb401a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb48400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4010ff60 con 0x7ffb40105470 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb48400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4010fde0 con 0x7ffb40105e30 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb48400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb401100e0 con 0x7ffb40104280 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40104280 0x7ffb4019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40104280 0x7ffb4019a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44536/0 (socket says 172.21.15.146:44536) 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 -- 172.21.15.146:0/310577381 learned_addr learned my addr 172.21.15.146:0/310577381 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb46400640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb40105e30 0x7ffb401a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45000640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb40105470 0x7ffb4019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 -- 172.21.15.146:0/310577381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb40105e30 msgr2=0x7ffb401a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb40105e30 0x7ffb401a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 -- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb40105470 msgr2=0x7ffb4019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb40105470 0x7ffb4019aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.597+0000 7ffb45a00640 1 -- 172.21.15.146:0/310577381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb401a21d0 con 0x7ffb40104280 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.598+0000 7ffb45a00640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40104280 0x7ffb4019a5b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400e9c0 tx=0x7ffb3400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.598+0000 7ffb3e000640 1 -- 172.21.15.146:0/310577381 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3400cd80 con 0x7ffb40104280 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.598+0000 7ffb3e000640 1 -- 172.21.15.146:0/310577381 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb340035f0 con 0x7ffb40104280 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.598+0000 7ffb3e000640 1 -- 172.21.15.146:0/310577381 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb34010690 con 0x7ffb40104280 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.598+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb401a24c0 con 0x7ffb40104280 2024-04-24T08:21:14.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.598+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb401a29c0 con 0x7ffb40104280 2024-04-24T08:21:14.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.600+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb4010b4d0 con 0x7ffb40104280 2024-04-24T08:21:14.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.603+0000 7ffb3e000640 1 -- 172.21.15.146:0/310577381 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138818+0+0 (secure 0 0 0) 0x7ffb340037a0 con 0x7ffb40104280 2024-04-24T08:21:14.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.604+0000 7ffb3e000640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffb280a3e30 0x7ffb280a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.604+0000 7ffb3e000640 1 -- 172.21.15.146:0/310577381 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ffb340d0db0 con 0x7ffb40104280 2024-04-24T08:21:14.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.604+0000 7ffb45000640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffb280a3e30 0x7ffb280a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.605+0000 7ffb45000640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffb280a3e30 0x7ffb280a62f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffb4019bac0 tx=0x7ffb2c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:14.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.608+0000 7ffb3e000640 1 -- 172.21.15.146:0/310577381 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb3409d2f0 con 0x7ffb40104280 2024-04-24T08:21:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1869966322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/217157962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:14 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.829+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffb40062c00 con 0x7ffb40104280 2024-04-24T08:21:14.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.829+0000 7ffb3e000640 1 -- 172.21.15.146:0/310577381 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ffb3409cbf0 con 0x7ffb40104280 2024-04-24T08:21:14.831 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffb280a3e30 msgr2=0x7ffb280a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffb280a3e30 0x7ffb280a62f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffb4019bac0 tx=0x7ffb2c005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40104280 msgr2=0x7ffb4019a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40104280 0x7ffb4019a5b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400e9c0 tx=0x7ffb3400ee80 comp rx=0 tx=0).stop 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 shutdown_connections 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffb280a3e30 0x7ffb280a62f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb40105e30 0x7ffb401a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb40105470 0x7ffb4019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 --2- 172.21.15.146:0/310577381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb40104280 0x7ffb4019a5b0 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400e9c0 tx=0x7ffb3400ee80 comp rx=0 tx=0).stop 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 >> 172.21.15.146:0/310577381 conn(0x7ffb400ff7f0 msgr2=0x7ffb401086a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 shutdown_connections 2024-04-24T08:21:14.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:14.832+0000 7ffb48400640 1 -- 172.21.15.146:0/310577381 wait complete. 2024-04-24T08:21:14.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:14.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:14.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-24T08:21:14.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1869966322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:14.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/217157962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:14.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.950 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.951 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.952 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:14 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1869966322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/217157962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:14.963 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:14 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 -- 172.21.15.146:0/1356675828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8100e10 msgr2=0x7f97e8101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 --2- 172.21.15.146:0/1356675828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8100e10 0x7f97e8101230 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009900 tx=0x7f97d802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 -- 172.21.15.146:0/1356675828 shutdown_connections 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 --2- 172.21.15.146:0/1356675828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97e8102920 0x7f97e8109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 --2- 172.21.15.146:0/1356675828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e8101f60 0x7f97e81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 --2- 172.21.15.146:0/1356675828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8100e10 0x7f97e8101230 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 -- 172.21.15.146:0/1356675828 >> 172.21.15.146:0/1356675828 conn(0x7f97e80fc3a0 msgr2=0x7f97e80fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 -- 172.21.15.146:0/1356675828 shutdown_connections 2024-04-24T08:21:15.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.051+0000 7f97e7e00640 1 -- 172.21.15.146:0/1356675828 wait complete. 2024-04-24T08:21:15.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 Processor -- start 2024-04-24T08:21:15.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 -- start start 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e8100e10 0x7f97e819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97e8101f60 0x7f97e819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8102920 0x7f97e81a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f97e810cb00 con 0x7f97e8100e10 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f97e810c980 con 0x7f97e8101f60 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.052+0000 7f97e7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97e810cc80 con 0x7f97e8102920 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e8100e10 0x7f97e819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e8100e10 0x7f97e819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39114/0 (socket says 172.21.15.146:39114) 2024-04-24T08:21:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e6a00640 1 -- 172.21.15.146:0/804336048 learned_addr learned my addr 172.21.15.146:0/804336048 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e7400640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8102920 0x7f97e81a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e6000640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97e8101f60 0x7f97e819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e7400640 1 -- 172.21.15.146:0/804336048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97e8101f60 msgr2=0x7f97e819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e7400640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97e8101f60 0x7f97e819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e7400640 1 -- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e8100e10 msgr2=0x7f97e819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e7400640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e8100e10 0x7f97e819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e7400640 1 -- 172.21.15.146:0/804336048 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e81a20f0 con 0x7f97e8102920 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.053+0000 7f97e7400640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8102920 0x7f97e81a1a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009fd0 tx=0x7f97e000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.054+0000 7f97eca00640 1 -- 172.21.15.146:0/804336048 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e000ce20 con 0x7f97e8102920 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.054+0000 7f97eca00640 1 -- 172.21.15.146:0/804336048 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97e0005020 con 0x7f97e8102920 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.054+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e81a23e0 con 0x7f97e8102920 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.054+0000 7f97eca00640 1 -- 172.21.15.146:0/804336048 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0010720 con 0x7f97e8102920 2024-04-24T08:21:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.054+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e81a2980 con 0x7f97e8102920 2024-04-24T08:21:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.055+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e81a2c40 con 0x7f97e8102920 2024-04-24T08:21:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.057+0000 7f97eca00640 1 -- 172.21.15.146:0/804336048 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138818+0+0 (secure 0 0 0) 0x7f97e0005540 con 0x7f97e8102920 2024-04-24T08:21:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.058+0000 7f97eca00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f97cc0a3e30 0x7f97cc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.058+0000 7f97eca00640 1 -- 172.21.15.146:0/804336048 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f97e00d0f60 con 0x7f97e8102920 2024-04-24T08:21:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.058+0000 7f97e6a00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f97cc0a3e30 0x7f97cc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.059+0000 7f97e6a00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f97cc0a3e30 0x7f97cc0a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97d80096c0 tx=0x7f97d80315b0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.062+0000 7f97eca00640 1 -- 172.21.15.146:0/804336048 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97e009d440 con 0x7f97e8102920 2024-04-24T08:21:15.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.260+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97e81a2e10 con 0x7f97e8102920 2024-04-24T08:21:15.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.261+0000 7f97eca00640 1 -- 172.21.15.146:0/804336048 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f97e009cd40 con 0x7f97e8102920 2024-04-24T08:21:15.263 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:21:15.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f97cc0a3e30 msgr2=0x7f97cc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f97cc0a3e30 0x7f97cc0a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97d80096c0 tx=0x7f97d80315b0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8102920 msgr2=0x7f97e81a1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8102920 0x7f97e81a1a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009fd0 tx=0x7f97e000ef10 comp rx=0 tx=0).stop 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 shutdown_connections 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f97cc0a3e30 0x7f97cc0a62f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97e8102920 0x7f97e81a1a80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97e8101f60 0x7f97e819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 --2- 172.21.15.146:0/804336048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97e8100e10 0x7f97e819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.263+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 >> 172.21.15.146:0/804336048 conn(0x7f97e80fc3a0 msgr2=0x7f97e80fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.264+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 shutdown_connections 2024-04-24T08:21:15.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.264+0000 7f97e7e00640 1 -- 172.21.15.146:0/804336048 wait complete. 2024-04-24T08:21:15.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1866728276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4102990 msgr2=0x7f7cf4104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1866728276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4102990 0x7f7cf4104d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009900 tx=0x7f7ce402a410 comp rx=0 tx=0).stop 2024-04-24T08:21:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1866728276 shutdown_connections 2024-04-24T08:21:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1866728276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cf4107bf0 0x7f7cf410a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1866728276 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 0x7f7cf41076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1866728276 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4102990 0x7f7cf4104d80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1866728276 >> 172.21.15.146:0/1866728276 conn(0x7f7cf40fc3a0 msgr2=0x7f7cf40fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:15.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1866728276 shutdown_connections 2024-04-24T08:21:15.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.484+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1866728276 wait complete. 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 Processor -- start 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 -- start start 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cf4102990 0x7f7cf419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 0x7f7cf419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4107bf0 0x7f7cf41a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf410db80 con 0x7f7cf41052c0 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf410da00 con 0x7f7cf4107bf0 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.485+0000 7f7cf8a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf410dd00 con 0x7f7cf4102990 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 0x7f7cf419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 0x7f7cf419a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39120/0 (socket says 172.21.15.146:39120) 2024-04-24T08:21:15.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 -- 172.21.15.146:0/1844869718 learned_addr learned my addr 172.21.15.146:0/1844869718 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf3400640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cf4102990 0x7f7cf419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf3e00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4107bf0 0x7f7cf41a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 -- 172.21.15.146:0/1844869718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cf4102990 msgr2=0x7f7cf419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cf4102990 0x7f7cf419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 -- 172.21.15.146:0/1844869718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4107bf0 msgr2=0x7f7cf41a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4107bf0 0x7f7cf41a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf2a00640 1 -- 172.21.15.146:0/1844869718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf41a2050 con 0x7f7cf41052c0 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf3400640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cf4102990 0x7f7cf419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:15.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.486+0000 7f7cf3e00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4107bf0 0x7f7cf41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:15.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.487+0000 7f7cf2a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 0x7f7cf419a970 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9c0 tx=0x7f7ce000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:15.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.487+0000 7f7cebe00640 1 -- 172.21.15.146:0/1844869718 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000cd80 con 0x7f7cf41052c0 2024-04-24T08:21:15.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.487+0000 7f7cebe00640 1 -- 172.21.15.146:0/1844869718 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0005500 con 0x7f7cf41052c0 2024-04-24T08:21:15.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.487+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf41a2340 con 0x7f7cf41052c0 2024-04-24T08:21:15.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.487+0000 7f7cebe00640 1 -- 172.21.15.146:0/1844869718 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0010690 con 0x7f7cf41052c0 2024-04-24T08:21:15.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.487+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf41a2890 con 0x7f7cf41052c0 2024-04-24T08:21:15.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.488+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf4109740 con 0x7f7cf41052c0 2024-04-24T08:21:15.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.491+0000 7f7cebe00640 1 -- 172.21.15.146:0/1844869718 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 24) ==== 138818+0+0 (secure 0 0 0) 0x7f7ce00046c0 con 0x7f7cf41052c0 2024-04-24T08:21:15.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.492+0000 7f7cebe00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f7cd80a3db0 0x7f7cd80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.493+0000 7f7cebe00640 1 -- 172.21.15.146:0/1844869718 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f7ce00d0d60 con 0x7f7cf41052c0 2024-04-24T08:21:15.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.493+0000 7f7cf3400640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f7cd80a3db0 0x7f7cd80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.493+0000 7f7cf3400640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f7cd80a3db0 0x7f7cd80a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40314e0 tx=0x7f7ce4000bc0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:15.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.501+0000 7f7cebe00640 1 -- 172.21.15.146:0/1844869718 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ce009d240 con 0x7f7cf41052c0 2024-04-24T08:21:15.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.714+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7cf402d870 con 0x7f7cf41052c0 2024-04-24T08:21:15.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.714+0000 7f7cebe00640 1 -- 172.21.15.146:0/1844869718 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f7ce009cb40 con 0x7f7cf41052c0 2024-04-24T08:21:15.716 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f7cd80a3db0 msgr2=0x7f7cd80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f7cd80a3db0 0x7f7cd80a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40314e0 tx=0x7f7ce4000bc0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 msgr2=0x7f7cf419a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 0x7f7cf419a970 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9c0 tx=0x7f7ce000ee80 comp rx=0 tx=0).stop 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 shutdown_connections 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f7cd80a3db0 0x7f7cd80a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf4107bf0 0x7f7cf41a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cf41052c0 0x7f7cf419a970 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 --2- 172.21.15.146:0/1844869718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cf4102990 0x7f7cf419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 >> 172.21.15.146:0/1844869718 conn(0x7f7cf40fc3a0 msgr2=0x7f7cf4102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:15.719 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 shutdown_connections 2024-04-24T08:21:15.720 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.717+0000 7f7cf8a00640 1 -- 172.21.15.146:0/1844869718 wait complete. 2024-04-24T08:21:15.736 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-24T08:21:15.736 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:15.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 -- 172.21.15.146:0/386556372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc108480 msgr2=0x7f8edc1088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 --2- 172.21.15.146:0/386556372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc108480 0x7f8edc1088a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009ca0 tx=0x7f8ecc02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 -- 172.21.15.146:0/386556372 shutdown_connections 2024-04-24T08:21:15.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 --2- 172.21.15.146:0/386556372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8edc109670 0x7f8edc109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 --2- 172.21.15.146:0/386556372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc10a030 0x7f8edc1108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 --2- 172.21.15.146:0/386556372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc108480 0x7f8edc1088a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 -- 172.21.15.146:0/386556372 >> 172.21.15.146:0/386556372 conn(0x7f8edc077440 msgr2=0x7f8edc079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:15.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 -- 172.21.15.146:0/386556372 shutdown_connections 2024-04-24T08:21:15.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.903+0000 7f8edbe00640 1 -- 172.21.15.146:0/386556372 wait complete. 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 Processor -- start 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 -- start start 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc108480 0x7f8edc19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc109670 0x7f8edc19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8edc10a030 0x7f8edc1a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc114110 con 0x7f8edc108480 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc113f90 con 0x7f8edc109670 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.904+0000 7f8edbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc114290 con 0x7f8edc10a030 2024-04-24T08:21:15.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8edaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc108480 0x7f8edc19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8edaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc108480 0x7f8edc19e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39132/0 (socket says 172.21.15.146:39132) 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8edaa00640 1 -- 172.21.15.146:0/3125769609 learned_addr learned my addr 172.21.15.146:0/3125769609 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8eda000640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc109670 0x7f8edc19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8edb400640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8edc10a030 0x7f8edc1a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8eda000640 1 -- 172.21.15.146:0/3125769609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8edc10a030 msgr2=0x7f8edc1a5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8eda000640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8edc10a030 0x7f8edc1a5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8eda000640 1 -- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc108480 msgr2=0x7f8edc19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8eda000640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc108480 0x7f8edc19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8eda000640 1 -- 172.21.15.146:0/3125769609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc1a6410 con 0x7f8edc109670 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.905+0000 7f8edaa00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc108480 0x7f8edc19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.906+0000 7f8eda000640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc109670 0x7f8edc19ed30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400e9f0 tx=0x7f8ec400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.906+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec400ccc0 con 0x7f8edc109670 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.906+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec4005020 con 0x7f8edc109670 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.906+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc1a6700 con 0x7f8edc109670 2024-04-24T08:21:15.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.906+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec40049e0 con 0x7f8edc109670 2024-04-24T08:21:15.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.906+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc1a6c00 con 0x7f8edc109670 2024-04-24T08:21:15.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.907+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc10f820 con 0x7f8edc109670 2024-04-24T08:21:15.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.909+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 24) ==== 138818+0+0 (secure 0 0 0) 0x7f8ec4005560 con 0x7f8edc109670 2024-04-24T08:21:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.910+0000 7f8ed3400640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ec00a3db0 0x7f8ec00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.911+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8ec40d0980 con 0x7f8edc109670 2024-04-24T08:21:15.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.911+0000 7f8edaa00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ec00a3db0 0x7f8ec00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:15.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.911+0000 7f8edaa00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ec00a3db0 0x7f8ec00a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc006010 tx=0x7f8ecc000f40 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:15.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:15.915+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ec40051d0 con 0x7f8edc109670 2024-04-24T08:21:15.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-24T08:21:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-24T08:21:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-24T08:21:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-24T08:21:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/310577381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:15.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:15 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/804336048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:15.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-24T08:21:15.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-24T08:21:15.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-24T08:21:15.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-24T08:21:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/310577381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:15.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:15 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/804336048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:16.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.060+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f8ec4098dd0 con 0x7f8edc109670 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/310577381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:16.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:15 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/804336048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:16.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.190+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8edc02d870 con 0x7f8edc109670 2024-04-24T08:21:16.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.191+0000 7f8ed3400640 1 -- 172.21.15.146:0/3125769609 <== mon.2 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11891 (secure 0 0 0) 0x7f8ec409ce60 con 0x7f8edc109670 2024-04-24T08:21:16.193 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:16.193 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:21:10.741287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24553,24568]},"id":1}]} 2024-04-24T08:21:16.193 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 24 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ec00a3db0 msgr2=0x7f8ec00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ec00a3db0 0x7f8ec00a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc006010 tx=0x7f8ecc000f40 comp rx=0 tx=0).stop 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc109670 msgr2=0x7f8edc19ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc109670 0x7f8edc19ed30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400e9f0 tx=0x7f8ec400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 shutdown_connections 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ec00a3db0 0x7f8ec00a6270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8edc10a030 0x7f8edc1a5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8edc109670 0x7f8edc19ed30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 --2- 172.21.15.146:0/3125769609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8edc108480 0x7f8edc19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 >> 172.21.15.146:0/3125769609 conn(0x7f8edc077440 msgr2=0x7f8edc10c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:16.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 shutdown_connections 2024-04-24T08:21:16.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.194+0000 7f8edbe00640 1 -- 172.21.15.146:0/3125769609 wait complete. 2024-04-24T08:21:16.213 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-24T08:21:16.213 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-24T08:21:16.214 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 -- 172.21.15.146:0/3427005173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388105400 msgr2=0x7f03881077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 --2- 172.21.15.146:0/3427005173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388105400 0x7f03881077f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0370009900 tx=0x7f037002a400 comp rx=0 tx=0).stop 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 -- 172.21.15.146:0/3427005173 shutdown_connections 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 --2- 172.21.15.146:0/3427005173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388105400 0x7f03881077f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 --2- 172.21.15.146:0/3427005173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0388107d30 0x7f038810a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 --2- 172.21.15.146:0/3427005173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388102ad0 0x7f0388104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 -- 172.21.15.146:0/3427005173 >> 172.21.15.146:0/3427005173 conn(0x7f03880fc4e0 msgr2=0x7f03880fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 -- 172.21.15.146:0/3427005173 shutdown_connections 2024-04-24T08:21:16.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.385+0000 7f038e800640 1 -- 172.21.15.146:0/3427005173 wait complete. 2024-04-24T08:21:16.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 Processor -- start 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 -- start start 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0388102ad0 0x7f0388102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388105400 0x7f0388100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388107d30 0x7f0388100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f038810dce0 con 0x7f0388102ad0 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f038810db60 con 0x7f0388105400 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.386+0000 7f038e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f038810de60 con 0x7f0388107d30 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f0387400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0388102ad0 0x7f0388102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f0387400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0388102ad0 0x7f0388102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39150/0 (socket says 172.21.15.146:39150) 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f0387400640 1 -- 172.21.15.146:0/1758843726 learned_addr learned my addr 172.21.15.146:0/1758843726 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f037fe00640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388105400 0x7f0388100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f0387e00640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388107d30 0x7f0388100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f037fe00640 1 -- 172.21.15.146:0/1758843726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388107d30 msgr2=0x7f0388100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f037fe00640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388107d30 0x7f0388100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f037fe00640 1 -- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0388102ad0 msgr2=0x7f0388102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f037fe00640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0388102ad0 0x7f0388102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f037fe00640 1 -- 172.21.15.146:0/1758843726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0388101850 con 0x7f0388105400 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.387+0000 7f037fe00640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388105400 0x7f0388100a80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0370009230 tx=0x7f0370004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.388+0000 7f0384c00640 1 -- 172.21.15.146:0/1758843726 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037003f070 con 0x7f0388105400 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.388+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03881a2450 con 0x7f0388105400 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.388+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03881a29c0 con 0x7f0388105400 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.388+0000 7f0384c00640 1 -- 172.21.15.146:0/1758843726 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0370003cd0 con 0x7f0388105400 2024-04-24T08:21:16.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.388+0000 7f0384c00640 1 -- 172.21.15.146:0/1758843726 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0370048060 con 0x7f0388105400 2024-04-24T08:21:16.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.389+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03640052c0 con 0x7f0388105400 2024-04-24T08:21:16.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.391+0000 7f0384c00640 1 -- 172.21.15.146:0/1758843726 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f037003b460 con 0x7f0388105400 2024-04-24T08:21:16.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.392+0000 7f0384c00640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f036c0a3e40 0x7f036c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.393+0000 7f0384c00640 1 -- 172.21.15.146:0/1758843726 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f03700f0950 con 0x7f0388105400 2024-04-24T08:21:16.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.393+0000 7f0387400640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f036c0a3e40 0x7f036c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.394+0000 7f0387400640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f036c0a3e40 0x7f036c0a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0378009cd0 tx=0x7f0378005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:16.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.397+0000 7f0384c00640 1 -- 172.21.15.146:0/1758843726 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03700bc7d0 con 0x7f0388105400 2024-04-24T08:21:16.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.624+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0364000bf0 con 0x7f036c0a3e40 2024-04-24T08:21:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.650+0000 7f0384c00640 1 -- 172.21.15.146:0/1758843726 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257773 (secure 0 0 0) 0x7f0364000bf0 con 0x7f036c0a3e40 2024-04-24T08:21:16.656 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-24T08:21:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f036c0a3e40 msgr2=0x7f036c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f036c0a3e40 0x7f036c0a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0378009cd0 tx=0x7f0378005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388105400 msgr2=0x7f0388100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388105400 0x7f0388100a80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0370009230 tx=0x7f0370004270 comp rx=0 tx=0).stop 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 shutdown_connections 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f036c0a3e40 0x7f036c0a6300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0388107d30 0x7f0388100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0388105400 0x7f0388100a80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 --2- 172.21.15.146:0/1758843726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0388102ad0 0x7f0388102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 >> 172.21.15.146:0/1758843726 conn(0x7f03880fc4e0 msgr2=0x7f0388103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 shutdown_connections 2024-04-24T08:21:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.657+0000 7f038e800640 1 -- 172.21.15.146:0/1758843726 wait complete. 2024-04-24T08:21:16.679 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-24T08:21:15.111012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1187744,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1102,"num_read_kb":711,"num_write":474,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377040,"kb_used_data":15312,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124564784,"statfs":{"total":1151940427776,"available":1151554338816,"internally_reserved":0,"allocated":15679488,"data_stored":9212597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.979129"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210587+0000","last_change":"2024-04-24T08:18:47.309666+0000","last_active":"2024-04-24T08:21:11.210587+0000","last_peered":"2024-04-24T08:21:11.210587+0000","last_clean":"2024-04-24T08:21:11.210587+0000","last_became_active":"2024-04-24T08:18:47.309413+0000","last_became_peered":"2024-04-24T08:18:47.309413+0000","last_unstale":"2024-04-24T08:21:11.210587+0000","last_undegraded":"2024-04-24T08:21:11.210587+0000","last_fullsized":"2024-04-24T08:21:11.210587+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:29.662505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416384+0000","last_change":"2024-04-24T08:18:47.310321+0000","last_active":"2024-04-24T08:21:11.416384+0000","last_peered":"2024-04-24T08:21:11.416384+0000","last_clean":"2024-04-24T08:21:11.416384+0000","last_became_active":"2024-04-24T08:18:47.309956+0000","last_became_peered":"2024-04-24T08:18:47.309956+0000","last_unstale":"2024-04-24T08:21:11.416384+0000","last_undegraded":"2024-04-24T08:21:11.416384+0000","last_fullsized":"2024-04-24T08:21:11.416384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:51:48.432725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416668+0000","last_change":"2024-04-24T08:18:47.312153+0000","last_active":"2024-04-24T08:21:11.416668+0000","last_peered":"2024-04-24T08:21:11.416668+0000","last_clean":"2024-04-24T08:21:11.416668+0000","last_became_active":"2024-04-24T08:18:47.309465+0000","last_became_peered":"2024-04-24T08:18:47.309465+0000","last_unstale":"2024-04-24T08:21:11.416668+0000","last_undegraded":"2024-04-24T08:21:11.416668+0000","last_fullsized":"2024-04-24T08:21:11.416668+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:55.829143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089098+0000","last_change":"2024-04-24T08:18:47.297017+0000","last_active":"2024-04-24T08:21:11.089098+0000","last_peered":"2024-04-24T08:21:11.089098+0000","last_clean":"2024-04-24T08:21:11.089098+0000","last_became_active":"2024-04-24T08:18:47.296693+0000","last_became_peered":"2024-04-24T08:18:47.296693+0000","last_unstale":"2024-04-24T08:21:11.089098+0000","last_undegraded":"2024-04-24T08:21:11.089098+0000","last_fullsized":"2024-04-24T08:21:11.089098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:23.400403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089279+0000","last_change":"2024-04-24T08:18:47.315376+0000","last_active":"2024-04-24T08:21:11.089279+0000","last_peered":"2024-04-24T08:21:11.089279+0000","last_clean":"2024-04-24T08:21:11.089279+0000","last_became_active":"2024-04-24T08:18:47.315027+0000","last_became_peered":"2024-04-24T08:18:47.315027+0000","last_unstale":"2024-04-24T08:21:11.089279+0000","last_undegraded":"2024-04-24T08:21:11.089279+0000","last_fullsized":"2024-04-24T08:21:11.089279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:12.504515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222417+0000","last_change":"2024-04-24T08:18:47.306388+0000","last_active":"2024-04-24T08:21:11.222417+0000","last_peered":"2024-04-24T08:21:11.222417+0000","last_clean":"2024-04-24T08:21:11.222417+0000","last_became_active":"2024-04-24T08:18:47.306082+0000","last_became_peered":"2024-04-24T08:18:47.306082+0000","last_unstale":"2024-04-24T08:21:11.222417+0000","last_undegraded":"2024-04-24T08:21:11.222417+0000","last_fullsized":"2024-04-24T08:21:11.222417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:34:52.612799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216773+0000","last_change":"2024-04-24T08:18:47.303336+0000","last_active":"2024-04-24T08:21:11.216773+0000","last_peered":"2024-04-24T08:21:11.216773+0000","last_clean":"2024-04-24T08:21:11.216773+0000","last_became_active":"2024-04-24T08:18:47.303008+0000","last_became_peered":"2024-04-24T08:18:47.303008+0000","last_unstale":"2024-04-24T08:21:11.216773+0000","last_undegraded":"2024-04-24T08:21:11.216773+0000","last_fullsized":"2024-04-24T08:21:11.216773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:57.404693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210549+0000","last_change":"2024-04-24T08:18:47.316672+0000","last_active":"2024-04-24T08:21:11.210549+0000","last_peered":"2024-04-24T08:21:11.210549+0000","last_clean":"2024-04-24T08:21:11.210549+0000","last_became_active":"2024-04-24T08:18:47.315741+0000","last_became_peered":"2024-04-24T08:18:47.315741+0000","last_unstale":"2024-04-24T08:21:11.210549+0000","last_undegraded":"2024-04-24T08:21:11.210549+0000","last_fullsized":"2024-04-24T08:21:11.210549+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:29.156710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409994+0000","last_change":"2024-04-24T08:18:47.291733+0000","last_active":"2024-04-24T08:21:11.409994+0000","last_peered":"2024-04-24T08:21:11.409994+0000","last_clean":"2024-04-24T08:21:11.409994+0000","last_became_active":"2024-04-24T08:18:47.291432+0000","last_became_peered":"2024-04-24T08:18:47.291432+0000","last_unstale":"2024-04-24T08:21:11.409994+0000","last_undegraded":"2024-04-24T08:21:11.409994+0000","last_fullsized":"2024-04-24T08:21:11.409994+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:38:03.601540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222808+0000","last_change":"2024-04-24T08:18:47.294544+0000","last_active":"2024-04-24T08:21:11.222808+0000","last_peered":"2024-04-24T08:21:11.222808+0000","last_clean":"2024-04-24T08:21:11.222808+0000","last_became_active":"2024-04-24T08:18:47.294183+0000","last_became_peered":"2024-04-24T08:18:47.294183+0000","last_unstale":"2024-04-24T08:21:11.222808+0000","last_undegraded":"2024-04-24T08:21:11.222808+0000","last_fullsized":"2024-04-24T08:21:11.222808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:53.082644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217198+0000","last_change":"2024-04-24T08:18:47.292615+0000","last_active":"2024-04-24T08:21:11.217198+0000","last_peered":"2024-04-24T08:21:11.217198+0000","last_clean":"2024-04-24T08:21:11.217198+0000","last_became_active":"2024-04-24T08:18:47.292300+0000","last_became_peered":"2024-04-24T08:18:47.292300+0000","last_unstale":"2024-04-24T08:21:11.217198+0000","last_undegraded":"2024-04-24T08:21:11.217198+0000","last_fullsized":"2024-04-24T08:21:11.217198+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:58.293253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089115+0000","last_change":"2024-04-24T08:18:47.298388+0000","last_active":"2024-04-24T08:21:11.089115+0000","last_peered":"2024-04-24T08:21:11.089115+0000","last_clean":"2024-04-24T08:21:11.089115+0000","last_became_active":"2024-04-24T08:18:47.298008+0000","last_became_peered":"2024-04-24T08:18:47.298008+0000","last_unstale":"2024-04-24T08:21:11.089115+0000","last_undegraded":"2024-04-24T08:21:11.089115+0000","last_fullsized":"2024-04-24T08:21:11.089115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:10:00.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.092591+0000","last_change":"2024-04-24T08:18:47.298809+0000","last_active":"2024-04-24T08:21:11.092591+0000","last_peered":"2024-04-24T08:21:11.092591+0000","last_clean":"2024-04-24T08:21:11.092591+0000","last_became_active":"2024-04-24T08:18:47.298103+0000","last_became_peered":"2024-04-24T08:18:47.298103+0000","last_unstale":"2024-04-24T08:21:11.092591+0000","last_undegraded":"2024-04-24T08:21:11.092591+0000","last_fullsized":"2024-04-24T08:21:11.092591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:38:53.625721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415961+0000","last_change":"2024-04-24T08:18:47.314572+0000","last_active":"2024-04-24T08:21:11.415961+0000","last_peered":"2024-04-24T08:21:11.415961+0000","last_clean":"2024-04-24T08:21:11.415961+0000","last_became_active":"2024-04-24T08:18:47.313372+0000","last_became_peered":"2024-04-24T08:18:47.313372+0000","last_unstale":"2024-04-24T08:21:11.415961+0000","last_undegraded":"2024-04-24T08:21:11.415961+0000","last_fullsized":"2024-04-24T08:21:11.415961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:56:01.954909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415978+0000","last_change":"2024-04-24T08:18:47.314822+0000","last_active":"2024-04-24T08:21:11.415978+0000","last_peered":"2024-04-24T08:21:11.415978+0000","last_clean":"2024-04-24T08:21:11.415978+0000","last_became_active":"2024-04-24T08:18:47.310774+0000","last_became_peered":"2024-04-24T08:18:47.310774+0000","last_unstale":"2024-04-24T08:21:11.415978+0000","last_undegraded":"2024-04-24T08:21:11.415978+0000","last_fullsized":"2024-04-24T08:21:11.415978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:24.934100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409946+0000","last_change":"2024-04-24T08:18:47.307208+0000","last_active":"2024-04-24T08:21:11.409946+0000","last_peered":"2024-04-24T08:21:11.409946+0000","last_clean":"2024-04-24T08:21:11.409946+0000","last_became_active":"2024-04-24T08:18:47.306889+0000","last_became_peered":"2024-04-24T08:18:47.306889+0000","last_unstale":"2024-04-24T08:21:11.409946+0000","last_undegraded":"2024-04-24T08:21:11.409946+0000","last_fullsized":"2024-04-24T08:21:11.409946+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:20:01.438388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210570+0000","last_change":"2024-04-24T08:18:47.322929+0000","last_active":"2024-04-24T08:21:11.210570+0000","last_peered":"2024-04-24T08:21:11.210570+0000","last_clean":"2024-04-24T08:21:11.210570+0000","last_became_active":"2024-04-24T08:18:47.322593+0000","last_became_peered":"2024-04-24T08:18:47.322593+0000","last_unstale":"2024-04-24T08:21:11.210570+0000","last_undegraded":"2024-04-24T08:21:11.210570+0000","last_fullsized":"2024-04-24T08:21:11.210570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:09.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.208062+0000","last_change":"2024-04-24T08:18:47.304886+0000","last_active":"2024-04-24T08:21:11.208062+0000","last_peered":"2024-04-24T08:21:11.208062+0000","last_clean":"2024-04-24T08:21:11.208062+0000","last_became_active":"2024-04-24T08:18:47.302631+0000","last_became_peered":"2024-04-24T08:18:47.302631+0000","last_unstale":"2024-04-24T08:21:11.208062+0000","last_undegraded":"2024-04-24T08:21:11.208062+0000","last_fullsized":"2024-04-24T08:21:11.208062+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:13.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407979+0000","last_change":"2024-04-24T08:18:47.316828+0000","last_active":"2024-04-24T08:21:11.407979+0000","last_peered":"2024-04-24T08:21:11.407979+0000","last_clean":"2024-04-24T08:21:11.407979+0000","last_became_active":"2024-04-24T08:18:47.316540+0000","last_became_peered":"2024-04-24T08:18:47.316540+0000","last_unstale":"2024-04-24T08:21:11.407979+0000","last_undegraded":"2024-04-24T08:21:11.407979+0000","last_fullsized":"2024-04-24T08:21:11.407979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:41:42.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207816+0000","last_change":"2024-04-24T08:18:46.444744+0000","last_active":"2024-04-24T08:21:11.207816+0000","last_peered":"2024-04-24T08:21:11.207816+0000","last_clean":"2024-04-24T08:21:11.207816+0000","last_became_active":"2024-04-24T08:18:46.442495+0000","last_became_peered":"2024-04-24T08:18:46.442495+0000","last_unstale":"2024-04-24T08:21:11.207816+0000","last_undegraded":"2024-04-24T08:21:11.207816+0000","last_fullsized":"2024-04-24T08:21:11.207816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:33:32.270984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211523+0000","last_change":"2024-04-24T08:18:47.320462+0000","last_active":"2024-04-24T08:21:11.211523+0000","last_peered":"2024-04-24T08:21:11.211523+0000","last_clean":"2024-04-24T08:21:11.211523+0000","last_became_active":"2024-04-24T08:18:47.319267+0000","last_became_peered":"2024-04-24T08:18:47.319267+0000","last_unstale":"2024-04-24T08:21:11.211523+0000","last_undegraded":"2024-04-24T08:21:11.211523+0000","last_fullsized":"2024-04-24T08:21:11.211523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:50:15.882077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211656+0000","last_change":"2024-04-24T08:18:46.453278+0000","last_active":"2024-04-24T08:21:11.211656+0000","last_peered":"2024-04-24T08:21:11.211656+0000","last_clean":"2024-04-24T08:21:11.211656+0000","last_became_active":"2024-04-24T08:18:46.452913+0000","last_became_peered":"2024-04-24T08:18:46.452913+0000","last_unstale":"2024-04-24T08:21:11.211656+0000","last_undegraded":"2024-04-24T08:21:11.211656+0000","last_fullsized":"2024-04-24T08:21:11.211656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:48.272190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416239+0000","last_change":"2024-04-24T08:18:47.323610+0000","last_active":"2024-04-24T08:21:11.416239+0000","last_peered":"2024-04-24T08:21:11.416239+0000","last_clean":"2024-04-24T08:21:11.416239+0000","last_became_active":"2024-04-24T08:18:47.322795+0000","last_became_peered":"2024-04-24T08:18:47.322795+0000","last_unstale":"2024-04-24T08:21:11.416239+0000","last_undegraded":"2024-04-24T08:21:11.416239+0000","last_fullsized":"2024-04-24T08:21:11.416239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:32.951564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089694+0000","last_change":"2024-04-24T08:18:57.730353+0000","last_active":"2024-04-24T08:21:11.089694+0000","last_peered":"2024-04-24T08:21:11.089694+0000","last_clean":"2024-04-24T08:21:11.089694+0000","last_became_active":"2024-04-24T08:18:57.725936+0000","last_became_peered":"2024-04-24T08:18:57.725936+0000","last_unstale":"2024-04-24T08:21:11.089694+0000","last_undegraded":"2024-04-24T08:21:11.089694+0000","last_fullsized":"2024-04-24T08:21:11.089694+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:00:27.764723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410161+0000","last_change":"2024-04-24T08:18:47.304884+0000","last_active":"2024-04-24T08:21:11.410161+0000","last_peered":"2024-04-24T08:21:11.410161+0000","last_clean":"2024-04-24T08:21:11.410161+0000","last_became_active":"2024-04-24T08:18:47.304520+0000","last_became_peered":"2024-04-24T08:18:47.304520+0000","last_unstale":"2024-04-24T08:21:11.410161+0000","last_undegraded":"2024-04-24T08:21:11.410161+0000","last_fullsized":"2024-04-24T08:21:11.410161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:42.325010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089422+0000","last_change":"2024-04-24T08:18:46.441073+0000","last_active":"2024-04-24T08:21:11.089422+0000","last_peered":"2024-04-24T08:21:11.089422+0000","last_clean":"2024-04-24T08:21:11.089422+0000","last_became_active":"2024-04-24T08:18:46.440721+0000","last_became_peered":"2024-04-24T08:18:46.440721+0000","last_unstale":"2024-04-24T08:21:11.089422+0000","last_undegraded":"2024-04-24T08:21:11.089422+0000","last_fullsized":"2024-04-24T08:21:11.089422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:51:39.516458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219409+0000","last_change":"2024-04-24T08:18:47.293947+0000","last_active":"2024-04-24T08:21:11.219409+0000","last_peered":"2024-04-24T08:21:11.219409+0000","last_clean":"2024-04-24T08:21:11.219409+0000","last_became_active":"2024-04-24T08:18:47.293623+0000","last_became_peered":"2024-04-24T08:18:47.293623+0000","last_unstale":"2024-04-24T08:21:11.219409+0000","last_undegraded":"2024-04-24T08:21:11.219409+0000","last_fullsized":"2024-04-24T08:21:11.219409+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:23.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.741488+0000","last_change":"2024-04-24T08:18:46.439909+0000","last_active":"2024-04-24T08:21:11.741488+0000","last_peered":"2024-04-24T08:21:11.741488+0000","last_clean":"2024-04-24T08:21:11.741488+0000","last_became_active":"2024-04-24T08:18:46.439597+0000","last_became_peered":"2024-04-24T08:18:46.439597+0000","last_unstale":"2024-04-24T08:21:11.741488+0000","last_undegraded":"2024-04-24T08:21:11.741488+0000","last_fullsized":"2024-04-24T08:21:11.741488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:49:11.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089065+0000","last_change":"2024-04-24T08:18:47.306480+0000","last_active":"2024-04-24T08:21:11.089065+0000","last_peered":"2024-04-24T08:21:11.089065+0000","last_clean":"2024-04-24T08:21:11.089065+0000","last_became_active":"2024-04-24T08:18:47.306104+0000","last_became_peered":"2024-04-24T08:18:47.306104+0000","last_unstale":"2024-04-24T08:21:11.089065+0000","last_undegraded":"2024-04-24T08:21:11.089065+0000","last_fullsized":"2024-04-24T08:21:11.089065+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:56.340836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222262+0000","last_change":"2024-04-24T08:18:46.439381+0000","last_active":"2024-04-24T08:21:11.222262+0000","last_peered":"2024-04-24T08:21:11.222262+0000","last_clean":"2024-04-24T08:21:11.222262+0000","last_became_active":"2024-04-24T08:18:46.439033+0000","last_became_peered":"2024-04-24T08:18:46.439033+0000","last_unstale":"2024-04-24T08:21:11.222262+0000","last_undegraded":"2024-04-24T08:21:11.222262+0000","last_fullsized":"2024-04-24T08:21:11.222262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:48:28.833066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219251+0000","last_change":"2024-04-24T08:18:57.595549+0000","last_active":"2024-04-24T08:21:11.219251+0000","last_peered":"2024-04-24T08:21:11.219251+0000","last_clean":"2024-04-24T08:21:11.219251+0000","last_became_active":"2024-04-24T08:18:57.595098+0000","last_became_peered":"2024-04-24T08:18:57.595098+0000","last_unstale":"2024-04-24T08:21:11.219251+0000","last_undegraded":"2024-04-24T08:21:11.219251+0000","last_fullsized":"2024-04-24T08:21:11.219251+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:09:22.722996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.094360+0000","last_change":"2024-04-24T08:18:46.448151+0000","last_active":"2024-04-24T08:21:11.094360+0000","last_peered":"2024-04-24T08:21:11.094360+0000","last_clean":"2024-04-24T08:21:11.094360+0000","last_became_active":"2024-04-24T08:18:46.447077+0000","last_became_peered":"2024-04-24T08:18:46.447077+0000","last_unstale":"2024-04-24T08:21:11.094360+0000","last_undegraded":"2024-04-24T08:21:11.094360+0000","last_fullsized":"2024-04-24T08:21:11.094360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:00.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210379+0000","last_change":"2024-04-24T08:18:47.319423+0000","last_active":"2024-04-24T08:21:11.210379+0000","last_peered":"2024-04-24T08:21:11.210379+0000","last_clean":"2024-04-24T08:21:11.210379+0000","last_became_active":"2024-04-24T08:18:47.316284+0000","last_became_peered":"2024-04-24T08:18:47.316284+0000","last_unstale":"2024-04-24T08:21:11.210379+0000","last_undegraded":"2024-04-24T08:21:11.210379+0000","last_fullsized":"2024-04-24T08:21:11.210379+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:55.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219135+0000","last_change":"2024-04-24T08:18:46.436152+0000","last_active":"2024-04-24T08:21:11.219135+0000","last_peered":"2024-04-24T08:21:11.219135+0000","last_clean":"2024-04-24T08:21:11.219135+0000","last_became_active":"2024-04-24T08:18:46.435835+0000","last_became_peered":"2024-04-24T08:18:46.435835+0000","last_unstale":"2024-04-24T08:21:11.219135+0000","last_undegraded":"2024-04-24T08:21:11.219135+0000","last_fullsized":"2024-04-24T08:21:11.219135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:12:30.536455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207423+0000","last_change":"2024-04-24T08:18:57.725728+0000","last_active":"2024-04-24T08:21:11.207423+0000","last_peered":"2024-04-24T08:21:11.207423+0000","last_clean":"2024-04-24T08:21:11.207423+0000","last_became_active":"2024-04-24T08:18:57.724704+0000","last_became_peered":"2024-04-24T08:18:57.724704+0000","last_unstale":"2024-04-24T08:21:11.207423+0000","last_undegraded":"2024-04-24T08:21:11.207423+0000","last_fullsized":"2024-04-24T08:21:11.207423+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:27.501006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":167,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.740986+0000","last_change":"2024-04-24T08:18:46.446652+0000","last_active":"2024-04-24T08:21:11.740986+0000","last_peered":"2024-04-24T08:21:11.740986+0000","last_clean":"2024-04-24T08:21:11.740986+0000","last_became_active":"2024-04-24T08:18:46.446348+0000","last_became_peered":"2024-04-24T08:18:46.446348+0000","last_unstale":"2024-04-24T08:21:11.740986+0000","last_undegraded":"2024-04-24T08:21:11.740986+0000","last_fullsized":"2024-04-24T08:21:11.740986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:25.791359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410343+0000","last_change":"2024-04-24T08:18:47.309072+0000","last_active":"2024-04-24T08:21:11.410343+0000","last_peered":"2024-04-24T08:21:11.410343+0000","last_clean":"2024-04-24T08:21:11.410343+0000","last_became_active":"2024-04-24T08:18:47.308729+0000","last_became_peered":"2024-04-24T08:18:47.308729+0000","last_unstale":"2024-04-24T08:21:11.410343+0000","last_undegraded":"2024-04-24T08:21:11.410343+0000","last_fullsized":"2024-04-24T08:21:11.410343+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:58.256876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217308+0000","last_change":"2024-04-24T08:18:46.439190+0000","last_active":"2024-04-24T08:21:11.217308+0000","last_peered":"2024-04-24T08:21:11.217308+0000","last_clean":"2024-04-24T08:21:11.217308+0000","last_became_active":"2024-04-24T08:18:46.438442+0000","last_became_peered":"2024-04-24T08:18:46.438442+0000","last_unstale":"2024-04-24T08:21:11.217308+0000","last_undegraded":"2024-04-24T08:21:11.217308+0000","last_fullsized":"2024-04-24T08:21:11.217308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:42.614563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416325+0000","last_change":"2024-04-24T08:18:47.311447+0000","last_active":"2024-04-24T08:21:11.416325+0000","last_peered":"2024-04-24T08:21:11.416325+0000","last_clean":"2024-04-24T08:21:11.416325+0000","last_became_active":"2024-04-24T08:18:47.310939+0000","last_became_peered":"2024-04-24T08:18:47.310939+0000","last_unstale":"2024-04-24T08:21:11.416325+0000","last_undegraded":"2024-04-24T08:21:11.416325+0000","last_fullsized":"2024-04-24T08:21:11.416325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:42.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219601+0000","last_change":"2024-04-24T08:18:46.450596+0000","last_active":"2024-04-24T08:21:11.219601+0000","last_peered":"2024-04-24T08:21:11.219601+0000","last_clean":"2024-04-24T08:21:11.219601+0000","last_became_active":"2024-04-24T08:18:46.450226+0000","last_became_peered":"2024-04-24T08:18:46.450226+0000","last_unstale":"2024-04-24T08:21:11.219601+0000","last_undegraded":"2024-04-24T08:21:11.219601+0000","last_fullsized":"2024-04-24T08:21:11.219601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:05.722507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409988+0000","last_change":"2024-04-24T08:18:47.295266+0000","last_active":"2024-04-24T08:21:11.409988+0000","last_peered":"2024-04-24T08:21:11.409988+0000","last_clean":"2024-04-24T08:21:11.409988+0000","last_became_active":"2024-04-24T08:18:47.294963+0000","last_became_peered":"2024-04-24T08:18:47.294963+0000","last_unstale":"2024-04-24T08:21:11.409988+0000","last_undegraded":"2024-04-24T08:21:11.409988+0000","last_fullsized":"2024-04-24T08:21:11.409988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:07.638921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":348,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.746361+0000","last_change":"2024-04-24T08:18:46.435524+0000","last_active":"2024-04-24T08:21:11.746361+0000","last_peered":"2024-04-24T08:21:11.746361+0000","last_clean":"2024-04-24T08:21:11.746361+0000","last_became_active":"2024-04-24T08:18:46.435212+0000","last_became_peered":"2024-04-24T08:18:46.435212+0000","last_unstale":"2024-04-24T08:21:11.746361+0000","last_undegraded":"2024-04-24T08:21:11.746361+0000","last_fullsized":"2024-04-24T08:21:11.746361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:45.575905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66701,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":291,"num_read_kb":210,"num_write":36,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089464+0000","last_change":"2024-04-24T08:18:47.293648+0000","last_active":"2024-04-24T08:21:11.089464+0000","last_peered":"2024-04-24T08:21:11.089464+0000","last_clean":"2024-04-24T08:21:11.089464+0000","last_became_active":"2024-04-24T08:18:47.293260+0000","last_became_peered":"2024-04-24T08:18:47.293260+0000","last_unstale":"2024-04-24T08:21:11.089464+0000","last_undegraded":"2024-04-24T08:21:11.089464+0000","last_fullsized":"2024-04-24T08:21:11.089464+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:36.462606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408233+0000","last_change":"2024-04-24T08:18:46.441401+0000","last_active":"2024-04-24T08:21:11.408233+0000","last_peered":"2024-04-24T08:21:11.408233+0000","last_clean":"2024-04-24T08:21:11.408233+0000","last_became_active":"2024-04-24T08:18:46.440985+0000","last_became_peered":"2024-04-24T08:18:46.440985+0000","last_unstale":"2024-04-24T08:21:11.408233+0000","last_undegraded":"2024-04-24T08:21:11.408233+0000","last_fullsized":"2024-04-24T08:21:11.408233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:45.201518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089310+0000","last_change":"2024-04-24T08:18:47.320928+0000","last_active":"2024-04-24T08:21:11.089310+0000","last_peered":"2024-04-24T08:21:11.089310+0000","last_clean":"2024-04-24T08:21:11.089310+0000","last_became_active":"2024-04-24T08:18:47.320515+0000","last_became_peered":"2024-04-24T08:18:47.320515+0000","last_unstale":"2024-04-24T08:21:11.089310+0000","last_undegraded":"2024-04-24T08:21:11.089310+0000","last_fullsized":"2024-04-24T08:21:11.089310+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:55:42.899438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219120+0000","last_change":"2024-04-24T08:18:46.437677+0000","last_active":"2024-04-24T08:21:11.219120+0000","last_peered":"2024-04-24T08:21:11.219120+0000","last_clean":"2024-04-24T08:21:11.219120+0000","last_became_active":"2024-04-24T08:18:46.437364+0000","last_became_peered":"2024-04-24T08:18:46.437364+0000","last_unstale":"2024-04-24T08:21:11.219120+0000","last_undegraded":"2024-04-24T08:21:11.219120+0000","last_fullsized":"2024-04-24T08:21:11.219120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:25:55.408887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217154+0000","last_change":"2024-04-24T08:18:47.298396+0000","last_active":"2024-04-24T08:21:11.217154+0000","last_peered":"2024-04-24T08:21:11.217154+0000","last_clean":"2024-04-24T08:21:11.217154+0000","last_became_active":"2024-04-24T08:18:47.298021+0000","last_became_peered":"2024-04-24T08:18:47.298021+0000","last_unstale":"2024-04-24T08:21:11.217154+0000","last_undegraded":"2024-04-24T08:21:11.217154+0000","last_fullsized":"2024-04-24T08:21:11.217154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:24.015897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":170,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.830237+0000","last_change":"2024-04-24T08:18:46.436612+0000","last_active":"2024-04-24T08:21:11.830237+0000","last_peered":"2024-04-24T08:21:11.830237+0000","last_clean":"2024-04-24T08:21:11.830237+0000","last_became_active":"2024-04-24T08:18:46.436211+0000","last_became_peered":"2024-04-24T08:18:46.436211+0000","last_unstale":"2024-04-24T08:21:11.830237+0000","last_undegraded":"2024-04-24T08:21:11.830237+0000","last_fullsized":"2024-04-24T08:21:11.830237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:49:05.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217055+0000","last_change":"2024-04-24T08:18:47.290849+0000","last_active":"2024-04-24T08:21:11.217055+0000","last_peered":"2024-04-24T08:21:11.217055+0000","last_clean":"2024-04-24T08:21:11.217055+0000","last_became_active":"2024-04-24T08:18:47.290514+0000","last_became_peered":"2024-04-24T08:18:47.290514+0000","last_unstale":"2024-04-24T08:21:11.217055+0000","last_undegraded":"2024-04-24T08:21:11.217055+0000","last_fullsized":"2024-04-24T08:21:11.217055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:39:12.527477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089092+0000","last_change":"2024-04-24T08:18:46.451815+0000","last_active":"2024-04-24T08:21:11.089092+0000","last_peered":"2024-04-24T08:21:11.089092+0000","last_clean":"2024-04-24T08:21:11.089092+0000","last_became_active":"2024-04-24T08:18:46.451478+0000","last_became_peered":"2024-04-24T08:18:46.451478+0000","last_unstale":"2024-04-24T08:21:11.089092+0000","last_undegraded":"2024-04-24T08:21:11.089092+0000","last_fullsized":"2024-04-24T08:21:11.089092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:54:22.907143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.218992+0000","last_change":"2024-04-24T08:18:47.295181+0000","last_active":"2024-04-24T08:21:11.218992+0000","last_peered":"2024-04-24T08:21:11.218992+0000","last_clean":"2024-04-24T08:21:11.218992+0000","last_became_active":"2024-04-24T08:18:47.294842+0000","last_became_peered":"2024-04-24T08:18:47.294842+0000","last_unstale":"2024-04-24T08:21:11.218992+0000","last_undegraded":"2024-04-24T08:21:11.218992+0000","last_fullsized":"2024-04-24T08:21:11.218992+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:37.058025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.537280+0000","last_change":"2024-04-24T08:18:46.437967+0000","last_active":"2024-04-24T08:21:11.537280+0000","last_peered":"2024-04-24T08:21:11.537280+0000","last_clean":"2024-04-24T08:21:11.537280+0000","last_became_active":"2024-04-24T08:18:46.437617+0000","last_became_peered":"2024-04-24T08:18:46.437617+0000","last_unstale":"2024-04-24T08:21:11.537280+0000","last_undegraded":"2024-04-24T08:21:11.537280+0000","last_fullsized":"2024-04-24T08:21:11.537280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:07.303827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210446+0000","last_change":"2024-04-24T08:18:57.726323+0000","last_active":"2024-04-24T08:21:11.210446+0000","last_peered":"2024-04-24T08:21:11.210446+0000","last_clean":"2024-04-24T08:21:11.210446+0000","last_became_active":"2024-04-24T08:18:57.725924+0000","last_became_peered":"2024-04-24T08:18:57.725924+0000","last_unstale":"2024-04-24T08:21:11.210446+0000","last_undegraded":"2024-04-24T08:21:11.210446+0000","last_fullsized":"2024-04-24T08:21:11.210446+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:15:33.756804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207766+0000","last_change":"2024-04-24T08:18:57.591169+0000","last_active":"2024-04-24T08:21:11.207766+0000","last_peered":"2024-04-24T08:21:11.207766+0000","last_clean":"2024-04-24T08:21:11.207766+0000","last_became_active":"2024-04-24T08:18:57.590300+0000","last_became_peered":"2024-04-24T08:18:57.590300+0000","last_unstale":"2024-04-24T08:21:11.207766+0000","last_undegraded":"2024-04-24T08:21:11.207766+0000","last_fullsized":"2024-04-24T08:21:11.207766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:13.632780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222625+0000","last_change":"2024-04-24T08:18:47.316791+0000","last_active":"2024-04-24T08:21:11.222625+0000","last_peered":"2024-04-24T08:21:11.222625+0000","last_clean":"2024-04-24T08:21:11.222625+0000","last_became_active":"2024-04-24T08:18:47.316486+0000","last_became_peered":"2024-04-24T08:18:47.316486+0000","last_unstale":"2024-04-24T08:21:11.222625+0000","last_undegraded":"2024-04-24T08:21:11.222625+0000","last_fullsized":"2024-04-24T08:21:11.222625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:36:37.190732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217170+0000","last_change":"2024-04-24T08:18:46.432831+0000","last_active":"2024-04-24T08:21:11.217170+0000","last_peered":"2024-04-24T08:21:11.217170+0000","last_clean":"2024-04-24T08:21:11.217170+0000","last_became_active":"2024-04-24T08:18:46.432483+0000","last_became_peered":"2024-04-24T08:18:46.432483+0000","last_unstale":"2024-04-24T08:21:11.217170+0000","last_undegraded":"2024-04-24T08:21:11.217170+0000","last_fullsized":"2024-04-24T08:21:11.217170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:43:24.802099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222393+0000","last_change":"2024-04-24T08:18:47.309408+0000","last_active":"2024-04-24T08:21:11.222393+0000","last_peered":"2024-04-24T08:21:11.222393+0000","last_clean":"2024-04-24T08:21:11.222393+0000","last_became_active":"2024-04-24T08:18:47.309083+0000","last_became_peered":"2024-04-24T08:18:47.309083+0000","last_unstale":"2024-04-24T08:21:11.222393+0000","last_undegraded":"2024-04-24T08:21:11.222393+0000","last_fullsized":"2024-04-24T08:21:11.222393+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:15:44.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219335+0000","last_change":"2024-04-24T08:18:47.291060+0000","last_active":"2024-04-24T08:21:11.219335+0000","last_peered":"2024-04-24T08:21:11.219335+0000","last_clean":"2024-04-24T08:21:11.219335+0000","last_became_active":"2024-04-24T08:18:47.290635+0000","last_became_peered":"2024-04-24T08:18:47.290635+0000","last_unstale":"2024-04-24T08:21:11.219335+0000","last_undegraded":"2024-04-24T08:21:11.219335+0000","last_fullsized":"2024-04-24T08:21:11.219335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:20.372853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":167,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415606+0000","last_change":"2024-04-24T08:18:46.449850+0000","last_active":"2024-04-24T08:21:11.415606+0000","last_peered":"2024-04-24T08:21:11.415606+0000","last_clean":"2024-04-24T08:21:11.415606+0000","last_became_active":"2024-04-24T08:18:46.448486+0000","last_became_peered":"2024-04-24T08:18:46.448486+0000","last_unstale":"2024-04-24T08:21:11.415606+0000","last_undegraded":"2024-04-24T08:21:11.415606+0000","last_fullsized":"2024-04-24T08:21:11.415606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:54.611867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211035+0000","last_change":"2024-04-24T08:18:47.328051+0000","last_active":"2024-04-24T08:21:11.211035+0000","last_peered":"2024-04-24T08:21:11.211035+0000","last_clean":"2024-04-24T08:21:11.211035+0000","last_became_active":"2024-04-24T08:18:47.327474+0000","last_became_peered":"2024-04-24T08:18:47.327474+0000","last_unstale":"2024-04-24T08:21:11.211035+0000","last_undegraded":"2024-04-24T08:21:11.211035+0000","last_fullsized":"2024-04-24T08:21:11.211035+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:07.577741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.088963+0000","last_change":"2024-04-24T08:18:46.444375+0000","last_active":"2024-04-24T08:21:11.088963+0000","last_peered":"2024-04-24T08:21:11.088963+0000","last_clean":"2024-04-24T08:21:11.088963+0000","last_became_active":"2024-04-24T08:18:46.443963+0000","last_became_peered":"2024-04-24T08:18:46.443963+0000","last_unstale":"2024-04-24T08:21:11.088963+0000","last_undegraded":"2024-04-24T08:21:11.088963+0000","last_fullsized":"2024-04-24T08:21:11.088963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:22.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216787+0000","last_change":"2024-04-24T08:17:29.632219+0000","last_active":"2024-04-24T08:21:11.216787+0000","last_peered":"2024-04-24T08:21:11.216787+0000","last_clean":"2024-04-24T08:21:11.216787+0000","last_became_active":"2024-04-24T08:17:29.631841+0000","last_became_peered":"2024-04-24T08:17:29.631841+0000","last_unstale":"2024-04-24T08:21:11.216787+0000","last_undegraded":"2024-04-24T08:21:11.216787+0000","last_fullsized":"2024-04-24T08:21:11.216787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_clean_scrub_stamp":"2024-04-24T08:13:41.228801+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:02:55.415959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089104+0000","last_change":"2024-04-24T08:18:47.301876+0000","last_active":"2024-04-24T08:21:11.089104+0000","last_peered":"2024-04-24T08:21:11.089104+0000","last_clean":"2024-04-24T08:21:11.089104+0000","last_became_active":"2024-04-24T08:18:47.301545+0000","last_became_peered":"2024-04-24T08:18:47.301545+0000","last_unstale":"2024-04-24T08:21:11.089104+0000","last_undegraded":"2024-04-24T08:21:11.089104+0000","last_fullsized":"2024-04-24T08:21:11.089104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:34.650014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216699+0000","last_change":"2024-04-24T08:18:46.437915+0000","last_active":"2024-04-24T08:21:11.216699+0000","last_peered":"2024-04-24T08:21:11.216699+0000","last_clean":"2024-04-24T08:21:11.216699+0000","last_became_active":"2024-04-24T08:18:46.437567+0000","last_became_peered":"2024-04-24T08:18:46.437567+0000","last_unstale":"2024-04-24T08:21:11.216699+0000","last_undegraded":"2024-04-24T08:21:11.216699+0000","last_fullsized":"2024-04-24T08:21:11.216699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:36.801535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089647+0000","last_change":"2024-04-24T08:18:46.451834+0000","last_active":"2024-04-24T08:21:11.089647+0000","last_peered":"2024-04-24T08:21:11.089647+0000","last_clean":"2024-04-24T08:21:11.089647+0000","last_became_active":"2024-04-24T08:18:46.451505+0000","last_became_peered":"2024-04-24T08:18:46.451505+0000","last_unstale":"2024-04-24T08:21:11.089647+0000","last_undegraded":"2024-04-24T08:21:11.089647+0000","last_fullsized":"2024-04-24T08:21:11.089647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:26.468176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089798+0000","last_change":"2024-04-24T08:18:47.310490+0000","last_active":"2024-04-24T08:21:11.089798+0000","last_peered":"2024-04-24T08:21:11.089798+0000","last_clean":"2024-04-24T08:21:11.089798+0000","last_became_active":"2024-04-24T08:18:47.309615+0000","last_became_peered":"2024-04-24T08:18:47.309615+0000","last_unstale":"2024-04-24T08:21:11.089798+0000","last_undegraded":"2024-04-24T08:21:11.089798+0000","last_fullsized":"2024-04-24T08:21:11.089798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:22:55.063485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'46","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.354674+0000","last_change":"2024-04-24T08:18:57.602098+0000","last_active":"2024-04-24T08:21:11.354674+0000","last_peered":"2024-04-24T08:21:11.354674+0000","last_clean":"2024-04-24T08:21:11.354674+0000","last_became_active":"2024-04-24T08:18:57.595899+0000","last_became_peered":"2024-04-24T08:18:57.595899+0000","last_unstale":"2024-04-24T08:21:11.354674+0000","last_undegraded":"2024-04-24T08:21:11.354674+0000","last_fullsized":"2024-04-24T08:21:11.354674+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:31.951803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211883+0000","last_change":"2024-04-24T08:18:47.319022+0000","last_active":"2024-04-24T08:21:11.211883+0000","last_peered":"2024-04-24T08:21:11.211883+0000","last_clean":"2024-04-24T08:21:11.211883+0000","last_became_active":"2024-04-24T08:18:47.318167+0000","last_became_peered":"2024-04-24T08:18:47.318167+0000","last_unstale":"2024-04-24T08:21:11.211883+0000","last_undegraded":"2024-04-24T08:21:11.211883+0000","last_fullsized":"2024-04-24T08:21:11.211883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:18.980856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089397+0000","last_change":"2024-04-24T08:18:46.433018+0000","last_active":"2024-04-24T08:21:11.089397+0000","last_peered":"2024-04-24T08:21:11.089397+0000","last_clean":"2024-04-24T08:21:11.089397+0000","last_became_active":"2024-04-24T08:18:46.432681+0000","last_became_peered":"2024-04-24T08:18:46.432681+0000","last_unstale":"2024-04-24T08:21:11.089397+0000","last_undegraded":"2024-04-24T08:21:11.089397+0000","last_fullsized":"2024-04-24T08:21:11.089397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:53.691684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416092+0000","last_change":"2024-04-24T08:18:47.326055+0000","last_active":"2024-04-24T08:21:11.416092+0000","last_peered":"2024-04-24T08:21:11.416092+0000","last_clean":"2024-04-24T08:21:11.416092+0000","last_became_active":"2024-04-24T08:18:47.324956+0000","last_became_peered":"2024-04-24T08:18:47.324956+0000","last_unstale":"2024-04-24T08:21:11.416092+0000","last_undegraded":"2024-04-24T08:21:11.416092+0000","last_fullsized":"2024-04-24T08:21:11.416092+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:44:37.431264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410298+0000","last_change":"2024-04-24T08:18:46.450710+0000","last_active":"2024-04-24T08:21:11.410298+0000","last_peered":"2024-04-24T08:21:11.410298+0000","last_clean":"2024-04-24T08:21:11.410298+0000","last_became_active":"2024-04-24T08:18:46.450391+0000","last_became_peered":"2024-04-24T08:18:46.450391+0000","last_unstale":"2024-04-24T08:21:11.410298+0000","last_undegraded":"2024-04-24T08:21:11.410298+0000","last_fullsized":"2024-04-24T08:21:11.410298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:19.905524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089025+0000","last_change":"2024-04-24T08:18:47.292114+0000","last_active":"2024-04-24T08:21:11.089025+0000","last_peered":"2024-04-24T08:21:11.089025+0000","last_clean":"2024-04-24T08:21:11.089025+0000","last_became_active":"2024-04-24T08:18:47.291787+0000","last_became_peered":"2024-04-24T08:18:47.291787+0000","last_unstale":"2024-04-24T08:21:11.089025+0000","last_undegraded":"2024-04-24T08:21:11.089025+0000","last_fullsized":"2024-04-24T08:21:11.089025+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:37.020368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210671+0000","last_change":"2024-04-24T08:18:46.446027+0000","last_active":"2024-04-24T08:21:11.210671+0000","last_peered":"2024-04-24T08:21:11.210671+0000","last_clean":"2024-04-24T08:21:11.210671+0000","last_became_active":"2024-04-24T08:18:46.445634+0000","last_became_peered":"2024-04-24T08:18:46.445634+0000","last_unstale":"2024-04-24T08:21:11.210671+0000","last_undegraded":"2024-04-24T08:21:11.210671+0000","last_fullsized":"2024-04-24T08:21:11.210671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:46.208560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210162+0000","last_change":"2024-04-24T08:18:47.307800+0000","last_active":"2024-04-24T08:21:11.210162+0000","last_peered":"2024-04-24T08:21:11.210162+0000","last_clean":"2024-04-24T08:21:11.210162+0000","last_became_active":"2024-04-24T08:18:47.307501+0000","last_became_peered":"2024-04-24T08:18:47.307501+0000","last_unstale":"2024-04-24T08:21:11.210162+0000","last_undegraded":"2024-04-24T08:21:11.210162+0000","last_fullsized":"2024-04-24T08:21:11.210162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:04:05.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408128+0000","last_change":"2024-04-24T08:18:46.442811+0000","last_active":"2024-04-24T08:21:11.408128+0000","last_peered":"2024-04-24T08:21:11.408128+0000","last_clean":"2024-04-24T08:21:11.408128+0000","last_became_active":"2024-04-24T08:18:46.442479+0000","last_became_peered":"2024-04-24T08:18:46.442479+0000","last_unstale":"2024-04-24T08:21:11.408128+0000","last_undegraded":"2024-04-24T08:21:11.408128+0000","last_fullsized":"2024-04-24T08:21:11.408128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:24:07.236529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210897+0000","last_change":"2024-04-24T08:18:47.325046+0000","last_active":"2024-04-24T08:21:11.210897+0000","last_peered":"2024-04-24T08:21:11.210897+0000","last_clean":"2024-04-24T08:21:11.210897+0000","last_became_active":"2024-04-24T08:18:47.324720+0000","last_became_peered":"2024-04-24T08:18:47.324720+0000","last_unstale":"2024-04-24T08:21:11.210897+0000","last_undegraded":"2024-04-24T08:21:11.210897+0000","last_fullsized":"2024-04-24T08:21:11.210897+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:11.240388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089232+0000","last_change":"2024-04-24T08:18:46.440572+0000","last_active":"2024-04-24T08:21:11.089232+0000","last_peered":"2024-04-24T08:21:11.089232+0000","last_clean":"2024-04-24T08:21:11.089232+0000","last_became_active":"2024-04-24T08:18:46.440216+0000","last_became_peered":"2024-04-24T08:18:46.440216+0000","last_unstale":"2024-04-24T08:21:11.089232+0000","last_undegraded":"2024-04-24T08:21:11.089232+0000","last_fullsized":"2024-04-24T08:21:11.089232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:51.352874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217093+0000","last_change":"2024-04-24T08:18:47.293465+0000","last_active":"2024-04-24T08:21:11.217093+0000","last_peered":"2024-04-24T08:21:11.217093+0000","last_clean":"2024-04-24T08:21:11.217093+0000","last_became_active":"2024-04-24T08:18:47.293062+0000","last_became_peered":"2024-04-24T08:18:47.293062+0000","last_unstale":"2024-04-24T08:21:11.217093+0000","last_undegraded":"2024-04-24T08:21:11.217093+0000","last_fullsized":"2024-04-24T08:21:11.217093+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:36.029511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408270+0000","last_change":"2024-04-24T08:18:46.451007+0000","last_active":"2024-04-24T08:21:11.408270+0000","last_peered":"2024-04-24T08:21:11.408270+0000","last_clean":"2024-04-24T08:21:11.408270+0000","last_became_active":"2024-04-24T08:18:46.450676+0000","last_became_peered":"2024-04-24T08:18:46.450676+0000","last_unstale":"2024-04-24T08:21:11.408270+0000","last_undegraded":"2024-04-24T08:21:11.408270+0000","last_fullsized":"2024-04-24T08:21:11.408270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:40:00.329311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217356+0000","last_change":"2024-04-24T08:18:47.293859+0000","last_active":"2024-04-24T08:21:11.217356+0000","last_peered":"2024-04-24T08:21:11.217356+0000","last_clean":"2024-04-24T08:21:11.217356+0000","last_became_active":"2024-04-24T08:18:47.293509+0000","last_became_peered":"2024-04-24T08:18:47.293509+0000","last_unstale":"2024-04-24T08:21:11.217356+0000","last_undegraded":"2024-04-24T08:21:11.217356+0000","last_fullsized":"2024-04-24T08:21:11.217356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:41.224037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.356078+0000","last_change":"2024-04-24T08:18:46.436661+0000","last_active":"2024-04-24T08:21:11.356078+0000","last_peered":"2024-04-24T08:21:11.356078+0000","last_clean":"2024-04-24T08:21:11.356078+0000","last_became_active":"2024-04-24T08:18:46.436353+0000","last_became_peered":"2024-04-24T08:18:46.436353+0000","last_unstale":"2024-04-24T08:21:11.356078+0000","last_undegraded":"2024-04-24T08:21:11.356078+0000","last_fullsized":"2024-04-24T08:21:11.356078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:57:25.559706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217135+0000","last_change":"2024-04-24T08:18:57.597302+0000","last_active":"2024-04-24T08:21:11.217135+0000","last_peered":"2024-04-24T08:21:11.217135+0000","last_clean":"2024-04-24T08:21:11.217135+0000","last_became_active":"2024-04-24T08:18:57.591420+0000","last_became_peered":"2024-04-24T08:18:57.591420+0000","last_unstale":"2024-04-24T08:21:11.217135+0000","last_undegraded":"2024-04-24T08:21:11.217135+0000","last_fullsized":"2024-04-24T08:21:11.217135+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:03:34.286436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089289+0000","last_change":"2024-04-24T08:18:46.432499+0000","last_active":"2024-04-24T08:21:11.089289+0000","last_peered":"2024-04-24T08:21:11.089289+0000","last_clean":"2024-04-24T08:21:11.089289+0000","last_became_active":"2024-04-24T08:18:46.432085+0000","last_became_peered":"2024-04-24T08:18:46.432085+0000","last_unstale":"2024-04-24T08:21:11.089289+0000","last_undegraded":"2024-04-24T08:21:11.089289+0000","last_fullsized":"2024-04-24T08:21:11.089289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:03.794091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217293+0000","last_change":"2024-04-24T08:18:47.291193+0000","last_active":"2024-04-24T08:21:11.217293+0000","last_peered":"2024-04-24T08:21:11.217293+0000","last_clean":"2024-04-24T08:21:11.217293+0000","last_became_active":"2024-04-24T08:18:47.289903+0000","last_became_peered":"2024-04-24T08:18:47.289903+0000","last_unstale":"2024-04-24T08:21:11.217293+0000","last_undegraded":"2024-04-24T08:21:11.217293+0000","last_fullsized":"2024-04-24T08:21:11.217293+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:35:18.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210090+0000","last_change":"2024-04-24T08:18:47.292628+0000","last_active":"2024-04-24T08:21:11.210090+0000","last_peered":"2024-04-24T08:21:11.210090+0000","last_clean":"2024-04-24T08:21:11.210090+0000","last_became_active":"2024-04-24T08:18:47.292060+0000","last_became_peered":"2024-04-24T08:18:47.292060+0000","last_unstale":"2024-04-24T08:21:11.210090+0000","last_undegraded":"2024-04-24T08:21:11.210090+0000","last_fullsized":"2024-04-24T08:21:11.210090+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:53.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407920+0000","last_change":"2024-04-24T08:18:47.297121+0000","last_active":"2024-04-24T08:21:11.407920+0000","last_peered":"2024-04-24T08:21:11.407920+0000","last_clean":"2024-04-24T08:21:11.407920+0000","last_became_active":"2024-04-24T08:18:47.296787+0000","last_became_peered":"2024-04-24T08:18:47.296787+0000","last_unstale":"2024-04-24T08:21:11.407920+0000","last_undegraded":"2024-04-24T08:21:11.407920+0000","last_fullsized":"2024-04-24T08:21:11.407920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:44:10.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407884+0000","last_change":"2024-04-24T08:18:47.320669+0000","last_active":"2024-04-24T08:21:11.407884+0000","last_peered":"2024-04-24T08:21:11.407884+0000","last_clean":"2024-04-24T08:21:11.407884+0000","last_became_active":"2024-04-24T08:18:47.320287+0000","last_became_peered":"2024-04-24T08:18:47.320287+0000","last_unstale":"2024-04-24T08:21:11.407884+0000","last_undegraded":"2024-04-24T08:21:11.407884+0000","last_fullsized":"2024-04-24T08:21:11.407884+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:39:03.275189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.417067+0000","last_change":"2024-04-24T08:18:57.585568+0000","last_active":"2024-04-24T08:21:11.417067+0000","last_peered":"2024-04-24T08:21:11.417067+0000","last_clean":"2024-04-24T08:21:11.417067+0000","last_became_active":"2024-04-24T08:18:57.584799+0000","last_became_peered":"2024-04-24T08:18:57.584799+0000","last_unstale":"2024-04-24T08:21:11.417067+0000","last_undegraded":"2024-04-24T08:21:11.417067+0000","last_fullsized":"2024-04-24T08:21:11.417067+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:47.202344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089676+0000","last_change":"2024-04-24T08:18:47.316068+0000","last_active":"2024-04-24T08:21:11.089676+0000","last_peered":"2024-04-24T08:21:11.089676+0000","last_clean":"2024-04-24T08:21:11.089676+0000","last_became_active":"2024-04-24T08:18:47.315647+0000","last_became_peered":"2024-04-24T08:18:47.315647+0000","last_unstale":"2024-04-24T08:21:11.089676+0000","last_undegraded":"2024-04-24T08:21:11.089676+0000","last_fullsized":"2024-04-24T08:21:11.089676+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:33.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210316+0000","last_change":"2024-04-24T08:18:47.294966+0000","last_active":"2024-04-24T08:21:11.210316+0000","last_peered":"2024-04-24T08:21:11.210316+0000","last_clean":"2024-04-24T08:21:11.210316+0000","last_became_active":"2024-04-24T08:18:47.294672+0000","last_became_peered":"2024-04-24T08:18:47.294672+0000","last_unstale":"2024-04-24T08:21:11.210316+0000","last_undegraded":"2024-04-24T08:21:11.210316+0000","last_fullsized":"2024-04-24T08:21:11.210316+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:37.359114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407962+0000","last_change":"2024-04-24T08:18:47.294548+0000","last_active":"2024-04-24T08:21:11.407962+0000","last_peered":"2024-04-24T08:21:11.407962+0000","last_clean":"2024-04-24T08:21:11.407962+0000","last_became_active":"2024-04-24T08:18:47.293589+0000","last_became_peered":"2024-04-24T08:18:47.293589+0000","last_unstale":"2024-04-24T08:21:11.407962+0000","last_undegraded":"2024-04-24T08:21:11.407962+0000","last_fullsized":"2024-04-24T08:21:11.407962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:34:56.208453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211773+0000","last_change":"2024-04-24T08:18:47.313156+0000","last_active":"2024-04-24T08:21:11.211773+0000","last_peered":"2024-04-24T08:21:11.211773+0000","last_clean":"2024-04-24T08:21:11.211773+0000","last_became_active":"2024-04-24T08:18:47.312829+0000","last_became_peered":"2024-04-24T08:18:47.312829+0000","last_unstale":"2024-04-24T08:21:11.211773+0000","last_undegraded":"2024-04-24T08:21:11.211773+0000","last_fullsized":"2024-04-24T08:21:11.211773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:28:36.638514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219278+0000","last_change":"2024-04-24T08:18:47.320022+0000","last_active":"2024-04-24T08:21:11.219278+0000","last_peered":"2024-04-24T08:21:11.219278+0000","last_clean":"2024-04-24T08:21:11.219278+0000","last_became_active":"2024-04-24T08:18:47.319707+0000","last_became_peered":"2024-04-24T08:18:47.319707+0000","last_unstale":"2024-04-24T08:21:11.219278+0000","last_undegraded":"2024-04-24T08:21:11.219278+0000","last_fullsized":"2024-04-24T08:21:11.219278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:51.330874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210674+0000","last_change":"2024-04-24T08:18:47.323187+0000","last_active":"2024-04-24T08:21:11.210674+0000","last_peered":"2024-04-24T08:21:11.210674+0000","last_clean":"2024-04-24T08:21:11.210674+0000","last_became_active":"2024-04-24T08:18:47.320258+0000","last_became_peered":"2024-04-24T08:18:47.320258+0000","last_unstale":"2024-04-24T08:21:11.210674+0000","last_undegraded":"2024-04-24T08:21:11.210674+0000","last_fullsized":"2024-04-24T08:21:11.210674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:42:32.770950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409986+0000","last_change":"2024-04-24T08:18:47.321926+0000","last_active":"2024-04-24T08:21:11.409986+0000","last_peered":"2024-04-24T08:21:11.409986+0000","last_clean":"2024-04-24T08:21:11.409986+0000","last_became_active":"2024-04-24T08:18:47.321584+0000","last_became_peered":"2024-04-24T08:18:47.321584+0000","last_unstale":"2024-04-24T08:21:11.409986+0000","last_undegraded":"2024-04-24T08:21:11.409986+0000","last_fullsized":"2024-04-24T08:21:11.409986+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:24.357157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.093027+0000","last_change":"2024-04-24T08:18:47.313310+0000","last_active":"2024-04-24T08:21:11.093027+0000","last_peered":"2024-04-24T08:21:11.093027+0000","last_clean":"2024-04-24T08:21:11.093027+0000","last_became_active":"2024-04-24T08:18:47.312968+0000","last_became_peered":"2024-04-24T08:18:47.312968+0000","last_unstale":"2024-04-24T08:21:11.093027+0000","last_undegraded":"2024-04-24T08:21:11.093027+0000","last_fullsized":"2024-04-24T08:21:11.093027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:08.967577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210366+0000","last_change":"2024-04-24T08:18:57.591543+0000","last_active":"2024-04-24T08:21:11.210366+0000","last_peered":"2024-04-24T08:21:11.210366+0000","last_clean":"2024-04-24T08:21:11.210366+0000","last_became_active":"2024-04-24T08:18:57.591043+0000","last_became_peered":"2024-04-24T08:18:57.591043+0000","last_unstale":"2024-04-24T08:21:11.210366+0000","last_undegraded":"2024-04-24T08:21:11.210366+0000","last_fullsized":"2024-04-24T08:21:11.210366+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:52:38.669469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597142,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":966,"num_read_kb":592,"num_write":291,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1791426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285609,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28380,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716772,"statfs":{"total":95995035648,"available":95965974528,"internally_reserved":0,"allocated":1392640,"data_stored":871799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547246,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1765376,"data_stored":1197725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710773,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36720,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708432,"statfs":{"total":95995035648,"available":95957434368,"internally_reserved":0,"allocated":1413120,"data_stored":872152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972411,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":478739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1609728,"data_stored":1072293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430277,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":478208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1073152,"data_stored":544702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":479479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":532842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1003218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888145,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":478645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37112,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708040,"statfs":{"total":95995035648,"available":95957032960,"internally_reserved":0,"allocated":1757184,"data_stored":1202795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":200661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:21:16.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:21:16.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 -- 172.21.15.146:0/1222998644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105450 msgr2=0x7efe341058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 --2- 172.21.15.146:0/1222998644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105450 0x7efe341058d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efe20009900 tx=0x7efe2002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 -- 172.21.15.146:0/1222998644 shutdown_connections 2024-04-24T08:21:16.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 --2- 172.21.15.146:0/1222998644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34105e10 0x7efe3410c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 --2- 172.21.15.146:0/1222998644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105450 0x7efe341058d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 --2- 172.21.15.146:0/1222998644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe34104260 0x7efe34104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 -- 172.21.15.146:0/1222998644 >> 172.21.15.146:0/1222998644 conn(0x7efe340ff7f0 msgr2=0x7efe34101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.916+0000 7efe39600640 1 -- 172.21.15.146:0/1222998644 shutdown_connections 2024-04-24T08:21:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.917+0000 7efe39600640 1 -- 172.21.15.146:0/1222998644 wait complete. 2024-04-24T08:21:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.918+0000 7efe39600640 1 Processor -- start 2024-04-24T08:21:16.920 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:16 smithi146 ceph-mon[27805]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:16.920 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:16 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1844869718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:16.920 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:16 smithi146 ceph-mon[27805]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-24T08:21:16.920 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:16 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3125769609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.918+0000 7efe39600640 1 -- start start 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.918+0000 7efe39600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34104260 0x7efe3419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.918+0000 7efe39600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe34105450 0x7efe3419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.918+0000 7efe39600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105e10 0x7efe341a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe39600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7efe3410ff40 con 0x7efe34105e10 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe39600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe3410fdc0 con 0x7efe34105450 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe39600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efe341100c0 con 0x7efe34104260 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe33400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105e10 0x7efe341a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34104260 0x7efe3419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34104260 0x7efe3419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44634/0 (socket says 172.21.15.146:44634) 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe33400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105e10 0x7efe341a1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39160/0 (socket says 172.21.15.146:39160) 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 -- 172.21.15.146:0/1277831493 learned_addr learned my addr 172.21.15.146:0/1277831493 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32000640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe34105450 0x7efe3419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 -- 172.21.15.146:0/1277831493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe34105450 msgr2=0x7efe3419aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe34105450 0x7efe3419aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 -- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105e10 msgr2=0x7efe341a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105e10 0x7efe341a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe32a00640 1 -- 172.21.15.146:0/1277831493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe341a2190 con 0x7efe34104260 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.919+0000 7efe33400640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105e10 0x7efe341a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.920+0000 7efe32000640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe34105450 0x7efe3419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.920+0000 7efe32a00640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34104260 0x7efe3419a570 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efe2400e9c0 tx=0x7efe2400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.920+0000 7efe2b400640 1 -- 172.21.15.146:0/1277831493 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2400cde0 con 0x7efe34104260 2024-04-24T08:21:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.920+0000 7efe2b400640 1 -- 172.21.15.146:0/1277831493 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe24010910 con 0x7efe34104260 2024-04-24T08:21:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.920+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe341a2480 con 0x7efe34104260 2024-04-24T08:21:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.920+0000 7efe2b400640 1 -- 172.21.15.146:0/1277831493 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe24018a00 con 0x7efe34104260 2024-04-24T08:21:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.920+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe341a2a20 con 0x7efe34104260 2024-04-24T08:21:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.921+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe3410b580 con 0x7efe34104260 2024-04-24T08:21:16.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.923+0000 7efe2b400640 1 -- 172.21.15.146:0/1277831493 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7efe2401f020 con 0x7efe34104260 2024-04-24T08:21:16.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.925+0000 7efe2b400640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7efe180a3ec0 0x7efe180a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.925+0000 7efe2b400640 1 -- 172.21.15.146:0/1277831493 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7efe240d0e80 con 0x7efe34104260 2024-04-24T08:21:16.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.925+0000 7efe32000640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7efe180a3ec0 0x7efe180a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.926+0000 7efe32000640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7efe180a3ec0 0x7efe180a6380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe2002a910 tx=0x7efe20000bc0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:16.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:16.927+0000 7efe2b400640 1 -- 172.21.15.146:0/1277831493 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe2409d280 con 0x7efe34104260 2024-04-24T08:21:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:16 smithi153 ceph-mon[32746]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:16 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1844869718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:16 smithi153 ceph-mon[32746]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-24T08:21:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:16 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3125769609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:16 smithi177 ceph-mon[33046]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:16 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1844869718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:16 smithi177 ceph-mon[33046]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-24T08:21:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:16 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3125769609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:17.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.156+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe34062440 con 0x7efe180a3ec0 2024-04-24T08:21:17.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.164+0000 7efe2b400640 1 -- 172.21.15.146:0/1277831493 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257773 (secure 0 0 0) 0x7efe34062440 con 0x7efe180a3ec0 2024-04-24T08:21:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:17.170 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7efe180a3ec0 msgr2=0x7efe180a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7efe180a3ec0 0x7efe180a6380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe2002a910 tx=0x7efe20000bc0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34104260 msgr2=0x7efe3419a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34104260 0x7efe3419a570 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efe2400e9c0 tx=0x7efe2400ee80 comp rx=0 tx=0).stop 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 shutdown_connections 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7efe180a3ec0 0x7efe180a6380 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efe34105e10 0x7efe341a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe34105450 0x7efe3419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 --2- 172.21.15.146:0/1277831493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efe34104260 0x7efe3419a570 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 >> 172.21.15.146:0/1277831493 conn(0x7efe340ff7f0 msgr2=0x7efe34108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 shutdown_connections 2024-04-24T08:21:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.168+0000 7efe39600640 1 -- 172.21.15.146:0/1277831493 wait complete. 2024-04-24T08:21:17.190 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-24T08:21:17.111381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1187744,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1102,"num_read_kb":711,"num_write":474,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377040,"kb_used_data":15312,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124564784,"statfs":{"total":1151940427776,"available":1151554338816,"internally_reserved":0,"allocated":15679488,"data_stored":9212597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.979498"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210587+0000","last_change":"2024-04-24T08:18:47.309666+0000","last_active":"2024-04-24T08:21:11.210587+0000","last_peered":"2024-04-24T08:21:11.210587+0000","last_clean":"2024-04-24T08:21:11.210587+0000","last_became_active":"2024-04-24T08:18:47.309413+0000","last_became_peered":"2024-04-24T08:18:47.309413+0000","last_unstale":"2024-04-24T08:21:11.210587+0000","last_undegraded":"2024-04-24T08:21:11.210587+0000","last_fullsized":"2024-04-24T08:21:11.210587+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:29.662505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416384+0000","last_change":"2024-04-24T08:18:47.310321+0000","last_active":"2024-04-24T08:21:11.416384+0000","last_peered":"2024-04-24T08:21:11.416384+0000","last_clean":"2024-04-24T08:21:11.416384+0000","last_became_active":"2024-04-24T08:18:47.309956+0000","last_became_peered":"2024-04-24T08:18:47.309956+0000","last_unstale":"2024-04-24T08:21:11.416384+0000","last_undegraded":"2024-04-24T08:21:11.416384+0000","last_fullsized":"2024-04-24T08:21:11.416384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:51:48.432725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416668+0000","last_change":"2024-04-24T08:18:47.312153+0000","last_active":"2024-04-24T08:21:11.416668+0000","last_peered":"2024-04-24T08:21:11.416668+0000","last_clean":"2024-04-24T08:21:11.416668+0000","last_became_active":"2024-04-24T08:18:47.309465+0000","last_became_peered":"2024-04-24T08:18:47.309465+0000","last_unstale":"2024-04-24T08:21:11.416668+0000","last_undegraded":"2024-04-24T08:21:11.416668+0000","last_fullsized":"2024-04-24T08:21:11.416668+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:55.829143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089098+0000","last_change":"2024-04-24T08:18:47.297017+0000","last_active":"2024-04-24T08:21:11.089098+0000","last_peered":"2024-04-24T08:21:11.089098+0000","last_clean":"2024-04-24T08:21:11.089098+0000","last_became_active":"2024-04-24T08:18:47.296693+0000","last_became_peered":"2024-04-24T08:18:47.296693+0000","last_unstale":"2024-04-24T08:21:11.089098+0000","last_undegraded":"2024-04-24T08:21:11.089098+0000","last_fullsized":"2024-04-24T08:21:11.089098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:23.400403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089279+0000","last_change":"2024-04-24T08:18:47.315376+0000","last_active":"2024-04-24T08:21:11.089279+0000","last_peered":"2024-04-24T08:21:11.089279+0000","last_clean":"2024-04-24T08:21:11.089279+0000","last_became_active":"2024-04-24T08:18:47.315027+0000","last_became_peered":"2024-04-24T08:18:47.315027+0000","last_unstale":"2024-04-24T08:21:11.089279+0000","last_undegraded":"2024-04-24T08:21:11.089279+0000","last_fullsized":"2024-04-24T08:21:11.089279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:12.504515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222417+0000","last_change":"2024-04-24T08:18:47.306388+0000","last_active":"2024-04-24T08:21:11.222417+0000","last_peered":"2024-04-24T08:21:11.222417+0000","last_clean":"2024-04-24T08:21:11.222417+0000","last_became_active":"2024-04-24T08:18:47.306082+0000","last_became_peered":"2024-04-24T08:18:47.306082+0000","last_unstale":"2024-04-24T08:21:11.222417+0000","last_undegraded":"2024-04-24T08:21:11.222417+0000","last_fullsized":"2024-04-24T08:21:11.222417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:34:52.612799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216773+0000","last_change":"2024-04-24T08:18:47.303336+0000","last_active":"2024-04-24T08:21:11.216773+0000","last_peered":"2024-04-24T08:21:11.216773+0000","last_clean":"2024-04-24T08:21:11.216773+0000","last_became_active":"2024-04-24T08:18:47.303008+0000","last_became_peered":"2024-04-24T08:18:47.303008+0000","last_unstale":"2024-04-24T08:21:11.216773+0000","last_undegraded":"2024-04-24T08:21:11.216773+0000","last_fullsized":"2024-04-24T08:21:11.216773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:57.404693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210549+0000","last_change":"2024-04-24T08:18:47.316672+0000","last_active":"2024-04-24T08:21:11.210549+0000","last_peered":"2024-04-24T08:21:11.210549+0000","last_clean":"2024-04-24T08:21:11.210549+0000","last_became_active":"2024-04-24T08:18:47.315741+0000","last_became_peered":"2024-04-24T08:18:47.315741+0000","last_unstale":"2024-04-24T08:21:11.210549+0000","last_undegraded":"2024-04-24T08:21:11.210549+0000","last_fullsized":"2024-04-24T08:21:11.210549+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:29.156710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409994+0000","last_change":"2024-04-24T08:18:47.291733+0000","last_active":"2024-04-24T08:21:11.409994+0000","last_peered":"2024-04-24T08:21:11.409994+0000","last_clean":"2024-04-24T08:21:11.409994+0000","last_became_active":"2024-04-24T08:18:47.291432+0000","last_became_peered":"2024-04-24T08:18:47.291432+0000","last_unstale":"2024-04-24T08:21:11.409994+0000","last_undegraded":"2024-04-24T08:21:11.409994+0000","last_fullsized":"2024-04-24T08:21:11.409994+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:38:03.601540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222808+0000","last_change":"2024-04-24T08:18:47.294544+0000","last_active":"2024-04-24T08:21:11.222808+0000","last_peered":"2024-04-24T08:21:11.222808+0000","last_clean":"2024-04-24T08:21:11.222808+0000","last_became_active":"2024-04-24T08:18:47.294183+0000","last_became_peered":"2024-04-24T08:18:47.294183+0000","last_unstale":"2024-04-24T08:21:11.222808+0000","last_undegraded":"2024-04-24T08:21:11.222808+0000","last_fullsized":"2024-04-24T08:21:11.222808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:53.082644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217198+0000","last_change":"2024-04-24T08:18:47.292615+0000","last_active":"2024-04-24T08:21:11.217198+0000","last_peered":"2024-04-24T08:21:11.217198+0000","last_clean":"2024-04-24T08:21:11.217198+0000","last_became_active":"2024-04-24T08:18:47.292300+0000","last_became_peered":"2024-04-24T08:18:47.292300+0000","last_unstale":"2024-04-24T08:21:11.217198+0000","last_undegraded":"2024-04-24T08:21:11.217198+0000","last_fullsized":"2024-04-24T08:21:11.217198+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:58.293253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089115+0000","last_change":"2024-04-24T08:18:47.298388+0000","last_active":"2024-04-24T08:21:11.089115+0000","last_peered":"2024-04-24T08:21:11.089115+0000","last_clean":"2024-04-24T08:21:11.089115+0000","last_became_active":"2024-04-24T08:18:47.298008+0000","last_became_peered":"2024-04-24T08:18:47.298008+0000","last_unstale":"2024-04-24T08:21:11.089115+0000","last_undegraded":"2024-04-24T08:21:11.089115+0000","last_fullsized":"2024-04-24T08:21:11.089115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:10:00.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.092591+0000","last_change":"2024-04-24T08:18:47.298809+0000","last_active":"2024-04-24T08:21:11.092591+0000","last_peered":"2024-04-24T08:21:11.092591+0000","last_clean":"2024-04-24T08:21:11.092591+0000","last_became_active":"2024-04-24T08:18:47.298103+0000","last_became_peered":"2024-04-24T08:18:47.298103+0000","last_unstale":"2024-04-24T08:21:11.092591+0000","last_undegraded":"2024-04-24T08:21:11.092591+0000","last_fullsized":"2024-04-24T08:21:11.092591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:38:53.625721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415961+0000","last_change":"2024-04-24T08:18:47.314572+0000","last_active":"2024-04-24T08:21:11.415961+0000","last_peered":"2024-04-24T08:21:11.415961+0000","last_clean":"2024-04-24T08:21:11.415961+0000","last_became_active":"2024-04-24T08:18:47.313372+0000","last_became_peered":"2024-04-24T08:18:47.313372+0000","last_unstale":"2024-04-24T08:21:11.415961+0000","last_undegraded":"2024-04-24T08:21:11.415961+0000","last_fullsized":"2024-04-24T08:21:11.415961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:56:01.954909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415978+0000","last_change":"2024-04-24T08:18:47.314822+0000","last_active":"2024-04-24T08:21:11.415978+0000","last_peered":"2024-04-24T08:21:11.415978+0000","last_clean":"2024-04-24T08:21:11.415978+0000","last_became_active":"2024-04-24T08:18:47.310774+0000","last_became_peered":"2024-04-24T08:18:47.310774+0000","last_unstale":"2024-04-24T08:21:11.415978+0000","last_undegraded":"2024-04-24T08:21:11.415978+0000","last_fullsized":"2024-04-24T08:21:11.415978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:24.934100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409946+0000","last_change":"2024-04-24T08:18:47.307208+0000","last_active":"2024-04-24T08:21:11.409946+0000","last_peered":"2024-04-24T08:21:11.409946+0000","last_clean":"2024-04-24T08:21:11.409946+0000","last_became_active":"2024-04-24T08:18:47.306889+0000","last_became_peered":"2024-04-24T08:18:47.306889+0000","last_unstale":"2024-04-24T08:21:11.409946+0000","last_undegraded":"2024-04-24T08:21:11.409946+0000","last_fullsized":"2024-04-24T08:21:11.409946+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:20:01.438388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210570+0000","last_change":"2024-04-24T08:18:47.322929+0000","last_active":"2024-04-24T08:21:11.210570+0000","last_peered":"2024-04-24T08:21:11.210570+0000","last_clean":"2024-04-24T08:21:11.210570+0000","last_became_active":"2024-04-24T08:18:47.322593+0000","last_became_peered":"2024-04-24T08:18:47.322593+0000","last_unstale":"2024-04-24T08:21:11.210570+0000","last_undegraded":"2024-04-24T08:21:11.210570+0000","last_fullsized":"2024-04-24T08:21:11.210570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:09.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.208062+0000","last_change":"2024-04-24T08:18:47.304886+0000","last_active":"2024-04-24T08:21:11.208062+0000","last_peered":"2024-04-24T08:21:11.208062+0000","last_clean":"2024-04-24T08:21:11.208062+0000","last_became_active":"2024-04-24T08:18:47.302631+0000","last_became_peered":"2024-04-24T08:18:47.302631+0000","last_unstale":"2024-04-24T08:21:11.208062+0000","last_undegraded":"2024-04-24T08:21:11.208062+0000","last_fullsized":"2024-04-24T08:21:11.208062+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:13.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407979+0000","last_change":"2024-04-24T08:18:47.316828+0000","last_active":"2024-04-24T08:21:11.407979+0000","last_peered":"2024-04-24T08:21:11.407979+0000","last_clean":"2024-04-24T08:21:11.407979+0000","last_became_active":"2024-04-24T08:18:47.316540+0000","last_became_peered":"2024-04-24T08:18:47.316540+0000","last_unstale":"2024-04-24T08:21:11.407979+0000","last_undegraded":"2024-04-24T08:21:11.407979+0000","last_fullsized":"2024-04-24T08:21:11.407979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:41:42.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207816+0000","last_change":"2024-04-24T08:18:46.444744+0000","last_active":"2024-04-24T08:21:11.207816+0000","last_peered":"2024-04-24T08:21:11.207816+0000","last_clean":"2024-04-24T08:21:11.207816+0000","last_became_active":"2024-04-24T08:18:46.442495+0000","last_became_peered":"2024-04-24T08:18:46.442495+0000","last_unstale":"2024-04-24T08:21:11.207816+0000","last_undegraded":"2024-04-24T08:21:11.207816+0000","last_fullsized":"2024-04-24T08:21:11.207816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:33:32.270984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211523+0000","last_change":"2024-04-24T08:18:47.320462+0000","last_active":"2024-04-24T08:21:11.211523+0000","last_peered":"2024-04-24T08:21:11.211523+0000","last_clean":"2024-04-24T08:21:11.211523+0000","last_became_active":"2024-04-24T08:18:47.319267+0000","last_became_peered":"2024-04-24T08:18:47.319267+0000","last_unstale":"2024-04-24T08:21:11.211523+0000","last_undegraded":"2024-04-24T08:21:11.211523+0000","last_fullsized":"2024-04-24T08:21:11.211523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:50:15.882077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211656+0000","last_change":"2024-04-24T08:18:46.453278+0000","last_active":"2024-04-24T08:21:11.211656+0000","last_peered":"2024-04-24T08:21:11.211656+0000","last_clean":"2024-04-24T08:21:11.211656+0000","last_became_active":"2024-04-24T08:18:46.452913+0000","last_became_peered":"2024-04-24T08:18:46.452913+0000","last_unstale":"2024-04-24T08:21:11.211656+0000","last_undegraded":"2024-04-24T08:21:11.211656+0000","last_fullsized":"2024-04-24T08:21:11.211656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:48.272190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416239+0000","last_change":"2024-04-24T08:18:47.323610+0000","last_active":"2024-04-24T08:21:11.416239+0000","last_peered":"2024-04-24T08:21:11.416239+0000","last_clean":"2024-04-24T08:21:11.416239+0000","last_became_active":"2024-04-24T08:18:47.322795+0000","last_became_peered":"2024-04-24T08:18:47.322795+0000","last_unstale":"2024-04-24T08:21:11.416239+0000","last_undegraded":"2024-04-24T08:21:11.416239+0000","last_fullsized":"2024-04-24T08:21:11.416239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:32.951564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089694+0000","last_change":"2024-04-24T08:18:57.730353+0000","last_active":"2024-04-24T08:21:11.089694+0000","last_peered":"2024-04-24T08:21:11.089694+0000","last_clean":"2024-04-24T08:21:11.089694+0000","last_became_active":"2024-04-24T08:18:57.725936+0000","last_became_peered":"2024-04-24T08:18:57.725936+0000","last_unstale":"2024-04-24T08:21:11.089694+0000","last_undegraded":"2024-04-24T08:21:11.089694+0000","last_fullsized":"2024-04-24T08:21:11.089694+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:00:27.764723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410161+0000","last_change":"2024-04-24T08:18:47.304884+0000","last_active":"2024-04-24T08:21:11.410161+0000","last_peered":"2024-04-24T08:21:11.410161+0000","last_clean":"2024-04-24T08:21:11.410161+0000","last_became_active":"2024-04-24T08:18:47.304520+0000","last_became_peered":"2024-04-24T08:18:47.304520+0000","last_unstale":"2024-04-24T08:21:11.410161+0000","last_undegraded":"2024-04-24T08:21:11.410161+0000","last_fullsized":"2024-04-24T08:21:11.410161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:42.325010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089422+0000","last_change":"2024-04-24T08:18:46.441073+0000","last_active":"2024-04-24T08:21:11.089422+0000","last_peered":"2024-04-24T08:21:11.089422+0000","last_clean":"2024-04-24T08:21:11.089422+0000","last_became_active":"2024-04-24T08:18:46.440721+0000","last_became_peered":"2024-04-24T08:18:46.440721+0000","last_unstale":"2024-04-24T08:21:11.089422+0000","last_undegraded":"2024-04-24T08:21:11.089422+0000","last_fullsized":"2024-04-24T08:21:11.089422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:51:39.516458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219409+0000","last_change":"2024-04-24T08:18:47.293947+0000","last_active":"2024-04-24T08:21:11.219409+0000","last_peered":"2024-04-24T08:21:11.219409+0000","last_clean":"2024-04-24T08:21:11.219409+0000","last_became_active":"2024-04-24T08:18:47.293623+0000","last_became_peered":"2024-04-24T08:18:47.293623+0000","last_unstale":"2024-04-24T08:21:11.219409+0000","last_undegraded":"2024-04-24T08:21:11.219409+0000","last_fullsized":"2024-04-24T08:21:11.219409+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:23.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.741488+0000","last_change":"2024-04-24T08:18:46.439909+0000","last_active":"2024-04-24T08:21:11.741488+0000","last_peered":"2024-04-24T08:21:11.741488+0000","last_clean":"2024-04-24T08:21:11.741488+0000","last_became_active":"2024-04-24T08:18:46.439597+0000","last_became_peered":"2024-04-24T08:18:46.439597+0000","last_unstale":"2024-04-24T08:21:11.741488+0000","last_undegraded":"2024-04-24T08:21:11.741488+0000","last_fullsized":"2024-04-24T08:21:11.741488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:49:11.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089065+0000","last_change":"2024-04-24T08:18:47.306480+0000","last_active":"2024-04-24T08:21:11.089065+0000","last_peered":"2024-04-24T08:21:11.089065+0000","last_clean":"2024-04-24T08:21:11.089065+0000","last_became_active":"2024-04-24T08:18:47.306104+0000","last_became_peered":"2024-04-24T08:18:47.306104+0000","last_unstale":"2024-04-24T08:21:11.089065+0000","last_undegraded":"2024-04-24T08:21:11.089065+0000","last_fullsized":"2024-04-24T08:21:11.089065+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:56.340836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222262+0000","last_change":"2024-04-24T08:18:46.439381+0000","last_active":"2024-04-24T08:21:11.222262+0000","last_peered":"2024-04-24T08:21:11.222262+0000","last_clean":"2024-04-24T08:21:11.222262+0000","last_became_active":"2024-04-24T08:18:46.439033+0000","last_became_peered":"2024-04-24T08:18:46.439033+0000","last_unstale":"2024-04-24T08:21:11.222262+0000","last_undegraded":"2024-04-24T08:21:11.222262+0000","last_fullsized":"2024-04-24T08:21:11.222262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:48:28.833066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219251+0000","last_change":"2024-04-24T08:18:57.595549+0000","last_active":"2024-04-24T08:21:11.219251+0000","last_peered":"2024-04-24T08:21:11.219251+0000","last_clean":"2024-04-24T08:21:11.219251+0000","last_became_active":"2024-04-24T08:18:57.595098+0000","last_became_peered":"2024-04-24T08:18:57.595098+0000","last_unstale":"2024-04-24T08:21:11.219251+0000","last_undegraded":"2024-04-24T08:21:11.219251+0000","last_fullsized":"2024-04-24T08:21:11.219251+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:09:22.722996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.094360+0000","last_change":"2024-04-24T08:18:46.448151+0000","last_active":"2024-04-24T08:21:11.094360+0000","last_peered":"2024-04-24T08:21:11.094360+0000","last_clean":"2024-04-24T08:21:11.094360+0000","last_became_active":"2024-04-24T08:18:46.447077+0000","last_became_peered":"2024-04-24T08:18:46.447077+0000","last_unstale":"2024-04-24T08:21:11.094360+0000","last_undegraded":"2024-04-24T08:21:11.094360+0000","last_fullsized":"2024-04-24T08:21:11.094360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:00.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210379+0000","last_change":"2024-04-24T08:18:47.319423+0000","last_active":"2024-04-24T08:21:11.210379+0000","last_peered":"2024-04-24T08:21:11.210379+0000","last_clean":"2024-04-24T08:21:11.210379+0000","last_became_active":"2024-04-24T08:18:47.316284+0000","last_became_peered":"2024-04-24T08:18:47.316284+0000","last_unstale":"2024-04-24T08:21:11.210379+0000","last_undegraded":"2024-04-24T08:21:11.210379+0000","last_fullsized":"2024-04-24T08:21:11.210379+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:55.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219135+0000","last_change":"2024-04-24T08:18:46.436152+0000","last_active":"2024-04-24T08:21:11.219135+0000","last_peered":"2024-04-24T08:21:11.219135+0000","last_clean":"2024-04-24T08:21:11.219135+0000","last_became_active":"2024-04-24T08:18:46.435835+0000","last_became_peered":"2024-04-24T08:18:46.435835+0000","last_unstale":"2024-04-24T08:21:11.219135+0000","last_undegraded":"2024-04-24T08:21:11.219135+0000","last_fullsized":"2024-04-24T08:21:11.219135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:12:30.536455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207423+0000","last_change":"2024-04-24T08:18:57.725728+0000","last_active":"2024-04-24T08:21:11.207423+0000","last_peered":"2024-04-24T08:21:11.207423+0000","last_clean":"2024-04-24T08:21:11.207423+0000","last_became_active":"2024-04-24T08:18:57.724704+0000","last_became_peered":"2024-04-24T08:18:57.724704+0000","last_unstale":"2024-04-24T08:21:11.207423+0000","last_undegraded":"2024-04-24T08:21:11.207423+0000","last_fullsized":"2024-04-24T08:21:11.207423+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:27.501006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":167,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.740986+0000","last_change":"2024-04-24T08:18:46.446652+0000","last_active":"2024-04-24T08:21:11.740986+0000","last_peered":"2024-04-24T08:21:11.740986+0000","last_clean":"2024-04-24T08:21:11.740986+0000","last_became_active":"2024-04-24T08:18:46.446348+0000","last_became_peered":"2024-04-24T08:18:46.446348+0000","last_unstale":"2024-04-24T08:21:11.740986+0000","last_undegraded":"2024-04-24T08:21:11.740986+0000","last_fullsized":"2024-04-24T08:21:11.740986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:25.791359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410343+0000","last_change":"2024-04-24T08:18:47.309072+0000","last_active":"2024-04-24T08:21:11.410343+0000","last_peered":"2024-04-24T08:21:11.410343+0000","last_clean":"2024-04-24T08:21:11.410343+0000","last_became_active":"2024-04-24T08:18:47.308729+0000","last_became_peered":"2024-04-24T08:18:47.308729+0000","last_unstale":"2024-04-24T08:21:11.410343+0000","last_undegraded":"2024-04-24T08:21:11.410343+0000","last_fullsized":"2024-04-24T08:21:11.410343+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:58.256876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217308+0000","last_change":"2024-04-24T08:18:46.439190+0000","last_active":"2024-04-24T08:21:11.217308+0000","last_peered":"2024-04-24T08:21:11.217308+0000","last_clean":"2024-04-24T08:21:11.217308+0000","last_became_active":"2024-04-24T08:18:46.438442+0000","last_became_peered":"2024-04-24T08:18:46.438442+0000","last_unstale":"2024-04-24T08:21:11.217308+0000","last_undegraded":"2024-04-24T08:21:11.217308+0000","last_fullsized":"2024-04-24T08:21:11.217308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:42.614563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416325+0000","last_change":"2024-04-24T08:18:47.311447+0000","last_active":"2024-04-24T08:21:11.416325+0000","last_peered":"2024-04-24T08:21:11.416325+0000","last_clean":"2024-04-24T08:21:11.416325+0000","last_became_active":"2024-04-24T08:18:47.310939+0000","last_became_peered":"2024-04-24T08:18:47.310939+0000","last_unstale":"2024-04-24T08:21:11.416325+0000","last_undegraded":"2024-04-24T08:21:11.416325+0000","last_fullsized":"2024-04-24T08:21:11.416325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:42.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219601+0000","last_change":"2024-04-24T08:18:46.450596+0000","last_active":"2024-04-24T08:21:11.219601+0000","last_peered":"2024-04-24T08:21:11.219601+0000","last_clean":"2024-04-24T08:21:11.219601+0000","last_became_active":"2024-04-24T08:18:46.450226+0000","last_became_peered":"2024-04-24T08:18:46.450226+0000","last_unstale":"2024-04-24T08:21:11.219601+0000","last_undegraded":"2024-04-24T08:21:11.219601+0000","last_fullsized":"2024-04-24T08:21:11.219601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:05.722507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409988+0000","last_change":"2024-04-24T08:18:47.295266+0000","last_active":"2024-04-24T08:21:11.409988+0000","last_peered":"2024-04-24T08:21:11.409988+0000","last_clean":"2024-04-24T08:21:11.409988+0000","last_became_active":"2024-04-24T08:18:47.294963+0000","last_became_peered":"2024-04-24T08:18:47.294963+0000","last_unstale":"2024-04-24T08:21:11.409988+0000","last_undegraded":"2024-04-24T08:21:11.409988+0000","last_fullsized":"2024-04-24T08:21:11.409988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:07.638921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":348,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.746361+0000","last_change":"2024-04-24T08:18:46.435524+0000","last_active":"2024-04-24T08:21:11.746361+0000","last_peered":"2024-04-24T08:21:11.746361+0000","last_clean":"2024-04-24T08:21:11.746361+0000","last_became_active":"2024-04-24T08:18:46.435212+0000","last_became_peered":"2024-04-24T08:18:46.435212+0000","last_unstale":"2024-04-24T08:21:11.746361+0000","last_undegraded":"2024-04-24T08:21:11.746361+0000","last_fullsized":"2024-04-24T08:21:11.746361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:45.575905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66701,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":291,"num_read_kb":210,"num_write":36,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089464+0000","last_change":"2024-04-24T08:18:47.293648+0000","last_active":"2024-04-24T08:21:11.089464+0000","last_peered":"2024-04-24T08:21:11.089464+0000","last_clean":"2024-04-24T08:21:11.089464+0000","last_became_active":"2024-04-24T08:18:47.293260+0000","last_became_peered":"2024-04-24T08:18:47.293260+0000","last_unstale":"2024-04-24T08:21:11.089464+0000","last_undegraded":"2024-04-24T08:21:11.089464+0000","last_fullsized":"2024-04-24T08:21:11.089464+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:36.462606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408233+0000","last_change":"2024-04-24T08:18:46.441401+0000","last_active":"2024-04-24T08:21:11.408233+0000","last_peered":"2024-04-24T08:21:11.408233+0000","last_clean":"2024-04-24T08:21:11.408233+0000","last_became_active":"2024-04-24T08:18:46.440985+0000","last_became_peered":"2024-04-24T08:18:46.440985+0000","last_unstale":"2024-04-24T08:21:11.408233+0000","last_undegraded":"2024-04-24T08:21:11.408233+0000","last_fullsized":"2024-04-24T08:21:11.408233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:45.201518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089310+0000","last_change":"2024-04-24T08:18:47.320928+0000","last_active":"2024-04-24T08:21:11.089310+0000","last_peered":"2024-04-24T08:21:11.089310+0000","last_clean":"2024-04-24T08:21:11.089310+0000","last_became_active":"2024-04-24T08:18:47.320515+0000","last_became_peered":"2024-04-24T08:18:47.320515+0000","last_unstale":"2024-04-24T08:21:11.089310+0000","last_undegraded":"2024-04-24T08:21:11.089310+0000","last_fullsized":"2024-04-24T08:21:11.089310+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:55:42.899438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219120+0000","last_change":"2024-04-24T08:18:46.437677+0000","last_active":"2024-04-24T08:21:11.219120+0000","last_peered":"2024-04-24T08:21:11.219120+0000","last_clean":"2024-04-24T08:21:11.219120+0000","last_became_active":"2024-04-24T08:18:46.437364+0000","last_became_peered":"2024-04-24T08:18:46.437364+0000","last_unstale":"2024-04-24T08:21:11.219120+0000","last_undegraded":"2024-04-24T08:21:11.219120+0000","last_fullsized":"2024-04-24T08:21:11.219120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:25:55.408887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217154+0000","last_change":"2024-04-24T08:18:47.298396+0000","last_active":"2024-04-24T08:21:11.217154+0000","last_peered":"2024-04-24T08:21:11.217154+0000","last_clean":"2024-04-24T08:21:11.217154+0000","last_became_active":"2024-04-24T08:18:47.298021+0000","last_became_peered":"2024-04-24T08:18:47.298021+0000","last_unstale":"2024-04-24T08:21:11.217154+0000","last_undegraded":"2024-04-24T08:21:11.217154+0000","last_fullsized":"2024-04-24T08:21:11.217154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:24.015897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":170,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.830237+0000","last_change":"2024-04-24T08:18:46.436612+0000","last_active":"2024-04-24T08:21:11.830237+0000","last_peered":"2024-04-24T08:21:11.830237+0000","last_clean":"2024-04-24T08:21:11.830237+0000","last_became_active":"2024-04-24T08:18:46.436211+0000","last_became_peered":"2024-04-24T08:18:46.436211+0000","last_unstale":"2024-04-24T08:21:11.830237+0000","last_undegraded":"2024-04-24T08:21:11.830237+0000","last_fullsized":"2024-04-24T08:21:11.830237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:49:05.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217055+0000","last_change":"2024-04-24T08:18:47.290849+0000","last_active":"2024-04-24T08:21:11.217055+0000","last_peered":"2024-04-24T08:21:11.217055+0000","last_clean":"2024-04-24T08:21:11.217055+0000","last_became_active":"2024-04-24T08:18:47.290514+0000","last_became_peered":"2024-04-24T08:18:47.290514+0000","last_unstale":"2024-04-24T08:21:11.217055+0000","last_undegraded":"2024-04-24T08:21:11.217055+0000","last_fullsized":"2024-04-24T08:21:11.217055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:39:12.527477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089092+0000","last_change":"2024-04-24T08:18:46.451815+0000","last_active":"2024-04-24T08:21:11.089092+0000","last_peered":"2024-04-24T08:21:11.089092+0000","last_clean":"2024-04-24T08:21:11.089092+0000","last_became_active":"2024-04-24T08:18:46.451478+0000","last_became_peered":"2024-04-24T08:18:46.451478+0000","last_unstale":"2024-04-24T08:21:11.089092+0000","last_undegraded":"2024-04-24T08:21:11.089092+0000","last_fullsized":"2024-04-24T08:21:11.089092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:54:22.907143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.218992+0000","last_change":"2024-04-24T08:18:47.295181+0000","last_active":"2024-04-24T08:21:11.218992+0000","last_peered":"2024-04-24T08:21:11.218992+0000","last_clean":"2024-04-24T08:21:11.218992+0000","last_became_active":"2024-04-24T08:18:47.294842+0000","last_became_peered":"2024-04-24T08:18:47.294842+0000","last_unstale":"2024-04-24T08:21:11.218992+0000","last_undegraded":"2024-04-24T08:21:11.218992+0000","last_fullsized":"2024-04-24T08:21:11.218992+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:37.058025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.537280+0000","last_change":"2024-04-24T08:18:46.437967+0000","last_active":"2024-04-24T08:21:11.537280+0000","last_peered":"2024-04-24T08:21:11.537280+0000","last_clean":"2024-04-24T08:21:11.537280+0000","last_became_active":"2024-04-24T08:18:46.437617+0000","last_became_peered":"2024-04-24T08:18:46.437617+0000","last_unstale":"2024-04-24T08:21:11.537280+0000","last_undegraded":"2024-04-24T08:21:11.537280+0000","last_fullsized":"2024-04-24T08:21:11.537280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:07.303827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210446+0000","last_change":"2024-04-24T08:18:57.726323+0000","last_active":"2024-04-24T08:21:11.210446+0000","last_peered":"2024-04-24T08:21:11.210446+0000","last_clean":"2024-04-24T08:21:11.210446+0000","last_became_active":"2024-04-24T08:18:57.725924+0000","last_became_peered":"2024-04-24T08:18:57.725924+0000","last_unstale":"2024-04-24T08:21:11.210446+0000","last_undegraded":"2024-04-24T08:21:11.210446+0000","last_fullsized":"2024-04-24T08:21:11.210446+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:15:33.756804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207766+0000","last_change":"2024-04-24T08:18:57.591169+0000","last_active":"2024-04-24T08:21:11.207766+0000","last_peered":"2024-04-24T08:21:11.207766+0000","last_clean":"2024-04-24T08:21:11.207766+0000","last_became_active":"2024-04-24T08:18:57.590300+0000","last_became_peered":"2024-04-24T08:18:57.590300+0000","last_unstale":"2024-04-24T08:21:11.207766+0000","last_undegraded":"2024-04-24T08:21:11.207766+0000","last_fullsized":"2024-04-24T08:21:11.207766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:13.632780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222625+0000","last_change":"2024-04-24T08:18:47.316791+0000","last_active":"2024-04-24T08:21:11.222625+0000","last_peered":"2024-04-24T08:21:11.222625+0000","last_clean":"2024-04-24T08:21:11.222625+0000","last_became_active":"2024-04-24T08:18:47.316486+0000","last_became_peered":"2024-04-24T08:18:47.316486+0000","last_unstale":"2024-04-24T08:21:11.222625+0000","last_undegraded":"2024-04-24T08:21:11.222625+0000","last_fullsized":"2024-04-24T08:21:11.222625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:36:37.190732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217170+0000","last_change":"2024-04-24T08:18:46.432831+0000","last_active":"2024-04-24T08:21:11.217170+0000","last_peered":"2024-04-24T08:21:11.217170+0000","last_clean":"2024-04-24T08:21:11.217170+0000","last_became_active":"2024-04-24T08:18:46.432483+0000","last_became_peered":"2024-04-24T08:18:46.432483+0000","last_unstale":"2024-04-24T08:21:11.217170+0000","last_undegraded":"2024-04-24T08:21:11.217170+0000","last_fullsized":"2024-04-24T08:21:11.217170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:43:24.802099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222393+0000","last_change":"2024-04-24T08:18:47.309408+0000","last_active":"2024-04-24T08:21:11.222393+0000","last_peered":"2024-04-24T08:21:11.222393+0000","last_clean":"2024-04-24T08:21:11.222393+0000","last_became_active":"2024-04-24T08:18:47.309083+0000","last_became_peered":"2024-04-24T08:18:47.309083+0000","last_unstale":"2024-04-24T08:21:11.222393+0000","last_undegraded":"2024-04-24T08:21:11.222393+0000","last_fullsized":"2024-04-24T08:21:11.222393+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:15:44.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219335+0000","last_change":"2024-04-24T08:18:47.291060+0000","last_active":"2024-04-24T08:21:11.219335+0000","last_peered":"2024-04-24T08:21:11.219335+0000","last_clean":"2024-04-24T08:21:11.219335+0000","last_became_active":"2024-04-24T08:18:47.290635+0000","last_became_peered":"2024-04-24T08:18:47.290635+0000","last_unstale":"2024-04-24T08:21:11.219335+0000","last_undegraded":"2024-04-24T08:21:11.219335+0000","last_fullsized":"2024-04-24T08:21:11.219335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:20.372853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":167,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415606+0000","last_change":"2024-04-24T08:18:46.449850+0000","last_active":"2024-04-24T08:21:11.415606+0000","last_peered":"2024-04-24T08:21:11.415606+0000","last_clean":"2024-04-24T08:21:11.415606+0000","last_became_active":"2024-04-24T08:18:46.448486+0000","last_became_peered":"2024-04-24T08:18:46.448486+0000","last_unstale":"2024-04-24T08:21:11.415606+0000","last_undegraded":"2024-04-24T08:21:11.415606+0000","last_fullsized":"2024-04-24T08:21:11.415606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:54.611867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211035+0000","last_change":"2024-04-24T08:18:47.328051+0000","last_active":"2024-04-24T08:21:11.211035+0000","last_peered":"2024-04-24T08:21:11.211035+0000","last_clean":"2024-04-24T08:21:11.211035+0000","last_became_active":"2024-04-24T08:18:47.327474+0000","last_became_peered":"2024-04-24T08:18:47.327474+0000","last_unstale":"2024-04-24T08:21:11.211035+0000","last_undegraded":"2024-04-24T08:21:11.211035+0000","last_fullsized":"2024-04-24T08:21:11.211035+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:07.577741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.088963+0000","last_change":"2024-04-24T08:18:46.444375+0000","last_active":"2024-04-24T08:21:11.088963+0000","last_peered":"2024-04-24T08:21:11.088963+0000","last_clean":"2024-04-24T08:21:11.088963+0000","last_became_active":"2024-04-24T08:18:46.443963+0000","last_became_peered":"2024-04-24T08:18:46.443963+0000","last_unstale":"2024-04-24T08:21:11.088963+0000","last_undegraded":"2024-04-24T08:21:11.088963+0000","last_fullsized":"2024-04-24T08:21:11.088963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:22.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216787+0000","last_change":"2024-04-24T08:17:29.632219+0000","last_active":"2024-04-24T08:21:11.216787+0000","last_peered":"2024-04-24T08:21:11.216787+0000","last_clean":"2024-04-24T08:21:11.216787+0000","last_became_active":"2024-04-24T08:17:29.631841+0000","last_became_peered":"2024-04-24T08:17:29.631841+0000","last_unstale":"2024-04-24T08:21:11.216787+0000","last_undegraded":"2024-04-24T08:21:11.216787+0000","last_fullsized":"2024-04-24T08:21:11.216787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_clean_scrub_stamp":"2024-04-24T08:13:41.228801+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:02:55.415959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089104+0000","last_change":"2024-04-24T08:18:47.301876+0000","last_active":"2024-04-24T08:21:11.089104+0000","last_peered":"2024-04-24T08:21:11.089104+0000","last_clean":"2024-04-24T08:21:11.089104+0000","last_became_active":"2024-04-24T08:18:47.301545+0000","last_became_peered":"2024-04-24T08:18:47.301545+0000","last_unstale":"2024-04-24T08:21:11.089104+0000","last_undegraded":"2024-04-24T08:21:11.089104+0000","last_fullsized":"2024-04-24T08:21:11.089104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:34.650014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216699+0000","last_change":"2024-04-24T08:18:46.437915+0000","last_active":"2024-04-24T08:21:11.216699+0000","last_peered":"2024-04-24T08:21:11.216699+0000","last_clean":"2024-04-24T08:21:11.216699+0000","last_became_active":"2024-04-24T08:18:46.437567+0000","last_became_peered":"2024-04-24T08:18:46.437567+0000","last_unstale":"2024-04-24T08:21:11.216699+0000","last_undegraded":"2024-04-24T08:21:11.216699+0000","last_fullsized":"2024-04-24T08:21:11.216699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:36.801535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089647+0000","last_change":"2024-04-24T08:18:46.451834+0000","last_active":"2024-04-24T08:21:11.089647+0000","last_peered":"2024-04-24T08:21:11.089647+0000","last_clean":"2024-04-24T08:21:11.089647+0000","last_became_active":"2024-04-24T08:18:46.451505+0000","last_became_peered":"2024-04-24T08:18:46.451505+0000","last_unstale":"2024-04-24T08:21:11.089647+0000","last_undegraded":"2024-04-24T08:21:11.089647+0000","last_fullsized":"2024-04-24T08:21:11.089647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:26.468176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089798+0000","last_change":"2024-04-24T08:18:47.310490+0000","last_active":"2024-04-24T08:21:11.089798+0000","last_peered":"2024-04-24T08:21:11.089798+0000","last_clean":"2024-04-24T08:21:11.089798+0000","last_became_active":"2024-04-24T08:18:47.309615+0000","last_became_peered":"2024-04-24T08:18:47.309615+0000","last_unstale":"2024-04-24T08:21:11.089798+0000","last_undegraded":"2024-04-24T08:21:11.089798+0000","last_fullsized":"2024-04-24T08:21:11.089798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:22:55.063485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'46","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.354674+0000","last_change":"2024-04-24T08:18:57.602098+0000","last_active":"2024-04-24T08:21:11.354674+0000","last_peered":"2024-04-24T08:21:11.354674+0000","last_clean":"2024-04-24T08:21:11.354674+0000","last_became_active":"2024-04-24T08:18:57.595899+0000","last_became_peered":"2024-04-24T08:18:57.595899+0000","last_unstale":"2024-04-24T08:21:11.354674+0000","last_undegraded":"2024-04-24T08:21:11.354674+0000","last_fullsized":"2024-04-24T08:21:11.354674+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:31.951803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211883+0000","last_change":"2024-04-24T08:18:47.319022+0000","last_active":"2024-04-24T08:21:11.211883+0000","last_peered":"2024-04-24T08:21:11.211883+0000","last_clean":"2024-04-24T08:21:11.211883+0000","last_became_active":"2024-04-24T08:18:47.318167+0000","last_became_peered":"2024-04-24T08:18:47.318167+0000","last_unstale":"2024-04-24T08:21:11.211883+0000","last_undegraded":"2024-04-24T08:21:11.211883+0000","last_fullsized":"2024-04-24T08:21:11.211883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:18.980856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089397+0000","last_change":"2024-04-24T08:18:46.433018+0000","last_active":"2024-04-24T08:21:11.089397+0000","last_peered":"2024-04-24T08:21:11.089397+0000","last_clean":"2024-04-24T08:21:11.089397+0000","last_became_active":"2024-04-24T08:18:46.432681+0000","last_became_peered":"2024-04-24T08:18:46.432681+0000","last_unstale":"2024-04-24T08:21:11.089397+0000","last_undegraded":"2024-04-24T08:21:11.089397+0000","last_fullsized":"2024-04-24T08:21:11.089397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:53.691684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416092+0000","last_change":"2024-04-24T08:18:47.326055+0000","last_active":"2024-04-24T08:21:11.416092+0000","last_peered":"2024-04-24T08:21:11.416092+0000","last_clean":"2024-04-24T08:21:11.416092+0000","last_became_active":"2024-04-24T08:18:47.324956+0000","last_became_peered":"2024-04-24T08:18:47.324956+0000","last_unstale":"2024-04-24T08:21:11.416092+0000","last_undegraded":"2024-04-24T08:21:11.416092+0000","last_fullsized":"2024-04-24T08:21:11.416092+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:44:37.431264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410298+0000","last_change":"2024-04-24T08:18:46.450710+0000","last_active":"2024-04-24T08:21:11.410298+0000","last_peered":"2024-04-24T08:21:11.410298+0000","last_clean":"2024-04-24T08:21:11.410298+0000","last_became_active":"2024-04-24T08:18:46.450391+0000","last_became_peered":"2024-04-24T08:18:46.450391+0000","last_unstale":"2024-04-24T08:21:11.410298+0000","last_undegraded":"2024-04-24T08:21:11.410298+0000","last_fullsized":"2024-04-24T08:21:11.410298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:19.905524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089025+0000","last_change":"2024-04-24T08:18:47.292114+0000","last_active":"2024-04-24T08:21:11.089025+0000","last_peered":"2024-04-24T08:21:11.089025+0000","last_clean":"2024-04-24T08:21:11.089025+0000","last_became_active":"2024-04-24T08:18:47.291787+0000","last_became_peered":"2024-04-24T08:18:47.291787+0000","last_unstale":"2024-04-24T08:21:11.089025+0000","last_undegraded":"2024-04-24T08:21:11.089025+0000","last_fullsized":"2024-04-24T08:21:11.089025+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:37.020368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210671+0000","last_change":"2024-04-24T08:18:46.446027+0000","last_active":"2024-04-24T08:21:11.210671+0000","last_peered":"2024-04-24T08:21:11.210671+0000","last_clean":"2024-04-24T08:21:11.210671+0000","last_became_active":"2024-04-24T08:18:46.445634+0000","last_became_peered":"2024-04-24T08:18:46.445634+0000","last_unstale":"2024-04-24T08:21:11.210671+0000","last_undegraded":"2024-04-24T08:21:11.210671+0000","last_fullsized":"2024-04-24T08:21:11.210671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:46.208560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210162+0000","last_change":"2024-04-24T08:18:47.307800+0000","last_active":"2024-04-24T08:21:11.210162+0000","last_peered":"2024-04-24T08:21:11.210162+0000","last_clean":"2024-04-24T08:21:11.210162+0000","last_became_active":"2024-04-24T08:18:47.307501+0000","last_became_peered":"2024-04-24T08:18:47.307501+0000","last_unstale":"2024-04-24T08:21:11.210162+0000","last_undegraded":"2024-04-24T08:21:11.210162+0000","last_fullsized":"2024-04-24T08:21:11.210162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:04:05.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408128+0000","last_change":"2024-04-24T08:18:46.442811+0000","last_active":"2024-04-24T08:21:11.408128+0000","last_peered":"2024-04-24T08:21:11.408128+0000","last_clean":"2024-04-24T08:21:11.408128+0000","last_became_active":"2024-04-24T08:18:46.442479+0000","last_became_peered":"2024-04-24T08:18:46.442479+0000","last_unstale":"2024-04-24T08:21:11.408128+0000","last_undegraded":"2024-04-24T08:21:11.408128+0000","last_fullsized":"2024-04-24T08:21:11.408128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:24:07.236529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210897+0000","last_change":"2024-04-24T08:18:47.325046+0000","last_active":"2024-04-24T08:21:11.210897+0000","last_peered":"2024-04-24T08:21:11.210897+0000","last_clean":"2024-04-24T08:21:11.210897+0000","last_became_active":"2024-04-24T08:18:47.324720+0000","last_became_peered":"2024-04-24T08:18:47.324720+0000","last_unstale":"2024-04-24T08:21:11.210897+0000","last_undegraded":"2024-04-24T08:21:11.210897+0000","last_fullsized":"2024-04-24T08:21:11.210897+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:11.240388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089232+0000","last_change":"2024-04-24T08:18:46.440572+0000","last_active":"2024-04-24T08:21:11.089232+0000","last_peered":"2024-04-24T08:21:11.089232+0000","last_clean":"2024-04-24T08:21:11.089232+0000","last_became_active":"2024-04-24T08:18:46.440216+0000","last_became_peered":"2024-04-24T08:18:46.440216+0000","last_unstale":"2024-04-24T08:21:11.089232+0000","last_undegraded":"2024-04-24T08:21:11.089232+0000","last_fullsized":"2024-04-24T08:21:11.089232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:51.352874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217093+0000","last_change":"2024-04-24T08:18:47.293465+0000","last_active":"2024-04-24T08:21:11.217093+0000","last_peered":"2024-04-24T08:21:11.217093+0000","last_clean":"2024-04-24T08:21:11.217093+0000","last_became_active":"2024-04-24T08:18:47.293062+0000","last_became_peered":"2024-04-24T08:18:47.293062+0000","last_unstale":"2024-04-24T08:21:11.217093+0000","last_undegraded":"2024-04-24T08:21:11.217093+0000","last_fullsized":"2024-04-24T08:21:11.217093+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:36.029511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408270+0000","last_change":"2024-04-24T08:18:46.451007+0000","last_active":"2024-04-24T08:21:11.408270+0000","last_peered":"2024-04-24T08:21:11.408270+0000","last_clean":"2024-04-24T08:21:11.408270+0000","last_became_active":"2024-04-24T08:18:46.450676+0000","last_became_peered":"2024-04-24T08:18:46.450676+0000","last_unstale":"2024-04-24T08:21:11.408270+0000","last_undegraded":"2024-04-24T08:21:11.408270+0000","last_fullsized":"2024-04-24T08:21:11.408270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:40:00.329311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217356+0000","last_change":"2024-04-24T08:18:47.293859+0000","last_active":"2024-04-24T08:21:11.217356+0000","last_peered":"2024-04-24T08:21:11.217356+0000","last_clean":"2024-04-24T08:21:11.217356+0000","last_became_active":"2024-04-24T08:18:47.293509+0000","last_became_peered":"2024-04-24T08:18:47.293509+0000","last_unstale":"2024-04-24T08:21:11.217356+0000","last_undegraded":"2024-04-24T08:21:11.217356+0000","last_fullsized":"2024-04-24T08:21:11.217356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:41.224037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.356078+0000","last_change":"2024-04-24T08:18:46.436661+0000","last_active":"2024-04-24T08:21:11.356078+0000","last_peered":"2024-04-24T08:21:11.356078+0000","last_clean":"2024-04-24T08:21:11.356078+0000","last_became_active":"2024-04-24T08:18:46.436353+0000","last_became_peered":"2024-04-24T08:18:46.436353+0000","last_unstale":"2024-04-24T08:21:11.356078+0000","last_undegraded":"2024-04-24T08:21:11.356078+0000","last_fullsized":"2024-04-24T08:21:11.356078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:57:25.559706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217135+0000","last_change":"2024-04-24T08:18:57.597302+0000","last_active":"2024-04-24T08:21:11.217135+0000","last_peered":"2024-04-24T08:21:11.217135+0000","last_clean":"2024-04-24T08:21:11.217135+0000","last_became_active":"2024-04-24T08:18:57.591420+0000","last_became_peered":"2024-04-24T08:18:57.591420+0000","last_unstale":"2024-04-24T08:21:11.217135+0000","last_undegraded":"2024-04-24T08:21:11.217135+0000","last_fullsized":"2024-04-24T08:21:11.217135+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:03:34.286436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089289+0000","last_change":"2024-04-24T08:18:46.432499+0000","last_active":"2024-04-24T08:21:11.089289+0000","last_peered":"2024-04-24T08:21:11.089289+0000","last_clean":"2024-04-24T08:21:11.089289+0000","last_became_active":"2024-04-24T08:18:46.432085+0000","last_became_peered":"2024-04-24T08:18:46.432085+0000","last_unstale":"2024-04-24T08:21:11.089289+0000","last_undegraded":"2024-04-24T08:21:11.089289+0000","last_fullsized":"2024-04-24T08:21:11.089289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:03.794091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217293+0000","last_change":"2024-04-24T08:18:47.291193+0000","last_active":"2024-04-24T08:21:11.217293+0000","last_peered":"2024-04-24T08:21:11.217293+0000","last_clean":"2024-04-24T08:21:11.217293+0000","last_became_active":"2024-04-24T08:18:47.289903+0000","last_became_peered":"2024-04-24T08:18:47.289903+0000","last_unstale":"2024-04-24T08:21:11.217293+0000","last_undegraded":"2024-04-24T08:21:11.217293+0000","last_fullsized":"2024-04-24T08:21:11.217293+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:35:18.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210090+0000","last_change":"2024-04-24T08:18:47.292628+0000","last_active":"2024-04-24T08:21:11.210090+0000","last_peered":"2024-04-24T08:21:11.210090+0000","last_clean":"2024-04-24T08:21:11.210090+0000","last_became_active":"2024-04-24T08:18:47.292060+0000","last_became_peered":"2024-04-24T08:18:47.292060+0000","last_unstale":"2024-04-24T08:21:11.210090+0000","last_undegraded":"2024-04-24T08:21:11.210090+0000","last_fullsized":"2024-04-24T08:21:11.210090+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:53.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407920+0000","last_change":"2024-04-24T08:18:47.297121+0000","last_active":"2024-04-24T08:21:11.407920+0000","last_peered":"2024-04-24T08:21:11.407920+0000","last_clean":"2024-04-24T08:21:11.407920+0000","last_became_active":"2024-04-24T08:18:47.296787+0000","last_became_peered":"2024-04-24T08:18:47.296787+0000","last_unstale":"2024-04-24T08:21:11.407920+0000","last_undegraded":"2024-04-24T08:21:11.407920+0000","last_fullsized":"2024-04-24T08:21:11.407920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:44:10.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407884+0000","last_change":"2024-04-24T08:18:47.320669+0000","last_active":"2024-04-24T08:21:11.407884+0000","last_peered":"2024-04-24T08:21:11.407884+0000","last_clean":"2024-04-24T08:21:11.407884+0000","last_became_active":"2024-04-24T08:18:47.320287+0000","last_became_peered":"2024-04-24T08:18:47.320287+0000","last_unstale":"2024-04-24T08:21:11.407884+0000","last_undegraded":"2024-04-24T08:21:11.407884+0000","last_fullsized":"2024-04-24T08:21:11.407884+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:39:03.275189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.417067+0000","last_change":"2024-04-24T08:18:57.585568+0000","last_active":"2024-04-24T08:21:11.417067+0000","last_peered":"2024-04-24T08:21:11.417067+0000","last_clean":"2024-04-24T08:21:11.417067+0000","last_became_active":"2024-04-24T08:18:57.584799+0000","last_became_peered":"2024-04-24T08:18:57.584799+0000","last_unstale":"2024-04-24T08:21:11.417067+0000","last_undegraded":"2024-04-24T08:21:11.417067+0000","last_fullsized":"2024-04-24T08:21:11.417067+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:47.202344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089676+0000","last_change":"2024-04-24T08:18:47.316068+0000","last_active":"2024-04-24T08:21:11.089676+0000","last_peered":"2024-04-24T08:21:11.089676+0000","last_clean":"2024-04-24T08:21:11.089676+0000","last_became_active":"2024-04-24T08:18:47.315647+0000","last_became_peered":"2024-04-24T08:18:47.315647+0000","last_unstale":"2024-04-24T08:21:11.089676+0000","last_undegraded":"2024-04-24T08:21:11.089676+0000","last_fullsized":"2024-04-24T08:21:11.089676+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:33.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210316+0000","last_change":"2024-04-24T08:18:47.294966+0000","last_active":"2024-04-24T08:21:11.210316+0000","last_peered":"2024-04-24T08:21:11.210316+0000","last_clean":"2024-04-24T08:21:11.210316+0000","last_became_active":"2024-04-24T08:18:47.294672+0000","last_became_peered":"2024-04-24T08:18:47.294672+0000","last_unstale":"2024-04-24T08:21:11.210316+0000","last_undegraded":"2024-04-24T08:21:11.210316+0000","last_fullsized":"2024-04-24T08:21:11.210316+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:37.359114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407962+0000","last_change":"2024-04-24T08:18:47.294548+0000","last_active":"2024-04-24T08:21:11.407962+0000","last_peered":"2024-04-24T08:21:11.407962+0000","last_clean":"2024-04-24T08:21:11.407962+0000","last_became_active":"2024-04-24T08:18:47.293589+0000","last_became_peered":"2024-04-24T08:18:47.293589+0000","last_unstale":"2024-04-24T08:21:11.407962+0000","last_undegraded":"2024-04-24T08:21:11.407962+0000","last_fullsized":"2024-04-24T08:21:11.407962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:34:56.208453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211773+0000","last_change":"2024-04-24T08:18:47.313156+0000","last_active":"2024-04-24T08:21:11.211773+0000","last_peered":"2024-04-24T08:21:11.211773+0000","last_clean":"2024-04-24T08:21:11.211773+0000","last_became_active":"2024-04-24T08:18:47.312829+0000","last_became_peered":"2024-04-24T08:18:47.312829+0000","last_unstale":"2024-04-24T08:21:11.211773+0000","last_undegraded":"2024-04-24T08:21:11.211773+0000","last_fullsized":"2024-04-24T08:21:11.211773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:28:36.638514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219278+0000","last_change":"2024-04-24T08:18:47.320022+0000","last_active":"2024-04-24T08:21:11.219278+0000","last_peered":"2024-04-24T08:21:11.219278+0000","last_clean":"2024-04-24T08:21:11.219278+0000","last_became_active":"2024-04-24T08:18:47.319707+0000","last_became_peered":"2024-04-24T08:18:47.319707+0000","last_unstale":"2024-04-24T08:21:11.219278+0000","last_undegraded":"2024-04-24T08:21:11.219278+0000","last_fullsized":"2024-04-24T08:21:11.219278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:51.330874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210674+0000","last_change":"2024-04-24T08:18:47.323187+0000","last_active":"2024-04-24T08:21:11.210674+0000","last_peered":"2024-04-24T08:21:11.210674+0000","last_clean":"2024-04-24T08:21:11.210674+0000","last_became_active":"2024-04-24T08:18:47.320258+0000","last_became_peered":"2024-04-24T08:18:47.320258+0000","last_unstale":"2024-04-24T08:21:11.210674+0000","last_undegraded":"2024-04-24T08:21:11.210674+0000","last_fullsized":"2024-04-24T08:21:11.210674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:42:32.770950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409986+0000","last_change":"2024-04-24T08:18:47.321926+0000","last_active":"2024-04-24T08:21:11.409986+0000","last_peered":"2024-04-24T08:21:11.409986+0000","last_clean":"2024-04-24T08:21:11.409986+0000","last_became_active":"2024-04-24T08:18:47.321584+0000","last_became_peered":"2024-04-24T08:18:47.321584+0000","last_unstale":"2024-04-24T08:21:11.409986+0000","last_undegraded":"2024-04-24T08:21:11.409986+0000","last_fullsized":"2024-04-24T08:21:11.409986+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:24.357157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.093027+0000","last_change":"2024-04-24T08:18:47.313310+0000","last_active":"2024-04-24T08:21:11.093027+0000","last_peered":"2024-04-24T08:21:11.093027+0000","last_clean":"2024-04-24T08:21:11.093027+0000","last_became_active":"2024-04-24T08:18:47.312968+0000","last_became_peered":"2024-04-24T08:18:47.312968+0000","last_unstale":"2024-04-24T08:21:11.093027+0000","last_undegraded":"2024-04-24T08:21:11.093027+0000","last_fullsized":"2024-04-24T08:21:11.093027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:08.967577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210366+0000","last_change":"2024-04-24T08:18:57.591543+0000","last_active":"2024-04-24T08:21:11.210366+0000","last_peered":"2024-04-24T08:21:11.210366+0000","last_clean":"2024-04-24T08:21:11.210366+0000","last_became_active":"2024-04-24T08:18:57.591043+0000","last_became_peered":"2024-04-24T08:18:57.591043+0000","last_unstale":"2024-04-24T08:21:11.210366+0000","last_undegraded":"2024-04-24T08:21:11.210366+0000","last_fullsized":"2024-04-24T08:21:11.210366+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:52:38.669469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597142,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":966,"num_read_kb":592,"num_write":291,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1791426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285609,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28380,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716772,"statfs":{"total":95995035648,"available":95965974528,"internally_reserved":0,"allocated":1392640,"data_stored":871799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547246,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1765376,"data_stored":1197725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710773,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36720,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708432,"statfs":{"total":95995035648,"available":95957434368,"internally_reserved":0,"allocated":1413120,"data_stored":872152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972411,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":478739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1609728,"data_stored":1072293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430277,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":478208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1073152,"data_stored":544702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":479479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":532842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1003218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888145,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":478645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37112,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708040,"statfs":{"total":95995035648,"available":95957032960,"internally_reserved":0,"allocated":1757184,"data_stored":1202795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":200661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:21:17.194 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-24T08:21:17.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:21:17.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 -- 172.21.15.146:0/648897406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c102ae0 msgr2=0x7f8b8c104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 --2- 172.21.15.146:0/648897406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c104ed0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009860 tx=0x7f8b7c02a340 comp rx=0 tx=0).stop 2024-04-24T08:21:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 -- 172.21.15.146:0/648897406 shutdown_connections 2024-04-24T08:21:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 --2- 172.21.15.146:0/648897406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8c107d40 0x7f8b8c10a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 --2- 172.21.15.146:0/648897406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c105410 0x7f8b8c107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 --2- 172.21.15.146:0/648897406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c104ed0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 -- 172.21.15.146:0/648897406 >> 172.21.15.146:0/648897406 conn(0x7f8b8c0fc4f0 msgr2=0x7f8b8c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 -- 172.21.15.146:0/648897406 shutdown_connections 2024-04-24T08:21:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.405+0000 7f8b91800640 1 -- 172.21.15.146:0/648897406 wait complete. 2024-04-24T08:21:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.407+0000 7f8b91800640 1 Processor -- start 2024-04-24T08:21:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.407+0000 7f8b91800640 1 -- start start 2024-04-24T08:21:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.407+0000 7f8b91800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.407+0000 7f8b91800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8c105410 0x7f8b8c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.407+0000 7f8b91800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c107d40 0x7f8b8c1a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b91800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c10dcf0 con 0x7f8b8c102ae0 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b91800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c10db70 con 0x7f8b8c105410 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b91800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c10de70 con 0x7f8b8c107d40 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c19a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39188/0 (socket says 172.21.15.146:39188) 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8aa00640 1 -- 172.21.15.146:0/2192453341 learned_addr learned my addr 172.21.15.146:0/2192453341 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8a000640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8c105410 0x7f8b8c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8b400640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c107d40 0x7f8b8c1a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8a000640 1 -- 172.21.15.146:0/2192453341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c107d40 msgr2=0x7f8b8c1a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8a000640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c107d40 0x7f8b8c1a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8a000640 1 -- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c102ae0 msgr2=0x7f8b8c19a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8a000640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c19a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8a000640 1 -- 172.21.15.146:0/2192453341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8c1a21a0 con 0x7f8b8c105410 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.408+0000 7f8b8aa00640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c19a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.409+0000 7f8b8a000640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8c105410 0x7f8b8c19aac0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c06b1a0 tx=0x7f8b7800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.409+0000 7f8b83400640 1 -- 172.21.15.146:0/2192453341 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7800cc90 con 0x7f8b8c105410 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.409+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8c1a2490 con 0x7f8b8c105410 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.409+0000 7f8b83400640 1 -- 172.21.15.146:0/2192453341 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b7800ce30 con 0x7f8b8c105410 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.409+0000 7f8b83400640 1 -- 172.21.15.146:0/2192453341 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78010430 con 0x7f8b8c105410 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.409+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8c1a2a30 con 0x7f8b8c105410 2024-04-24T08:21:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.411+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8c109890 con 0x7f8b8c105410 2024-04-24T08:21:17.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.413+0000 7f8b83400640 1 -- 172.21.15.146:0/2192453341 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f8b780035f0 con 0x7f8b8c105410 2024-04-24T08:21:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.414+0000 7f8b83400640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b700a3ec0 0x7f8b700a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.414+0000 7f8b83400640 1 -- 172.21.15.146:0/2192453341 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8b780d0cc0 con 0x7f8b8c105410 2024-04-24T08:21:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.414+0000 7f8b8aa00640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b700a3ec0 0x7f8b700a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.415+0000 7f8b8aa00640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b700a3ec0 0x7f8b700a6380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c000c00 tx=0x7f8b7c030550 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:17.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.418+0000 7f8b83400640 1 -- 172.21.15.146:0/2192453341 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b780d7080 con 0x7f8b8c105410 2024-04-24T08:21:17.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.622+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b8c062440 con 0x7f8b700a3ec0 2024-04-24T08:21:17.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.630+0000 7f8b83400640 1 -- 172.21.15.146:0/2192453341 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257773 (secure 0 0 0) 0x7f8b8c062440 con 0x7f8b700a3ec0 2024-04-24T08:21:17.633 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:17.636 INFO:teuthology.orchestra.run.smithi146.stderr:dumped all 2024-04-24T08:21:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b700a3ec0 msgr2=0x7f8b700a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b700a3ec0 0x7f8b700a6380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c000c00 tx=0x7f8b7c030550 comp rx=0 tx=0).stop 2024-04-24T08:21:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8c105410 msgr2=0x7f8b8c19aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8c105410 0x7f8b8c19aac0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c06b1a0 tx=0x7f8b7800ed90 comp rx=0 tx=0).stop 2024-04-24T08:21:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 shutdown_connections 2024-04-24T08:21:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b700a3ec0 0x7f8b700a6380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8c107d40 0x7f8b8c1a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b8c105410 0x7f8b8c19aac0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 --2- 172.21.15.146:0/2192453341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c102ae0 0x7f8b8c19a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 >> 172.21.15.146:0/2192453341 conn(0x7f8b8c0fc4f0 msgr2=0x7f8b8c102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 shutdown_connections 2024-04-24T08:21:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.636+0000 7f8b91800640 1 -- 172.21.15.146:0/2192453341 wait complete. 2024-04-24T08:21:17.654 INFO:teuthology.orchestra.run.smithi146.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-24T08:21:17.111381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1187744,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1102,"num_read_kb":711,"num_write":474,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180853,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377040,"kb_used_data":15312,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124564784,"statfs":{"total":1151940427776,"available":1151554338816,"internally_reserved":0,"allocated":15679488,"data_stored":9212597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.979498"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210587+0000","last_change":"2024-04-24T08:18:47.309666+0000","last_active":"2024-04-24T08:21:11.210587+0000","last_peered":"2024-04-24T08:21:11.210587+0000","last_clean":"2024-04-24T08:21:11.210587+0000","last_became_active":"2024-04-24T08:18:47.309413+0000","last_became_peered":"2024-04-24T08:18:47.309413+0000","last_unstale":"2024-04-24T08:21:11.210587+0000","last_undegraded":"2024-04-24T08:21:11.210587+0000","last_fullsized":"2024-04-24T08:21:11.210587+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:29.662505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416384+0000","last_change":"2024-04-24T08:18:47.310321+0000","last_active":"2024-04-24T08:21:11.416384+0000","last_peered":"2024-04-24T08:21:11.416384+0000","last_clean":"2024-04-24T08:21:11.416384+0000","last_became_active":"2024-04-24T08:18:47.309956+0000","last_became_peered":"2024-04-24T08:18:47.309956+0000","last_unstale":"2024-04-24T08:21:11.416384+0000","last_undegraded":"2024-04-24T08:21:11.416384+0000","last_fullsized":"2024-04-24T08:21:11.416384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:51:48.432725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416668+0000","last_change":"2024-04-24T08:18:47.312153+0000","last_active":"2024-04-24T08:21:11.416668+0000","last_peered":"2024-04-24T08:21:11.416668+0000","last_clean":"2024-04-24T08:21:11.416668+0000","last_became_active":"2024-04-24T08:18:47.309465+0000","last_became_peered":"2024-04-24T08:18:47.309465+0000","last_unstale":"2024-04-24T08:21:11.416668+0000","last_undegraded":"2024-04-24T08:21:11.416668+0000","last_fullsized":"2024-04-24T08:21:11.416668+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:55.829143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089098+0000","last_change":"2024-04-24T08:18:47.297017+0000","last_active":"2024-04-24T08:21:11.089098+0000","last_peered":"2024-04-24T08:21:11.089098+0000","last_clean":"2024-04-24T08:21:11.089098+0000","last_became_active":"2024-04-24T08:18:47.296693+0000","last_became_peered":"2024-04-24T08:18:47.296693+0000","last_unstale":"2024-04-24T08:21:11.089098+0000","last_undegraded":"2024-04-24T08:21:11.089098+0000","last_fullsized":"2024-04-24T08:21:11.089098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:23.400403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089279+0000","last_change":"2024-04-24T08:18:47.315376+0000","last_active":"2024-04-24T08:21:11.089279+0000","last_peered":"2024-04-24T08:21:11.089279+0000","last_clean":"2024-04-24T08:21:11.089279+0000","last_became_active":"2024-04-24T08:18:47.315027+0000","last_became_peered":"2024-04-24T08:18:47.315027+0000","last_unstale":"2024-04-24T08:21:11.089279+0000","last_undegraded":"2024-04-24T08:21:11.089279+0000","last_fullsized":"2024-04-24T08:21:11.089279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:12.504515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222417+0000","last_change":"2024-04-24T08:18:47.306388+0000","last_active":"2024-04-24T08:21:11.222417+0000","last_peered":"2024-04-24T08:21:11.222417+0000","last_clean":"2024-04-24T08:21:11.222417+0000","last_became_active":"2024-04-24T08:18:47.306082+0000","last_became_peered":"2024-04-24T08:18:47.306082+0000","last_unstale":"2024-04-24T08:21:11.222417+0000","last_undegraded":"2024-04-24T08:21:11.222417+0000","last_fullsized":"2024-04-24T08:21:11.222417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:34:52.612799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216773+0000","last_change":"2024-04-24T08:18:47.303336+0000","last_active":"2024-04-24T08:21:11.216773+0000","last_peered":"2024-04-24T08:21:11.216773+0000","last_clean":"2024-04-24T08:21:11.216773+0000","last_became_active":"2024-04-24T08:18:47.303008+0000","last_became_peered":"2024-04-24T08:18:47.303008+0000","last_unstale":"2024-04-24T08:21:11.216773+0000","last_undegraded":"2024-04-24T08:21:11.216773+0000","last_fullsized":"2024-04-24T08:21:11.216773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:57.404693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210549+0000","last_change":"2024-04-24T08:18:47.316672+0000","last_active":"2024-04-24T08:21:11.210549+0000","last_peered":"2024-04-24T08:21:11.210549+0000","last_clean":"2024-04-24T08:21:11.210549+0000","last_became_active":"2024-04-24T08:18:47.315741+0000","last_became_peered":"2024-04-24T08:18:47.315741+0000","last_unstale":"2024-04-24T08:21:11.210549+0000","last_undegraded":"2024-04-24T08:21:11.210549+0000","last_fullsized":"2024-04-24T08:21:11.210549+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:29.156710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409994+0000","last_change":"2024-04-24T08:18:47.291733+0000","last_active":"2024-04-24T08:21:11.409994+0000","last_peered":"2024-04-24T08:21:11.409994+0000","last_clean":"2024-04-24T08:21:11.409994+0000","last_became_active":"2024-04-24T08:18:47.291432+0000","last_became_peered":"2024-04-24T08:18:47.291432+0000","last_unstale":"2024-04-24T08:21:11.409994+0000","last_undegraded":"2024-04-24T08:21:11.409994+0000","last_fullsized":"2024-04-24T08:21:11.409994+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:38:03.601540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222808+0000","last_change":"2024-04-24T08:18:47.294544+0000","last_active":"2024-04-24T08:21:11.222808+0000","last_peered":"2024-04-24T08:21:11.222808+0000","last_clean":"2024-04-24T08:21:11.222808+0000","last_became_active":"2024-04-24T08:18:47.294183+0000","last_became_peered":"2024-04-24T08:18:47.294183+0000","last_unstale":"2024-04-24T08:21:11.222808+0000","last_undegraded":"2024-04-24T08:21:11.222808+0000","last_fullsized":"2024-04-24T08:21:11.222808+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:53.082644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217198+0000","last_change":"2024-04-24T08:18:47.292615+0000","last_active":"2024-04-24T08:21:11.217198+0000","last_peered":"2024-04-24T08:21:11.217198+0000","last_clean":"2024-04-24T08:21:11.217198+0000","last_became_active":"2024-04-24T08:18:47.292300+0000","last_became_peered":"2024-04-24T08:18:47.292300+0000","last_unstale":"2024-04-24T08:21:11.217198+0000","last_undegraded":"2024-04-24T08:21:11.217198+0000","last_fullsized":"2024-04-24T08:21:11.217198+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:58.293253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089115+0000","last_change":"2024-04-24T08:18:47.298388+0000","last_active":"2024-04-24T08:21:11.089115+0000","last_peered":"2024-04-24T08:21:11.089115+0000","last_clean":"2024-04-24T08:21:11.089115+0000","last_became_active":"2024-04-24T08:18:47.298008+0000","last_became_peered":"2024-04-24T08:18:47.298008+0000","last_unstale":"2024-04-24T08:21:11.089115+0000","last_undegraded":"2024-04-24T08:21:11.089115+0000","last_fullsized":"2024-04-24T08:21:11.089115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:10:00.904613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.092591+0000","last_change":"2024-04-24T08:18:47.298809+0000","last_active":"2024-04-24T08:21:11.092591+0000","last_peered":"2024-04-24T08:21:11.092591+0000","last_clean":"2024-04-24T08:21:11.092591+0000","last_became_active":"2024-04-24T08:18:47.298103+0000","last_became_peered":"2024-04-24T08:18:47.298103+0000","last_unstale":"2024-04-24T08:21:11.092591+0000","last_undegraded":"2024-04-24T08:21:11.092591+0000","last_fullsized":"2024-04-24T08:21:11.092591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:38:53.625721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415961+0000","last_change":"2024-04-24T08:18:47.314572+0000","last_active":"2024-04-24T08:21:11.415961+0000","last_peered":"2024-04-24T08:21:11.415961+0000","last_clean":"2024-04-24T08:21:11.415961+0000","last_became_active":"2024-04-24T08:18:47.313372+0000","last_became_peered":"2024-04-24T08:18:47.313372+0000","last_unstale":"2024-04-24T08:21:11.415961+0000","last_undegraded":"2024-04-24T08:21:11.415961+0000","last_fullsized":"2024-04-24T08:21:11.415961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:56:01.954909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415978+0000","last_change":"2024-04-24T08:18:47.314822+0000","last_active":"2024-04-24T08:21:11.415978+0000","last_peered":"2024-04-24T08:21:11.415978+0000","last_clean":"2024-04-24T08:21:11.415978+0000","last_became_active":"2024-04-24T08:18:47.310774+0000","last_became_peered":"2024-04-24T08:18:47.310774+0000","last_unstale":"2024-04-24T08:21:11.415978+0000","last_undegraded":"2024-04-24T08:21:11.415978+0000","last_fullsized":"2024-04-24T08:21:11.415978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:24.934100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409946+0000","last_change":"2024-04-24T08:18:47.307208+0000","last_active":"2024-04-24T08:21:11.409946+0000","last_peered":"2024-04-24T08:21:11.409946+0000","last_clean":"2024-04-24T08:21:11.409946+0000","last_became_active":"2024-04-24T08:18:47.306889+0000","last_became_peered":"2024-04-24T08:18:47.306889+0000","last_unstale":"2024-04-24T08:21:11.409946+0000","last_undegraded":"2024-04-24T08:21:11.409946+0000","last_fullsized":"2024-04-24T08:21:11.409946+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:20:01.438388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210570+0000","last_change":"2024-04-24T08:18:47.322929+0000","last_active":"2024-04-24T08:21:11.210570+0000","last_peered":"2024-04-24T08:21:11.210570+0000","last_clean":"2024-04-24T08:21:11.210570+0000","last_became_active":"2024-04-24T08:18:47.322593+0000","last_became_peered":"2024-04-24T08:18:47.322593+0000","last_unstale":"2024-04-24T08:21:11.210570+0000","last_undegraded":"2024-04-24T08:21:11.210570+0000","last_fullsized":"2024-04-24T08:21:11.210570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:09.114988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.208062+0000","last_change":"2024-04-24T08:18:47.304886+0000","last_active":"2024-04-24T08:21:11.208062+0000","last_peered":"2024-04-24T08:21:11.208062+0000","last_clean":"2024-04-24T08:21:11.208062+0000","last_became_active":"2024-04-24T08:18:47.302631+0000","last_became_peered":"2024-04-24T08:18:47.302631+0000","last_unstale":"2024-04-24T08:21:11.208062+0000","last_undegraded":"2024-04-24T08:21:11.208062+0000","last_fullsized":"2024-04-24T08:21:11.208062+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:13.186554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407979+0000","last_change":"2024-04-24T08:18:47.316828+0000","last_active":"2024-04-24T08:21:11.407979+0000","last_peered":"2024-04-24T08:21:11.407979+0000","last_clean":"2024-04-24T08:21:11.407979+0000","last_became_active":"2024-04-24T08:18:47.316540+0000","last_became_peered":"2024-04-24T08:18:47.316540+0000","last_unstale":"2024-04-24T08:21:11.407979+0000","last_undegraded":"2024-04-24T08:21:11.407979+0000","last_fullsized":"2024-04-24T08:21:11.407979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:41:42.837946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207816+0000","last_change":"2024-04-24T08:18:46.444744+0000","last_active":"2024-04-24T08:21:11.207816+0000","last_peered":"2024-04-24T08:21:11.207816+0000","last_clean":"2024-04-24T08:21:11.207816+0000","last_became_active":"2024-04-24T08:18:46.442495+0000","last_became_peered":"2024-04-24T08:18:46.442495+0000","last_unstale":"2024-04-24T08:21:11.207816+0000","last_undegraded":"2024-04-24T08:21:11.207816+0000","last_fullsized":"2024-04-24T08:21:11.207816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:33:32.270984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211523+0000","last_change":"2024-04-24T08:18:47.320462+0000","last_active":"2024-04-24T08:21:11.211523+0000","last_peered":"2024-04-24T08:21:11.211523+0000","last_clean":"2024-04-24T08:21:11.211523+0000","last_became_active":"2024-04-24T08:18:47.319267+0000","last_became_peered":"2024-04-24T08:18:47.319267+0000","last_unstale":"2024-04-24T08:21:11.211523+0000","last_undegraded":"2024-04-24T08:21:11.211523+0000","last_fullsized":"2024-04-24T08:21:11.211523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:50:15.882077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211656+0000","last_change":"2024-04-24T08:18:46.453278+0000","last_active":"2024-04-24T08:21:11.211656+0000","last_peered":"2024-04-24T08:21:11.211656+0000","last_clean":"2024-04-24T08:21:11.211656+0000","last_became_active":"2024-04-24T08:18:46.452913+0000","last_became_peered":"2024-04-24T08:18:46.452913+0000","last_unstale":"2024-04-24T08:21:11.211656+0000","last_undegraded":"2024-04-24T08:21:11.211656+0000","last_fullsized":"2024-04-24T08:21:11.211656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:48.272190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416239+0000","last_change":"2024-04-24T08:18:47.323610+0000","last_active":"2024-04-24T08:21:11.416239+0000","last_peered":"2024-04-24T08:21:11.416239+0000","last_clean":"2024-04-24T08:21:11.416239+0000","last_became_active":"2024-04-24T08:18:47.322795+0000","last_became_peered":"2024-04-24T08:18:47.322795+0000","last_unstale":"2024-04-24T08:21:11.416239+0000","last_undegraded":"2024-04-24T08:21:11.416239+0000","last_fullsized":"2024-04-24T08:21:11.416239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:22:32.951564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089694+0000","last_change":"2024-04-24T08:18:57.730353+0000","last_active":"2024-04-24T08:21:11.089694+0000","last_peered":"2024-04-24T08:21:11.089694+0000","last_clean":"2024-04-24T08:21:11.089694+0000","last_became_active":"2024-04-24T08:18:57.725936+0000","last_became_peered":"2024-04-24T08:18:57.725936+0000","last_unstale":"2024-04-24T08:21:11.089694+0000","last_undegraded":"2024-04-24T08:21:11.089694+0000","last_fullsized":"2024-04-24T08:21:11.089694+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:00:27.764723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410161+0000","last_change":"2024-04-24T08:18:47.304884+0000","last_active":"2024-04-24T08:21:11.410161+0000","last_peered":"2024-04-24T08:21:11.410161+0000","last_clean":"2024-04-24T08:21:11.410161+0000","last_became_active":"2024-04-24T08:18:47.304520+0000","last_became_peered":"2024-04-24T08:18:47.304520+0000","last_unstale":"2024-04-24T08:21:11.410161+0000","last_undegraded":"2024-04-24T08:21:11.410161+0000","last_fullsized":"2024-04-24T08:21:11.410161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:42.325010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089422+0000","last_change":"2024-04-24T08:18:46.441073+0000","last_active":"2024-04-24T08:21:11.089422+0000","last_peered":"2024-04-24T08:21:11.089422+0000","last_clean":"2024-04-24T08:21:11.089422+0000","last_became_active":"2024-04-24T08:18:46.440721+0000","last_became_peered":"2024-04-24T08:18:46.440721+0000","last_unstale":"2024-04-24T08:21:11.089422+0000","last_undegraded":"2024-04-24T08:21:11.089422+0000","last_fullsized":"2024-04-24T08:21:11.089422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:51:39.516458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219409+0000","last_change":"2024-04-24T08:18:47.293947+0000","last_active":"2024-04-24T08:21:11.219409+0000","last_peered":"2024-04-24T08:21:11.219409+0000","last_clean":"2024-04-24T08:21:11.219409+0000","last_became_active":"2024-04-24T08:18:47.293623+0000","last_became_peered":"2024-04-24T08:18:47.293623+0000","last_unstale":"2024-04-24T08:21:11.219409+0000","last_undegraded":"2024-04-24T08:21:11.219409+0000","last_fullsized":"2024-04-24T08:21:11.219409+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:33:23.620544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.741488+0000","last_change":"2024-04-24T08:18:46.439909+0000","last_active":"2024-04-24T08:21:11.741488+0000","last_peered":"2024-04-24T08:21:11.741488+0000","last_clean":"2024-04-24T08:21:11.741488+0000","last_became_active":"2024-04-24T08:18:46.439597+0000","last_became_peered":"2024-04-24T08:18:46.439597+0000","last_unstale":"2024-04-24T08:21:11.741488+0000","last_undegraded":"2024-04-24T08:21:11.741488+0000","last_fullsized":"2024-04-24T08:21:11.741488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:49:11.536727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089065+0000","last_change":"2024-04-24T08:18:47.306480+0000","last_active":"2024-04-24T08:21:11.089065+0000","last_peered":"2024-04-24T08:21:11.089065+0000","last_clean":"2024-04-24T08:21:11.089065+0000","last_became_active":"2024-04-24T08:18:47.306104+0000","last_became_peered":"2024-04-24T08:18:47.306104+0000","last_unstale":"2024-04-24T08:21:11.089065+0000","last_undegraded":"2024-04-24T08:21:11.089065+0000","last_fullsized":"2024-04-24T08:21:11.089065+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:56.340836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222262+0000","last_change":"2024-04-24T08:18:46.439381+0000","last_active":"2024-04-24T08:21:11.222262+0000","last_peered":"2024-04-24T08:21:11.222262+0000","last_clean":"2024-04-24T08:21:11.222262+0000","last_became_active":"2024-04-24T08:18:46.439033+0000","last_became_peered":"2024-04-24T08:18:46.439033+0000","last_unstale":"2024-04-24T08:21:11.222262+0000","last_undegraded":"2024-04-24T08:21:11.222262+0000","last_fullsized":"2024-04-24T08:21:11.222262+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:48:28.833066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219251+0000","last_change":"2024-04-24T08:18:57.595549+0000","last_active":"2024-04-24T08:21:11.219251+0000","last_peered":"2024-04-24T08:21:11.219251+0000","last_clean":"2024-04-24T08:21:11.219251+0000","last_became_active":"2024-04-24T08:18:57.595098+0000","last_became_peered":"2024-04-24T08:18:57.595098+0000","last_unstale":"2024-04-24T08:21:11.219251+0000","last_undegraded":"2024-04-24T08:21:11.219251+0000","last_fullsized":"2024-04-24T08:21:11.219251+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:09:22.722996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.094360+0000","last_change":"2024-04-24T08:18:46.448151+0000","last_active":"2024-04-24T08:21:11.094360+0000","last_peered":"2024-04-24T08:21:11.094360+0000","last_clean":"2024-04-24T08:21:11.094360+0000","last_became_active":"2024-04-24T08:18:46.447077+0000","last_became_peered":"2024-04-24T08:18:46.447077+0000","last_unstale":"2024-04-24T08:21:11.094360+0000","last_undegraded":"2024-04-24T08:21:11.094360+0000","last_fullsized":"2024-04-24T08:21:11.094360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:00.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210379+0000","last_change":"2024-04-24T08:18:47.319423+0000","last_active":"2024-04-24T08:21:11.210379+0000","last_peered":"2024-04-24T08:21:11.210379+0000","last_clean":"2024-04-24T08:21:11.210379+0000","last_became_active":"2024-04-24T08:18:47.316284+0000","last_became_peered":"2024-04-24T08:18:47.316284+0000","last_unstale":"2024-04-24T08:21:11.210379+0000","last_undegraded":"2024-04-24T08:21:11.210379+0000","last_fullsized":"2024-04-24T08:21:11.210379+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:55.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219135+0000","last_change":"2024-04-24T08:18:46.436152+0000","last_active":"2024-04-24T08:21:11.219135+0000","last_peered":"2024-04-24T08:21:11.219135+0000","last_clean":"2024-04-24T08:21:11.219135+0000","last_became_active":"2024-04-24T08:18:46.435835+0000","last_became_peered":"2024-04-24T08:18:46.435835+0000","last_unstale":"2024-04-24T08:21:11.219135+0000","last_undegraded":"2024-04-24T08:21:11.219135+0000","last_fullsized":"2024-04-24T08:21:11.219135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:12:30.536455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207423+0000","last_change":"2024-04-24T08:18:57.725728+0000","last_active":"2024-04-24T08:21:11.207423+0000","last_peered":"2024-04-24T08:21:11.207423+0000","last_clean":"2024-04-24T08:21:11.207423+0000","last_became_active":"2024-04-24T08:18:57.724704+0000","last_became_peered":"2024-04-24T08:18:57.724704+0000","last_unstale":"2024-04-24T08:21:11.207423+0000","last_undegraded":"2024-04-24T08:21:11.207423+0000","last_fullsized":"2024-04-24T08:21:11.207423+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:27.501006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":167,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.740986+0000","last_change":"2024-04-24T08:18:46.446652+0000","last_active":"2024-04-24T08:21:11.740986+0000","last_peered":"2024-04-24T08:21:11.740986+0000","last_clean":"2024-04-24T08:21:11.740986+0000","last_became_active":"2024-04-24T08:18:46.446348+0000","last_became_peered":"2024-04-24T08:18:46.446348+0000","last_unstale":"2024-04-24T08:21:11.740986+0000","last_undegraded":"2024-04-24T08:21:11.740986+0000","last_fullsized":"2024-04-24T08:21:11.740986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:25.791359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410343+0000","last_change":"2024-04-24T08:18:47.309072+0000","last_active":"2024-04-24T08:21:11.410343+0000","last_peered":"2024-04-24T08:21:11.410343+0000","last_clean":"2024-04-24T08:21:11.410343+0000","last_became_active":"2024-04-24T08:18:47.308729+0000","last_became_peered":"2024-04-24T08:18:47.308729+0000","last_unstale":"2024-04-24T08:21:11.410343+0000","last_undegraded":"2024-04-24T08:21:11.410343+0000","last_fullsized":"2024-04-24T08:21:11.410343+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:58.256876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217308+0000","last_change":"2024-04-24T08:18:46.439190+0000","last_active":"2024-04-24T08:21:11.217308+0000","last_peered":"2024-04-24T08:21:11.217308+0000","last_clean":"2024-04-24T08:21:11.217308+0000","last_became_active":"2024-04-24T08:18:46.438442+0000","last_became_peered":"2024-04-24T08:18:46.438442+0000","last_unstale":"2024-04-24T08:21:11.217308+0000","last_undegraded":"2024-04-24T08:21:11.217308+0000","last_fullsized":"2024-04-24T08:21:11.217308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:42.614563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416325+0000","last_change":"2024-04-24T08:18:47.311447+0000","last_active":"2024-04-24T08:21:11.416325+0000","last_peered":"2024-04-24T08:21:11.416325+0000","last_clean":"2024-04-24T08:21:11.416325+0000","last_became_active":"2024-04-24T08:18:47.310939+0000","last_became_peered":"2024-04-24T08:18:47.310939+0000","last_unstale":"2024-04-24T08:21:11.416325+0000","last_undegraded":"2024-04-24T08:21:11.416325+0000","last_fullsized":"2024-04-24T08:21:11.416325+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:42.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219601+0000","last_change":"2024-04-24T08:18:46.450596+0000","last_active":"2024-04-24T08:21:11.219601+0000","last_peered":"2024-04-24T08:21:11.219601+0000","last_clean":"2024-04-24T08:21:11.219601+0000","last_became_active":"2024-04-24T08:18:46.450226+0000","last_became_peered":"2024-04-24T08:18:46.450226+0000","last_unstale":"2024-04-24T08:21:11.219601+0000","last_undegraded":"2024-04-24T08:21:11.219601+0000","last_fullsized":"2024-04-24T08:21:11.219601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:05.722507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409988+0000","last_change":"2024-04-24T08:18:47.295266+0000","last_active":"2024-04-24T08:21:11.409988+0000","last_peered":"2024-04-24T08:21:11.409988+0000","last_clean":"2024-04-24T08:21:11.409988+0000","last_became_active":"2024-04-24T08:18:47.294963+0000","last_became_peered":"2024-04-24T08:18:47.294963+0000","last_unstale":"2024-04-24T08:21:11.409988+0000","last_undegraded":"2024-04-24T08:21:11.409988+0000","last_fullsized":"2024-04-24T08:21:11.409988+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:07.638921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":348,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.746361+0000","last_change":"2024-04-24T08:18:46.435524+0000","last_active":"2024-04-24T08:21:11.746361+0000","last_peered":"2024-04-24T08:21:11.746361+0000","last_clean":"2024-04-24T08:21:11.746361+0000","last_became_active":"2024-04-24T08:18:46.435212+0000","last_became_peered":"2024-04-24T08:18:46.435212+0000","last_unstale":"2024-04-24T08:21:11.746361+0000","last_undegraded":"2024-04-24T08:21:11.746361+0000","last_fullsized":"2024-04-24T08:21:11.746361+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:45.575905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66701,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":291,"num_read_kb":210,"num_write":36,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089464+0000","last_change":"2024-04-24T08:18:47.293648+0000","last_active":"2024-04-24T08:21:11.089464+0000","last_peered":"2024-04-24T08:21:11.089464+0000","last_clean":"2024-04-24T08:21:11.089464+0000","last_became_active":"2024-04-24T08:18:47.293260+0000","last_became_peered":"2024-04-24T08:18:47.293260+0000","last_unstale":"2024-04-24T08:21:11.089464+0000","last_undegraded":"2024-04-24T08:21:11.089464+0000","last_fullsized":"2024-04-24T08:21:11.089464+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:36.462606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408233+0000","last_change":"2024-04-24T08:18:46.441401+0000","last_active":"2024-04-24T08:21:11.408233+0000","last_peered":"2024-04-24T08:21:11.408233+0000","last_clean":"2024-04-24T08:21:11.408233+0000","last_became_active":"2024-04-24T08:18:46.440985+0000","last_became_peered":"2024-04-24T08:18:46.440985+0000","last_unstale":"2024-04-24T08:21:11.408233+0000","last_undegraded":"2024-04-24T08:21:11.408233+0000","last_fullsized":"2024-04-24T08:21:11.408233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:45.201518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089310+0000","last_change":"2024-04-24T08:18:47.320928+0000","last_active":"2024-04-24T08:21:11.089310+0000","last_peered":"2024-04-24T08:21:11.089310+0000","last_clean":"2024-04-24T08:21:11.089310+0000","last_became_active":"2024-04-24T08:18:47.320515+0000","last_became_peered":"2024-04-24T08:18:47.320515+0000","last_unstale":"2024-04-24T08:21:11.089310+0000","last_undegraded":"2024-04-24T08:21:11.089310+0000","last_fullsized":"2024-04-24T08:21:11.089310+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:55:42.899438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219120+0000","last_change":"2024-04-24T08:18:46.437677+0000","last_active":"2024-04-24T08:21:11.219120+0000","last_peered":"2024-04-24T08:21:11.219120+0000","last_clean":"2024-04-24T08:21:11.219120+0000","last_became_active":"2024-04-24T08:18:46.437364+0000","last_became_peered":"2024-04-24T08:18:46.437364+0000","last_unstale":"2024-04-24T08:21:11.219120+0000","last_undegraded":"2024-04-24T08:21:11.219120+0000","last_fullsized":"2024-04-24T08:21:11.219120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:25:55.408887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217154+0000","last_change":"2024-04-24T08:18:47.298396+0000","last_active":"2024-04-24T08:21:11.217154+0000","last_peered":"2024-04-24T08:21:11.217154+0000","last_clean":"2024-04-24T08:21:11.217154+0000","last_became_active":"2024-04-24T08:18:47.298021+0000","last_became_peered":"2024-04-24T08:18:47.298021+0000","last_unstale":"2024-04-24T08:21:11.217154+0000","last_undegraded":"2024-04-24T08:21:11.217154+0000","last_fullsized":"2024-04-24T08:21:11.217154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:46:24.015897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":170,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.830237+0000","last_change":"2024-04-24T08:18:46.436612+0000","last_active":"2024-04-24T08:21:11.830237+0000","last_peered":"2024-04-24T08:21:11.830237+0000","last_clean":"2024-04-24T08:21:11.830237+0000","last_became_active":"2024-04-24T08:18:46.436211+0000","last_became_peered":"2024-04-24T08:18:46.436211+0000","last_unstale":"2024-04-24T08:21:11.830237+0000","last_undegraded":"2024-04-24T08:21:11.830237+0000","last_fullsized":"2024-04-24T08:21:11.830237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:49:05.180130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217055+0000","last_change":"2024-04-24T08:18:47.290849+0000","last_active":"2024-04-24T08:21:11.217055+0000","last_peered":"2024-04-24T08:21:11.217055+0000","last_clean":"2024-04-24T08:21:11.217055+0000","last_became_active":"2024-04-24T08:18:47.290514+0000","last_became_peered":"2024-04-24T08:18:47.290514+0000","last_unstale":"2024-04-24T08:21:11.217055+0000","last_undegraded":"2024-04-24T08:21:11.217055+0000","last_fullsized":"2024-04-24T08:21:11.217055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:39:12.527477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089092+0000","last_change":"2024-04-24T08:18:46.451815+0000","last_active":"2024-04-24T08:21:11.089092+0000","last_peered":"2024-04-24T08:21:11.089092+0000","last_clean":"2024-04-24T08:21:11.089092+0000","last_became_active":"2024-04-24T08:18:46.451478+0000","last_became_peered":"2024-04-24T08:18:46.451478+0000","last_unstale":"2024-04-24T08:21:11.089092+0000","last_undegraded":"2024-04-24T08:21:11.089092+0000","last_fullsized":"2024-04-24T08:21:11.089092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:54:22.907143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.218992+0000","last_change":"2024-04-24T08:18:47.295181+0000","last_active":"2024-04-24T08:21:11.218992+0000","last_peered":"2024-04-24T08:21:11.218992+0000","last_clean":"2024-04-24T08:21:11.218992+0000","last_became_active":"2024-04-24T08:18:47.294842+0000","last_became_peered":"2024-04-24T08:18:47.294842+0000","last_unstale":"2024-04-24T08:21:11.218992+0000","last_undegraded":"2024-04-24T08:21:11.218992+0000","last_fullsized":"2024-04-24T08:21:11.218992+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:17:37.058025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.537280+0000","last_change":"2024-04-24T08:18:46.437967+0000","last_active":"2024-04-24T08:21:11.537280+0000","last_peered":"2024-04-24T08:21:11.537280+0000","last_clean":"2024-04-24T08:21:11.537280+0000","last_became_active":"2024-04-24T08:18:46.437617+0000","last_became_peered":"2024-04-24T08:18:46.437617+0000","last_unstale":"2024-04-24T08:21:11.537280+0000","last_undegraded":"2024-04-24T08:21:11.537280+0000","last_fullsized":"2024-04-24T08:21:11.537280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:07.303827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210446+0000","last_change":"2024-04-24T08:18:57.726323+0000","last_active":"2024-04-24T08:21:11.210446+0000","last_peered":"2024-04-24T08:21:11.210446+0000","last_clean":"2024-04-24T08:21:11.210446+0000","last_became_active":"2024-04-24T08:18:57.725924+0000","last_became_peered":"2024-04-24T08:18:57.725924+0000","last_unstale":"2024-04-24T08:21:11.210446+0000","last_undegraded":"2024-04-24T08:21:11.210446+0000","last_fullsized":"2024-04-24T08:21:11.210446+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:15:33.756804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.207766+0000","last_change":"2024-04-24T08:18:57.591169+0000","last_active":"2024-04-24T08:21:11.207766+0000","last_peered":"2024-04-24T08:21:11.207766+0000","last_clean":"2024-04-24T08:21:11.207766+0000","last_became_active":"2024-04-24T08:18:57.590300+0000","last_became_peered":"2024-04-24T08:18:57.590300+0000","last_unstale":"2024-04-24T08:21:11.207766+0000","last_undegraded":"2024-04-24T08:21:11.207766+0000","last_fullsized":"2024-04-24T08:21:11.207766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:13.632780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222625+0000","last_change":"2024-04-24T08:18:47.316791+0000","last_active":"2024-04-24T08:21:11.222625+0000","last_peered":"2024-04-24T08:21:11.222625+0000","last_clean":"2024-04-24T08:21:11.222625+0000","last_became_active":"2024-04-24T08:18:47.316486+0000","last_became_peered":"2024-04-24T08:18:47.316486+0000","last_unstale":"2024-04-24T08:21:11.222625+0000","last_undegraded":"2024-04-24T08:21:11.222625+0000","last_fullsized":"2024-04-24T08:21:11.222625+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:36:37.190732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217170+0000","last_change":"2024-04-24T08:18:46.432831+0000","last_active":"2024-04-24T08:21:11.217170+0000","last_peered":"2024-04-24T08:21:11.217170+0000","last_clean":"2024-04-24T08:21:11.217170+0000","last_became_active":"2024-04-24T08:18:46.432483+0000","last_became_peered":"2024-04-24T08:18:46.432483+0000","last_unstale":"2024-04-24T08:21:11.217170+0000","last_undegraded":"2024-04-24T08:21:11.217170+0000","last_fullsized":"2024-04-24T08:21:11.217170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:43:24.802099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.222393+0000","last_change":"2024-04-24T08:18:47.309408+0000","last_active":"2024-04-24T08:21:11.222393+0000","last_peered":"2024-04-24T08:21:11.222393+0000","last_clean":"2024-04-24T08:21:11.222393+0000","last_became_active":"2024-04-24T08:18:47.309083+0000","last_became_peered":"2024-04-24T08:18:47.309083+0000","last_unstale":"2024-04-24T08:21:11.222393+0000","last_undegraded":"2024-04-24T08:21:11.222393+0000","last_fullsized":"2024-04-24T08:21:11.222393+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:15:44.584297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219335+0000","last_change":"2024-04-24T08:18:47.291060+0000","last_active":"2024-04-24T08:21:11.219335+0000","last_peered":"2024-04-24T08:21:11.219335+0000","last_clean":"2024-04-24T08:21:11.219335+0000","last_became_active":"2024-04-24T08:18:47.290635+0000","last_became_peered":"2024-04-24T08:18:47.290635+0000","last_unstale":"2024-04-24T08:21:11.219335+0000","last_undegraded":"2024-04-24T08:21:11.219335+0000","last_fullsized":"2024-04-24T08:21:11.219335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:20.372853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":167,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.415606+0000","last_change":"2024-04-24T08:18:46.449850+0000","last_active":"2024-04-24T08:21:11.415606+0000","last_peered":"2024-04-24T08:21:11.415606+0000","last_clean":"2024-04-24T08:21:11.415606+0000","last_became_active":"2024-04-24T08:18:46.448486+0000","last_became_peered":"2024-04-24T08:18:46.448486+0000","last_unstale":"2024-04-24T08:21:11.415606+0000","last_undegraded":"2024-04-24T08:21:11.415606+0000","last_fullsized":"2024-04-24T08:21:11.415606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:54.611867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211035+0000","last_change":"2024-04-24T08:18:47.328051+0000","last_active":"2024-04-24T08:21:11.211035+0000","last_peered":"2024-04-24T08:21:11.211035+0000","last_clean":"2024-04-24T08:21:11.211035+0000","last_became_active":"2024-04-24T08:18:47.327474+0000","last_became_peered":"2024-04-24T08:18:47.327474+0000","last_unstale":"2024-04-24T08:21:11.211035+0000","last_undegraded":"2024-04-24T08:21:11.211035+0000","last_fullsized":"2024-04-24T08:21:11.211035+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:53:07.577741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.088963+0000","last_change":"2024-04-24T08:18:46.444375+0000","last_active":"2024-04-24T08:21:11.088963+0000","last_peered":"2024-04-24T08:21:11.088963+0000","last_clean":"2024-04-24T08:21:11.088963+0000","last_became_active":"2024-04-24T08:18:46.443963+0000","last_became_peered":"2024-04-24T08:18:46.443963+0000","last_unstale":"2024-04-24T08:21:11.088963+0000","last_undegraded":"2024-04-24T08:21:11.088963+0000","last_fullsized":"2024-04-24T08:21:11.088963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:27:22.378007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216787+0000","last_change":"2024-04-24T08:17:29.632219+0000","last_active":"2024-04-24T08:21:11.216787+0000","last_peered":"2024-04-24T08:21:11.216787+0000","last_clean":"2024-04-24T08:21:11.216787+0000","last_became_active":"2024-04-24T08:17:29.631841+0000","last_became_peered":"2024-04-24T08:17:29.631841+0000","last_unstale":"2024-04-24T08:21:11.216787+0000","last_undegraded":"2024-04-24T08:21:11.216787+0000","last_fullsized":"2024-04-24T08:21:11.216787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:41.228801+0000","last_clean_scrub_stamp":"2024-04-24T08:13:41.228801+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:02:55.415959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089104+0000","last_change":"2024-04-24T08:18:47.301876+0000","last_active":"2024-04-24T08:21:11.089104+0000","last_peered":"2024-04-24T08:21:11.089104+0000","last_clean":"2024-04-24T08:21:11.089104+0000","last_became_active":"2024-04-24T08:18:47.301545+0000","last_became_peered":"2024-04-24T08:18:47.301545+0000","last_unstale":"2024-04-24T08:21:11.089104+0000","last_undegraded":"2024-04-24T08:21:11.089104+0000","last_fullsized":"2024-04-24T08:21:11.089104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:34.650014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.216699+0000","last_change":"2024-04-24T08:18:46.437915+0000","last_active":"2024-04-24T08:21:11.216699+0000","last_peered":"2024-04-24T08:21:11.216699+0000","last_clean":"2024-04-24T08:21:11.216699+0000","last_became_active":"2024-04-24T08:18:46.437567+0000","last_became_peered":"2024-04-24T08:18:46.437567+0000","last_unstale":"2024-04-24T08:21:11.216699+0000","last_undegraded":"2024-04-24T08:21:11.216699+0000","last_fullsized":"2024-04-24T08:21:11.216699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:36.801535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089647+0000","last_change":"2024-04-24T08:18:46.451834+0000","last_active":"2024-04-24T08:21:11.089647+0000","last_peered":"2024-04-24T08:21:11.089647+0000","last_clean":"2024-04-24T08:21:11.089647+0000","last_became_active":"2024-04-24T08:18:46.451505+0000","last_became_peered":"2024-04-24T08:18:46.451505+0000","last_unstale":"2024-04-24T08:21:11.089647+0000","last_undegraded":"2024-04-24T08:21:11.089647+0000","last_fullsized":"2024-04-24T08:21:11.089647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:26.468176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089798+0000","last_change":"2024-04-24T08:18:47.310490+0000","last_active":"2024-04-24T08:21:11.089798+0000","last_peered":"2024-04-24T08:21:11.089798+0000","last_clean":"2024-04-24T08:21:11.089798+0000","last_became_active":"2024-04-24T08:18:47.309615+0000","last_became_peered":"2024-04-24T08:18:47.309615+0000","last_unstale":"2024-04-24T08:21:11.089798+0000","last_undegraded":"2024-04-24T08:21:11.089798+0000","last_fullsized":"2024-04-24T08:21:11.089798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:22:55.063485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'46","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.354674+0000","last_change":"2024-04-24T08:18:57.602098+0000","last_active":"2024-04-24T08:21:11.354674+0000","last_peered":"2024-04-24T08:21:11.354674+0000","last_clean":"2024-04-24T08:21:11.354674+0000","last_became_active":"2024-04-24T08:18:57.595899+0000","last_became_peered":"2024-04-24T08:18:57.595899+0000","last_unstale":"2024-04-24T08:21:11.354674+0000","last_undegraded":"2024-04-24T08:21:11.354674+0000","last_fullsized":"2024-04-24T08:21:11.354674+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:31.951803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211883+0000","last_change":"2024-04-24T08:18:47.319022+0000","last_active":"2024-04-24T08:21:11.211883+0000","last_peered":"2024-04-24T08:21:11.211883+0000","last_clean":"2024-04-24T08:21:11.211883+0000","last_became_active":"2024-04-24T08:18:47.318167+0000","last_became_peered":"2024-04-24T08:18:47.318167+0000","last_unstale":"2024-04-24T08:21:11.211883+0000","last_undegraded":"2024-04-24T08:21:11.211883+0000","last_fullsized":"2024-04-24T08:21:11.211883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:18.980856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089397+0000","last_change":"2024-04-24T08:18:46.433018+0000","last_active":"2024-04-24T08:21:11.089397+0000","last_peered":"2024-04-24T08:21:11.089397+0000","last_clean":"2024-04-24T08:21:11.089397+0000","last_became_active":"2024-04-24T08:18:46.432681+0000","last_became_peered":"2024-04-24T08:18:46.432681+0000","last_unstale":"2024-04-24T08:21:11.089397+0000","last_undegraded":"2024-04-24T08:21:11.089397+0000","last_fullsized":"2024-04-24T08:21:11.089397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:53.691684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.416092+0000","last_change":"2024-04-24T08:18:47.326055+0000","last_active":"2024-04-24T08:21:11.416092+0000","last_peered":"2024-04-24T08:21:11.416092+0000","last_clean":"2024-04-24T08:21:11.416092+0000","last_became_active":"2024-04-24T08:18:47.324956+0000","last_became_peered":"2024-04-24T08:18:47.324956+0000","last_unstale":"2024-04-24T08:21:11.416092+0000","last_undegraded":"2024-04-24T08:21:11.416092+0000","last_fullsized":"2024-04-24T08:21:11.416092+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:44:37.431264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.410298+0000","last_change":"2024-04-24T08:18:46.450710+0000","last_active":"2024-04-24T08:21:11.410298+0000","last_peered":"2024-04-24T08:21:11.410298+0000","last_clean":"2024-04-24T08:21:11.410298+0000","last_became_active":"2024-04-24T08:18:46.450391+0000","last_became_peered":"2024-04-24T08:18:46.450391+0000","last_unstale":"2024-04-24T08:21:11.410298+0000","last_undegraded":"2024-04-24T08:21:11.410298+0000","last_fullsized":"2024-04-24T08:21:11.410298+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:07:19.905524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089025+0000","last_change":"2024-04-24T08:18:47.292114+0000","last_active":"2024-04-24T08:21:11.089025+0000","last_peered":"2024-04-24T08:21:11.089025+0000","last_clean":"2024-04-24T08:21:11.089025+0000","last_became_active":"2024-04-24T08:18:47.291787+0000","last_became_peered":"2024-04-24T08:18:47.291787+0000","last_unstale":"2024-04-24T08:21:11.089025+0000","last_undegraded":"2024-04-24T08:21:11.089025+0000","last_fullsized":"2024-04-24T08:21:11.089025+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:37.020368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210671+0000","last_change":"2024-04-24T08:18:46.446027+0000","last_active":"2024-04-24T08:21:11.210671+0000","last_peered":"2024-04-24T08:21:11.210671+0000","last_clean":"2024-04-24T08:21:11.210671+0000","last_became_active":"2024-04-24T08:18:46.445634+0000","last_became_peered":"2024-04-24T08:18:46.445634+0000","last_unstale":"2024-04-24T08:21:11.210671+0000","last_undegraded":"2024-04-24T08:21:11.210671+0000","last_fullsized":"2024-04-24T08:21:11.210671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:46.208560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210162+0000","last_change":"2024-04-24T08:18:47.307800+0000","last_active":"2024-04-24T08:21:11.210162+0000","last_peered":"2024-04-24T08:21:11.210162+0000","last_clean":"2024-04-24T08:21:11.210162+0000","last_became_active":"2024-04-24T08:18:47.307501+0000","last_became_peered":"2024-04-24T08:18:47.307501+0000","last_unstale":"2024-04-24T08:21:11.210162+0000","last_undegraded":"2024-04-24T08:21:11.210162+0000","last_fullsized":"2024-04-24T08:21:11.210162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:04:05.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408128+0000","last_change":"2024-04-24T08:18:46.442811+0000","last_active":"2024-04-24T08:21:11.408128+0000","last_peered":"2024-04-24T08:21:11.408128+0000","last_clean":"2024-04-24T08:21:11.408128+0000","last_became_active":"2024-04-24T08:18:46.442479+0000","last_became_peered":"2024-04-24T08:18:46.442479+0000","last_unstale":"2024-04-24T08:21:11.408128+0000","last_undegraded":"2024-04-24T08:21:11.408128+0000","last_fullsized":"2024-04-24T08:21:11.408128+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:24:07.236529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210897+0000","last_change":"2024-04-24T08:18:47.325046+0000","last_active":"2024-04-24T08:21:11.210897+0000","last_peered":"2024-04-24T08:21:11.210897+0000","last_clean":"2024-04-24T08:21:11.210897+0000","last_became_active":"2024-04-24T08:18:47.324720+0000","last_became_peered":"2024-04-24T08:18:47.324720+0000","last_unstale":"2024-04-24T08:21:11.210897+0000","last_undegraded":"2024-04-24T08:21:11.210897+0000","last_fullsized":"2024-04-24T08:21:11.210897+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:03:11.240388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089232+0000","last_change":"2024-04-24T08:18:46.440572+0000","last_active":"2024-04-24T08:21:11.089232+0000","last_peered":"2024-04-24T08:21:11.089232+0000","last_clean":"2024-04-24T08:21:11.089232+0000","last_became_active":"2024-04-24T08:18:46.440216+0000","last_became_peered":"2024-04-24T08:18:46.440216+0000","last_unstale":"2024-04-24T08:21:11.089232+0000","last_undegraded":"2024-04-24T08:21:11.089232+0000","last_fullsized":"2024-04-24T08:21:11.089232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:51.352874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217093+0000","last_change":"2024-04-24T08:18:47.293465+0000","last_active":"2024-04-24T08:21:11.217093+0000","last_peered":"2024-04-24T08:21:11.217093+0000","last_clean":"2024-04-24T08:21:11.217093+0000","last_became_active":"2024-04-24T08:18:47.293062+0000","last_became_peered":"2024-04-24T08:18:47.293062+0000","last_unstale":"2024-04-24T08:21:11.217093+0000","last_undegraded":"2024-04-24T08:21:11.217093+0000","last_fullsized":"2024-04-24T08:21:11.217093+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:36.029511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.408270+0000","last_change":"2024-04-24T08:18:46.451007+0000","last_active":"2024-04-24T08:21:11.408270+0000","last_peered":"2024-04-24T08:21:11.408270+0000","last_clean":"2024-04-24T08:21:11.408270+0000","last_became_active":"2024-04-24T08:18:46.450676+0000","last_became_peered":"2024-04-24T08:18:46.450676+0000","last_unstale":"2024-04-24T08:21:11.408270+0000","last_undegraded":"2024-04-24T08:21:11.408270+0000","last_fullsized":"2024-04-24T08:21:11.408270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:40:00.329311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217356+0000","last_change":"2024-04-24T08:18:47.293859+0000","last_active":"2024-04-24T08:21:11.217356+0000","last_peered":"2024-04-24T08:21:11.217356+0000","last_clean":"2024-04-24T08:21:11.217356+0000","last_became_active":"2024-04-24T08:18:47.293509+0000","last_became_peered":"2024-04-24T08:18:47.293509+0000","last_unstale":"2024-04-24T08:21:11.217356+0000","last_undegraded":"2024-04-24T08:21:11.217356+0000","last_fullsized":"2024-04-24T08:21:11.217356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:41.224037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.356078+0000","last_change":"2024-04-24T08:18:46.436661+0000","last_active":"2024-04-24T08:21:11.356078+0000","last_peered":"2024-04-24T08:21:11.356078+0000","last_clean":"2024-04-24T08:21:11.356078+0000","last_became_active":"2024-04-24T08:18:46.436353+0000","last_became_peered":"2024-04-24T08:18:46.436353+0000","last_unstale":"2024-04-24T08:21:11.356078+0000","last_undegraded":"2024-04-24T08:21:11.356078+0000","last_fullsized":"2024-04-24T08:21:11.356078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:57:25.559706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217135+0000","last_change":"2024-04-24T08:18:57.597302+0000","last_active":"2024-04-24T08:21:11.217135+0000","last_peered":"2024-04-24T08:21:11.217135+0000","last_clean":"2024-04-24T08:21:11.217135+0000","last_became_active":"2024-04-24T08:18:57.591420+0000","last_became_peered":"2024-04-24T08:18:57.591420+0000","last_unstale":"2024-04-24T08:21:11.217135+0000","last_undegraded":"2024-04-24T08:21:11.217135+0000","last_fullsized":"2024-04-24T08:21:11.217135+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:03:34.286436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089289+0000","last_change":"2024-04-24T08:18:46.432499+0000","last_active":"2024-04-24T08:21:11.089289+0000","last_peered":"2024-04-24T08:21:11.089289+0000","last_clean":"2024-04-24T08:21:11.089289+0000","last_became_active":"2024-04-24T08:18:46.432085+0000","last_became_peered":"2024-04-24T08:18:46.432085+0000","last_unstale":"2024-04-24T08:21:11.089289+0000","last_undegraded":"2024-04-24T08:21:11.089289+0000","last_fullsized":"2024-04-24T08:21:11.089289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:45.405481+0000","last_clean_scrub_stamp":"2024-04-24T08:18:45.405481+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:03.794091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.217293+0000","last_change":"2024-04-24T08:18:47.291193+0000","last_active":"2024-04-24T08:21:11.217293+0000","last_peered":"2024-04-24T08:21:11.217293+0000","last_clean":"2024-04-24T08:21:11.217293+0000","last_became_active":"2024-04-24T08:18:47.289903+0000","last_became_peered":"2024-04-24T08:18:47.289903+0000","last_unstale":"2024-04-24T08:21:11.217293+0000","last_undegraded":"2024-04-24T08:21:11.217293+0000","last_fullsized":"2024-04-24T08:21:11.217293+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:35:18.929700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210090+0000","last_change":"2024-04-24T08:18:47.292628+0000","last_active":"2024-04-24T08:21:11.210090+0000","last_peered":"2024-04-24T08:21:11.210090+0000","last_clean":"2024-04-24T08:21:11.210090+0000","last_became_active":"2024-04-24T08:18:47.292060+0000","last_became_peered":"2024-04-24T08:18:47.292060+0000","last_unstale":"2024-04-24T08:21:11.210090+0000","last_undegraded":"2024-04-24T08:21:11.210090+0000","last_fullsized":"2024-04-24T08:21:11.210090+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:11:53.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407920+0000","last_change":"2024-04-24T08:18:47.297121+0000","last_active":"2024-04-24T08:21:11.407920+0000","last_peered":"2024-04-24T08:21:11.407920+0000","last_clean":"2024-04-24T08:21:11.407920+0000","last_became_active":"2024-04-24T08:18:47.296787+0000","last_became_peered":"2024-04-24T08:18:47.296787+0000","last_unstale":"2024-04-24T08:21:11.407920+0000","last_undegraded":"2024-04-24T08:21:11.407920+0000","last_fullsized":"2024-04-24T08:21:11.407920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:44:10.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407884+0000","last_change":"2024-04-24T08:18:47.320669+0000","last_active":"2024-04-24T08:21:11.407884+0000","last_peered":"2024-04-24T08:21:11.407884+0000","last_clean":"2024-04-24T08:21:11.407884+0000","last_became_active":"2024-04-24T08:18:47.320287+0000","last_became_peered":"2024-04-24T08:18:47.320287+0000","last_unstale":"2024-04-24T08:21:11.407884+0000","last_undegraded":"2024-04-24T08:21:11.407884+0000","last_fullsized":"2024-04-24T08:21:11.407884+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:39:03.275189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.417067+0000","last_change":"2024-04-24T08:18:57.585568+0000","last_active":"2024-04-24T08:21:11.417067+0000","last_peered":"2024-04-24T08:21:11.417067+0000","last_clean":"2024-04-24T08:21:11.417067+0000","last_became_active":"2024-04-24T08:18:57.584799+0000","last_became_peered":"2024-04-24T08:18:57.584799+0000","last_unstale":"2024-04-24T08:21:11.417067+0000","last_undegraded":"2024-04-24T08:21:11.417067+0000","last_fullsized":"2024-04-24T08:21:11.417067+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:00:47.202344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.089676+0000","last_change":"2024-04-24T08:18:47.316068+0000","last_active":"2024-04-24T08:21:11.089676+0000","last_peered":"2024-04-24T08:21:11.089676+0000","last_clean":"2024-04-24T08:21:11.089676+0000","last_became_active":"2024-04-24T08:18:47.315647+0000","last_became_peered":"2024-04-24T08:18:47.315647+0000","last_unstale":"2024-04-24T08:21:11.089676+0000","last_undegraded":"2024-04-24T08:21:11.089676+0000","last_fullsized":"2024-04-24T08:21:11.089676+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:33.510961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210316+0000","last_change":"2024-04-24T08:18:47.294966+0000","last_active":"2024-04-24T08:21:11.210316+0000","last_peered":"2024-04-24T08:21:11.210316+0000","last_clean":"2024-04-24T08:21:11.210316+0000","last_became_active":"2024-04-24T08:18:47.294672+0000","last_became_peered":"2024-04-24T08:18:47.294672+0000","last_unstale":"2024-04-24T08:21:11.210316+0000","last_undegraded":"2024-04-24T08:21:11.210316+0000","last_fullsized":"2024-04-24T08:21:11.210316+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:37.359114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.407962+0000","last_change":"2024-04-24T08:18:47.294548+0000","last_active":"2024-04-24T08:21:11.407962+0000","last_peered":"2024-04-24T08:21:11.407962+0000","last_clean":"2024-04-24T08:21:11.407962+0000","last_became_active":"2024-04-24T08:18:47.293589+0000","last_became_peered":"2024-04-24T08:18:47.293589+0000","last_unstale":"2024-04-24T08:21:11.407962+0000","last_undegraded":"2024-04-24T08:21:11.407962+0000","last_fullsized":"2024-04-24T08:21:11.407962+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:34:56.208453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.211773+0000","last_change":"2024-04-24T08:18:47.313156+0000","last_active":"2024-04-24T08:21:11.211773+0000","last_peered":"2024-04-24T08:21:11.211773+0000","last_clean":"2024-04-24T08:21:11.211773+0000","last_became_active":"2024-04-24T08:18:47.312829+0000","last_became_peered":"2024-04-24T08:18:47.312829+0000","last_unstale":"2024-04-24T08:21:11.211773+0000","last_undegraded":"2024-04-24T08:21:11.211773+0000","last_fullsized":"2024-04-24T08:21:11.211773+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:28:36.638514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.219278+0000","last_change":"2024-04-24T08:18:47.320022+0000","last_active":"2024-04-24T08:21:11.219278+0000","last_peered":"2024-04-24T08:21:11.219278+0000","last_clean":"2024-04-24T08:21:11.219278+0000","last_became_active":"2024-04-24T08:18:47.319707+0000","last_became_peered":"2024-04-24T08:18:47.319707+0000","last_unstale":"2024-04-24T08:21:11.219278+0000","last_undegraded":"2024-04-24T08:21:11.219278+0000","last_fullsized":"2024-04-24T08:21:11.219278+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:51.330874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210674+0000","last_change":"2024-04-24T08:18:47.323187+0000","last_active":"2024-04-24T08:21:11.210674+0000","last_peered":"2024-04-24T08:21:11.210674+0000","last_clean":"2024-04-24T08:21:11.210674+0000","last_became_active":"2024-04-24T08:18:47.320258+0000","last_became_peered":"2024-04-24T08:18:47.320258+0000","last_unstale":"2024-04-24T08:21:11.210674+0000","last_undegraded":"2024-04-24T08:21:11.210674+0000","last_fullsized":"2024-04-24T08:21:11.210674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:42:32.770950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.409986+0000","last_change":"2024-04-24T08:18:47.321926+0000","last_active":"2024-04-24T08:21:11.409986+0000","last_peered":"2024-04-24T08:21:11.409986+0000","last_clean":"2024-04-24T08:21:11.409986+0000","last_became_active":"2024-04-24T08:18:47.321584+0000","last_became_peered":"2024-04-24T08:18:47.321584+0000","last_unstale":"2024-04-24T08:21:11.409986+0000","last_undegraded":"2024-04-24T08:21:11.409986+0000","last_fullsized":"2024-04-24T08:21:11.409986+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:24.357157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.093027+0000","last_change":"2024-04-24T08:18:47.313310+0000","last_active":"2024-04-24T08:21:11.093027+0000","last_peered":"2024-04-24T08:21:11.093027+0000","last_clean":"2024-04-24T08:21:11.093027+0000","last_became_active":"2024-04-24T08:18:47.312968+0000","last_became_peered":"2024-04-24T08:18:47.312968+0000","last_unstale":"2024-04-24T08:21:11.093027+0000","last_undegraded":"2024-04-24T08:21:11.093027+0000","last_fullsized":"2024-04-24T08:21:11.093027+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:08.967577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:21:11.210366+0000","last_change":"2024-04-24T08:18:57.591543+0000","last_active":"2024-04-24T08:21:11.210366+0000","last_peered":"2024-04-24T08:21:11.210366+0000","last_clean":"2024-04-24T08:21:11.210366+0000","last_became_active":"2024-04-24T08:18:57.591043+0000","last_became_peered":"2024-04-24T08:18:57.591043+0000","last_unstale":"2024-04-24T08:21:11.210366+0000","last_undegraded":"2024-04-24T08:21:11.210366+0000","last_fullsized":"2024-04-24T08:21:11.210366+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:46.415488+0000","last_clean_scrub_stamp":"2024-04-24T08:18:46.415488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:52:38.669469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597142,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":966,"num_read_kb":592,"num_write":291,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":21,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1791426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285609,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28380,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716772,"statfs":{"total":95995035648,"available":95965974528,"internally_reserved":0,"allocated":1392640,"data_stored":871799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547246,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1765376,"data_stored":1197725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710773,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36720,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708432,"statfs":{"total":95995035648,"available":95957434368,"internally_reserved":0,"allocated":1413120,"data_stored":872152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972411,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":478739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1609728,"data_stored":1072293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430277,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":478208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1073152,"data_stored":544702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":479479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":532842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1003218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888145,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":478645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691915,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37112,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708040,"statfs":{"total":95995035648,"available":95957032960,"internally_reserved":0,"allocated":1757184,"data_stored":1202795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":200661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:21:17.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 -- 172.21.15.146:0/3854990456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0102b40 msgr2=0x7f9ab0104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3854990456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0102b40 0x7f9ab0104f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009900 tx=0x7f9aa002a410 comp rx=0 tx=0).stop 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 -- 172.21.15.146:0/3854990456 shutdown_connections 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3854990456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0107da0 0x7f9ab010a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3854990456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab0105470 0x7f9ab0107860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3854990456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0102b40 0x7f9ab0104f30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 -- 172.21.15.146:0/3854990456 >> 172.21.15.146:0/3854990456 conn(0x7f9ab00fc4f0 msgr2=0x7f9ab00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 -- 172.21.15.146:0/3854990456 shutdown_connections 2024-04-24T08:21:17.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.873+0000 7f9ab6000640 1 -- 172.21.15.146:0/3854990456 wait complete. 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 Processor -- start 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 -- start start 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0102b40 0x7f9ab019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0105470 0x7f9ab019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab0107da0 0x7f9ab01a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab010dcd0 con 0x7f9ab0107da0 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab010db50 con 0x7f9ab0105470 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.874+0000 7f9ab6000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab010de50 con 0x7f9ab0102b40 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0105470 0x7f9ab019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0102b40 0x7f9ab019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0102b40 0x7f9ab019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44664/0 (socket says 172.21.15.146:44664) 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0105470 0x7f9ab019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:51188/0 (socket says 172.21.15.146:51188) 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 -- 172.21.15.146:0/3215629558 learned_addr learned my addr 172.21.15.146:0/3215629558 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aafe00640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab0107da0 0x7f9ab01a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 -- 172.21.15.146:0/3215629558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0105470 msgr2=0x7f9ab019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0105470 0x7f9ab019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 -- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab0107da0 msgr2=0x7f9ab01a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab0107da0 0x7f9ab01a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 -- 172.21.15.146:0/3215629558 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab01a2140 con 0x7f9ab0102b40 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaea00640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0105470 0x7f9ab019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.875+0000 7f9aaf400640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0102b40 0x7f9ab019a520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009230 tx=0x7f9aa0031b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.876+0000 7f9aa7e00640 1 -- 172.21.15.146:0/3215629558 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa003f030 con 0x7f9ab0102b40 2024-04-24T08:21:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.876+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab01a23d0 con 0x7f9ab0102b40 2024-04-24T08:21:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.876+0000 7f9aa7e00640 1 -- 172.21.15.146:0/3215629558 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa0003ad0 con 0x7f9ab0102b40 2024-04-24T08:21:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.876+0000 7f9aa7e00640 1 -- 172.21.15.146:0/3215629558 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0035070 con 0x7f9ab0102b40 2024-04-24T08:21:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.876+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab01a2910 con 0x7f9ab0102b40 2024-04-24T08:21:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.877+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab0109890 con 0x7f9ab0102b40 2024-04-24T08:21:17.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.879+0000 7f9aa7e00640 1 -- 172.21.15.146:0/3215629558 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f9aa0042030 con 0x7f9ab0102b40 2024-04-24T08:21:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.880+0000 7f9aa7e00640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9a940a3e40 0x7f9a940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.880+0000 7f9aa7e00640 1 -- 172.21.15.146:0/3215629558 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9aa00f0ba0 con 0x7f9ab0102b40 2024-04-24T08:21:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.881+0000 7f9aaea00640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9a940a3e40 0x7f9a940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.881+0000 7f9aaea00640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9a940a3e40 0x7f9a940a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ab019ba30 tx=0x7f9a9c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:17.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:17.884+0000 7f9aa7e00640 1 -- 172.21.15.146:0/3215629558 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9aa00bca20 con 0x7f9ab0102b40 2024-04-24T08:21:17.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='client.25340 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:17.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:17 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='client.25340 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:17.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:17.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:17 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='client.25340 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:18.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:18.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:17 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:18.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.089+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ab002d870 con 0x7f9ab0102b40 2024-04-24T08:21:18.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.091+0000 7f9aa7e00640 1 -- 172.21.15.146:0/3215629558 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23068 (secure 0 0 0) 0x7f9aa00c0d20 con 0x7f9ab0102b40 2024-04-24T08:21:18.093 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:18.093 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":91,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:21:11.077519+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:0/4151754658":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2355630587":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6800/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/393585939":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/4023745007":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.093+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9a940a3e40 msgr2=0x7f9a940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.093+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9a940a3e40 0x7f9a940a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ab019ba30 tx=0x7f9a9c005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.093+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0102b40 msgr2=0x7f9ab019a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.093+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0102b40 0x7f9ab019a520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009230 tx=0x7f9aa0031b60 comp rx=0 tx=0).stop 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 shutdown_connections 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9a940a3e40 0x7f9a940a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ab0107da0 0x7f9ab01a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ab0105470 0x7f9ab019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 --2- 172.21.15.146:0/3215629558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ab0102b40 0x7f9ab019a520 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 >> 172.21.15.146:0/3215629558 conn(0x7f9ab00fc4f0 msgr2=0x7f9ab01022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 shutdown_connections 2024-04-24T08:21:18.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.094+0000 7f9ab6000640 1 -- 172.21.15.146:0/3215629558 wait complete. 2024-04-24T08:21:18.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:18.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 -- 172.21.15.146:0/1225931449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0074a60 msgr2=0x7f13d0074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 --2- 172.21.15.146:0/1225931449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0074a60 0x7f13d0074ee0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009900 tx=0x7f13bc02a410 comp rx=0 tx=0).stop 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 -- 172.21.15.146:0/1225931449 shutdown_connections 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 --2- 172.21.15.146:0/1225931449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0074a60 0x7f13d0074ee0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 --2- 172.21.15.146:0/1225931449 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d0109df0 0x7f13d010c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 --2- 172.21.15.146:0/1225931449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13d0076040 0x7f13d0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 -- 172.21.15.146:0/1225931449 >> 172.21.15.146:0/1225931449 conn(0x7f13d00fd4a0 msgr2=0x7f13d00ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 -- 172.21.15.146:0/1225931449 shutdown_connections 2024-04-24T08:21:18.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.303+0000 7f13d4c00640 1 -- 172.21.15.146:0/1225931449 wait complete. 2024-04-24T08:21:18.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.304+0000 7f13d4c00640 1 Processor -- start 2024-04-24T08:21:18.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13d4c00640 1 -- start start 2024-04-24T08:21:18.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13d4c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d0074a60 0x7f13d019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13d4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0076040 0x7f13d019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13d4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13d0109df0 0x7f13d01a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13d4c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f13d010fdd0 con 0x7f13d0109df0 2024-04-24T08:21:18.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13d4c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f13d010fc50 con 0x7f13d0074a60 2024-04-24T08:21:18.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13d4c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13d010ff50 con 0x7f13d0076040 2024-04-24T08:21:18.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13cea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0076040 0x7f13d019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13cf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d0074a60 0x7f13d019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13d0109df0 0x7f13d01a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13cea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0076040 0x7f13d019a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44694/0 (socket says 172.21.15.146:44694) 2024-04-24T08:21:18.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.305+0000 7f13cea00640 1 -- 172.21.15.146:0/3720821082 learned_addr learned my addr 172.21.15.146:0/3720821082 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:18.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13cf400640 1 -- 172.21.15.146:0/3720821082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0076040 msgr2=0x7f13d019a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13cf400640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0076040 0x7f13d019a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13cf400640 1 -- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13d0109df0 msgr2=0x7f13d01a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13cf400640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13d0109df0 0x7f13d01a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13cf400640 1 -- 172.21.15.146:0/3720821082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d01a2040 con 0x7f13d0074a60 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13cf400640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d0074a60 0x7f13d019a420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f13c000e9f0 tx=0x7f13c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13c7e00640 1 -- 172.21.15.146:0/3720821082 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c000ccc0 con 0x7f13d0074a60 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13c7e00640 1 -- 172.21.15.146:0/3720821082 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c0010910 con 0x7f13d0074a60 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13c7e00640 1 -- 172.21.15.146:0/3720821082 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0018960 con 0x7f13d0074a60 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d01a2330 con 0x7f13d0074a60 2024-04-24T08:21:18.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.306+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d01a28d0 con 0x7f13d0074a60 2024-04-24T08:21:18.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.308+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d0108b20 con 0x7f13d0074a60 2024-04-24T08:21:18.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.310+0000 7f13c7e00640 1 -- 172.21.15.146:0/3720821082 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f13c0010ac0 con 0x7f13d0074a60 2024-04-24T08:21:18.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.311+0000 7f13c7e00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f13b40a3ec0 0x7f13b40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.311+0000 7f13c7e00640 1 -- 172.21.15.146:0/3720821082 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f13c00d0e10 con 0x7f13d0074a60 2024-04-24T08:21:18.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.312+0000 7f13cea00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f13b40a3ec0 0x7f13b40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.312+0000 7f13cea00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f13b40a3ec0 0x7f13b40a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f13d019b930 tx=0x7f13bc0315b0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:18.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.315+0000 7f13c7e00640 1 -- 172.21.15.146:0/3720821082 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f13c00d4030 con 0x7f13d0074a60 2024-04-24T08:21:18.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.541+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13d002d9e0 con 0x7f13d0074a60 2024-04-24T08:21:18.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.542+0000 7f13c7e00640 1 -- 172.21.15.146:0/3720821082 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f13c009cbc0 con 0x7f13d0074a60 2024-04-24T08:21:18.544 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:21:18.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f13b40a3ec0 msgr2=0x7f13b40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f13b40a3ec0 0x7f13b40a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f13d019b930 tx=0x7f13bc0315b0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d0074a60 msgr2=0x7f13d019a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d0074a60 0x7f13d019a420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f13c000e9f0 tx=0x7f13c000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 shutdown_connections 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f13b40a3ec0 0x7f13b40a6380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d0076040 0x7f13d019a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13d0109df0 0x7f13d01a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 --2- 172.21.15.146:0/3720821082 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13d0074a60 0x7f13d019a420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.544+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 >> 172.21.15.146:0/3720821082 conn(0x7f13d00fd4a0 msgr2=0x7f13d0105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.545+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 shutdown_connections 2024-04-24T08:21:18.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.545+0000 7f13d4c00640 1 -- 172.21.15.146:0/3720821082 wait complete. 2024-04-24T08:21:18.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:18.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 -- 172.21.15.146:0/1675364323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50102050 msgr2=0x7eff50102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 --2- 172.21.15.146:0/1675364323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50102050 0x7eff50102470 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7eff44009860 tx=0x7eff4402a340 comp rx=0 tx=0).stop 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 -- 172.21.15.146:0/1675364323 shutdown_connections 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 --2- 172.21.15.146:0/1675364323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff50103c00 0x7eff5010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 --2- 172.21.15.146:0/1675364323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff50103240 0x7eff501036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 --2- 172.21.15.146:0/1675364323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50102050 0x7eff50102470 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 -- 172.21.15.146:0/1675364323 >> 172.21.15.146:0/1675364323 conn(0x7eff500fd5e0 msgr2=0x7eff500ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 -- 172.21.15.146:0/1675364323 shutdown_connections 2024-04-24T08:21:18.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.745+0000 7eff57600640 1 -- 172.21.15.146:0/1675364323 wait complete. 2024-04-24T08:21:18.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 Processor -- start 2024-04-24T08:21:18.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 -- start start 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff50102050 0x7eff50198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff50103240 0x7eff501988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50103c00 0x7eff5019f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7eff5010dce0 con 0x7eff50103c00 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7eff5010db60 con 0x7eff50102050 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff57600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7eff5010de60 con 0x7eff50103240 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.746+0000 7eff55600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50103c00 0x7eff5019f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff55600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50103c00 0x7eff5019f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39240/0 (socket says 172.21.15.146:39240) 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff55600640 1 -- 172.21.15.146:0/1638143852 learned_addr learned my addr 172.21.15.146:0/1638143852 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff4fe00640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff50103240 0x7eff501988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff54c00640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff50102050 0x7eff50198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff54c00640 1 -- 172.21.15.146:0/1638143852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff50103240 msgr2=0x7eff501988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff54c00640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff50103240 0x7eff501988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff54c00640 1 -- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50103c00 msgr2=0x7eff5019f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff54c00640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50103c00 0x7eff5019f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff54c00640 1 -- 172.21.15.146:0/1638143852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5019ffa0 con 0x7eff50102050 2024-04-24T08:21:18.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff55600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50103c00 0x7eff5019f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:18.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.747+0000 7eff54c00640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff50102050 0x7eff50198380 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7eff44035040 tx=0x7eff44033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:18.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.748+0000 7eff4d600640 1 -- 172.21.15.146:0/1638143852 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44037040 con 0x7eff50102050 2024-04-24T08:21:18.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.748+0000 7eff4d600640 1 -- 172.21.15.146:0/1638143852 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff44032770 con 0x7eff50102050 2024-04-24T08:21:18.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.748+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff501a0230 con 0x7eff50102050 2024-04-24T08:21:18.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.748+0000 7eff4d600640 1 -- 172.21.15.146:0/1638143852 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44005520 con 0x7eff50102050 2024-04-24T08:21:18.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.748+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff501a0770 con 0x7eff50102050 2024-04-24T08:21:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.749+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff50109390 con 0x7eff50102050 2024-04-24T08:21:18.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.751+0000 7eff4d600640 1 -- 172.21.15.146:0/1638143852 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7eff44042020 con 0x7eff50102050 2024-04-24T08:21:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.752+0000 7eff4d600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7eff380a3ec0 0x7eff380a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.752+0000 7eff4d600640 1 -- 172.21.15.146:0/1638143852 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7eff440f0140 con 0x7eff50102050 2024-04-24T08:21:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.752+0000 7eff4fe00640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7eff380a3ec0 0x7eff380a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.753+0000 7eff4fe00640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7eff380a3ec0 0x7eff380a6380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff50199890 tx=0x7eff3c009260 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:18.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.756+0000 7eff4d600640 1 -- 172.21.15.146:0/1638143852 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff44049020 con 0x7eff50102050 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi146 to 176.4M 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Unable to set osd_memory_target on smithi146 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi146:/etc/ceph/ceph.conf 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3215629558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:18 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3720821082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:18.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.964+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7eff50062c00 con 0x7eff50102050 2024-04-24T08:21:18.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.965+0000 7eff4d600640 1 -- 172.21.15.146:0/1638143852 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7eff440bc650 con 0x7eff50102050 2024-04-24T08:21:18.966 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7eff380a3ec0 msgr2=0x7eff380a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7eff380a3ec0 0x7eff380a6380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff50199890 tx=0x7eff3c009260 comp rx=0 tx=0).stop 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff50102050 msgr2=0x7eff50198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff50102050 0x7eff50198380 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7eff44035040 tx=0x7eff44033d90 comp rx=0 tx=0).stop 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 shutdown_connections 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7eff380a3ec0 0x7eff380a6380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff50103240 0x7eff501988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff50103c00 0x7eff5019f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 --2- 172.21.15.146:0/1638143852 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff50102050 0x7eff50198380 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 >> 172.21.15.146:0/1638143852 conn(0x7eff500fd5e0 msgr2=0x7eff50106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 shutdown_connections 2024-04-24T08:21:18.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:18.967+0000 7eff57600640 1 -- 172.21.15.146:0/1638143852 wait complete. 2024-04-24T08:21:18.986 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi146 to 176.4M 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Unable to set osd_memory_target on smithi146 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi146:/etc/ceph/ceph.conf 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:18.987 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:18.988 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3215629558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:18.988 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.988 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.988 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.988 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:18.988 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:18 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3720821082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:18.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:19.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi146 to 176.4M 2024-04-24T08:21:19.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Unable to set osd_memory_target on smithi146 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-24T08:21:19.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi146:/etc/ceph/ceph.conf 2024-04-24T08:21:19.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-24T08:21:19.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-24T08:21:19.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3215629558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:18 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3720821082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 -- 172.21.15.146:0/1594648844 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 msgr2=0x7fdb681154a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 --2- 172.21.15.146:0/1594648844 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 0x7fdb681154a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdb60009cd0 tx=0x7fdb6002a970 comp rx=0 tx=0).stop 2024-04-24T08:21:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 -- 172.21.15.146:0/1594648844 shutdown_connections 2024-04-24T08:21:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 --2- 172.21.15.146:0/1594648844 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 0x7fdb681154a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 --2- 172.21.15.146:0/1594648844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb68074bf0 0x7fdb68112b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 --2- 172.21.15.146:0/1594648844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb680742d0 0x7fdb680746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 -- 172.21.15.146:0/1594648844 >> 172.21.15.146:0/1594648844 conn(0x7fdb68079450 msgr2=0x7fdb68079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 -- 172.21.15.146:0/1594648844 shutdown_connections 2024-04-24T08:21:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.168+0000 7fdb70200640 1 -- 172.21.15.146:0/1594648844 wait complete. 2024-04-24T08:21:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.169+0000 7fdb70200640 1 Processor -- start 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.169+0000 7fdb70200640 1 -- start start 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb70200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb680742d0 0x7fdb6819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb70200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb68074bf0 0x7fdb6819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb70200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 0x7fdb681a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb70200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdb68118a20 con 0x7fdb681130b0 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb70200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdb681188a0 con 0x7fdb680742d0 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb70200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb68118ba0 con 0x7fdb68074bf0 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb6ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb68074bf0 0x7fdb6819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb6ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb68074bf0 0x7fdb6819f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44734/0 (socket says 172.21.15.146:44734) 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb6ce00640 1 -- 172.21.15.146:0/2629545294 learned_addr learned my addr 172.21.15.146:0/2629545294 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb6d800640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb680742d0 0x7fdb6819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.170+0000 7fdb6e200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 0x7fdb681a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6ce00640 1 -- 172.21.15.146:0/2629545294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb680742d0 msgr2=0x7fdb6819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6ce00640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb680742d0 0x7fdb6819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6ce00640 1 -- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 msgr2=0x7fdb681a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6ce00640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 0x7fdb681a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6ce00640 1 -- 172.21.15.146:0/2629545294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb681a3f10 con 0x7fdb68074bf0 2024-04-24T08:21:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6e200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 0x7fdb681a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6d800640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb680742d0 0x7fdb6819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb6ce00640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb68074bf0 0x7fdb6819f460 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400e9f0 tx=0x7fdb5400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb66000640 1 -- 172.21.15.146:0/2629545294 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400cd80 con 0x7fdb68074bf0 2024-04-24T08:21:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb66000640 1 -- 172.21.15.146:0/2629545294 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb54010910 con 0x7fdb68074bf0 2024-04-24T08:21:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb681a4200 con 0x7fdb68074bf0 2024-04-24T08:21:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb66000640 1 -- 172.21.15.146:0/2629545294 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54018990 con 0x7fdb68074bf0 2024-04-24T08:21:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.171+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb681aba90 con 0x7fdb68074bf0 2024-04-24T08:21:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.173+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb68069270 con 0x7fdb68074bf0 2024-04-24T08:21:19.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.175+0000 7fdb66000640 1 -- 172.21.15.146:0/2629545294 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7fdb54010ac0 con 0x7fdb68074bf0 2024-04-24T08:21:19.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.177+0000 7fdb66000640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdb500a3e40 0x7fdb500a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.177+0000 7fdb66000640 1 -- 172.21.15.146:0/2629545294 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdb540d0fe0 con 0x7fdb68074bf0 2024-04-24T08:21:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.177+0000 7fdb6d800640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdb500a3e40 0x7fdb500a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.178+0000 7fdb6d800640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdb500a3e40 0x7fdb500a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c005d00 tx=0x7fdb5c00b410 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.178+0000 7fdb66000640 1 -- 172.21.15.146:0/2629545294 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb5409d4e0 con 0x7fdb68074bf0 2024-04-24T08:21:19.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.391+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdb6802d620 con 0x7fdb68074bf0 2024-04-24T08:21:19.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.391+0000 7fdb66000640 1 -- 172.21.15.146:0/2629545294 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fdb5409cde0 con 0x7fdb68074bf0 2024-04-24T08:21:19.393 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdb500a3e40 msgr2=0x7fdb500a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdb500a3e40 0x7fdb500a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c005d00 tx=0x7fdb5c00b410 comp rx=0 tx=0).stop 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb68074bf0 msgr2=0x7fdb6819f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb68074bf0 0x7fdb6819f460 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400e9f0 tx=0x7fdb5400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 shutdown_connections 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdb500a3e40 0x7fdb500a6300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb68074bf0 0x7fdb6819f460 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb681130b0 0x7fdb681a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 --2- 172.21.15.146:0/2629545294 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb680742d0 0x7fdb6819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 >> 172.21.15.146:0/2629545294 conn(0x7fdb68079450 msgr2=0x7fdb680fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 shutdown_connections 2024-04-24T08:21:19.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.394+0000 7fdb70200640 1 -- 172.21.15.146:0/2629545294 wait complete. 2024-04-24T08:21:19.415 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:19.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.593+0000 7f4c9aa00640 1 -- 172.21.15.146:0/739056587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94102240 msgr2=0x7f4c94102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.593+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/739056587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94102240 0x7f4c94102620 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009ca0 tx=0x7f4c8402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.594+0000 7f4c9aa00640 1 -- 172.21.15.146:0/739056587 shutdown_connections 2024-04-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.594+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/739056587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c94107310 0x7f4c9410eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.594+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/739056587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c94102bf0 0x7f4c94106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.594+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/739056587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94102240 0x7f4c94102620 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.594+0000 7f4c9aa00640 1 -- 172.21.15.146:0/739056587 >> 172.21.15.146:0/739056587 conn(0x7f4c940793c0 msgr2=0x7f4c940797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.594+0000 7f4c9aa00640 1 -- 172.21.15.146:0/739056587 shutdown_connections 2024-04-24T08:21:19.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.594+0000 7f4c9aa00640 1 -- 172.21.15.146:0/739056587 wait complete. 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 Processor -- start 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 -- start start 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c94102240 0x7f4c94074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94107310 0x7f4c94074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c94074d80 0x7f4c941a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c94112420 con 0x7f4c94107310 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941122a0 con 0x7f4c94102240 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.595+0000 7f4c9aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c941125a0 con 0x7f4c94074d80 2024-04-24T08:21:19.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94107310 0x7f4c94074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94107310 0x7f4c94074840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39270/0 (socket says 172.21.15.146:39270) 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c98a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c94074d80 0x7f4c941a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 -- 172.21.15.146:0/3626058735 learned_addr learned my addr 172.21.15.146:0/3626058735 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c93e00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c94102240 0x7f4c94074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 -- 172.21.15.146:0/3626058735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c94074d80 msgr2=0x7f4c941a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c94074d80 0x7f4c941a6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 -- 172.21.15.146:0/3626058735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c94102240 msgr2=0x7f4c94074300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c94102240 0x7f4c94074300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 -- 172.21.15.146:0/3626058735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c941a71f0 con 0x7f4c94107310 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.596+0000 7f4c8b400640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94107310 0x7f4c94074840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c00e9c0 tx=0x7f4c7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.597+0000 7f4c91600640 1 -- 172.21.15.146:0/3626058735 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c00cd80 con 0x7f4c94107310 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.597+0000 7f4c91600640 1 -- 172.21.15.146:0/3626058735 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c7c005500 con 0x7f4c94107310 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.597+0000 7f4c91600640 1 -- 172.21.15.146:0/3626058735 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c010690 con 0x7f4c94107310 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.597+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c941a74e0 con 0x7f4c94107310 2024-04-24T08:21:19.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.597+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c941a7a30 con 0x7f4c94107310 2024-04-24T08:21:19.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.598+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c94069270 con 0x7f4c94107310 2024-04-24T08:21:19.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.600+0000 7f4c91600640 1 -- 172.21.15.146:0/3626058735 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f4c7c0046c0 con 0x7f4c94107310 2024-04-24T08:21:19.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.602+0000 7f4c91600640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c780a3e40 0x7f4c780a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.602+0000 7f4c91600640 1 -- 172.21.15.146:0/3626058735 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4c7c0d0e70 con 0x7f4c94107310 2024-04-24T08:21:19.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.602+0000 7f4c93e00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c780a3e40 0x7f4c780a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.603+0000 7f4c93e00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c780a3e40 0x7f4c780a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c84006010 tx=0x7f4c84009820 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.608+0000 7f4c91600640 1 -- 172.21.15.146:0/3626058735 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c7c09d270 con 0x7f4c94107310 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1638143852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2629545294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:19 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.868+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c940757e0 con 0x7f4c94107310 2024-04-24T08:21:19.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.869+0000 7f4c91600640 1 -- 172.21.15.146:0/3626058735 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11891 (secure 0 0 0) 0x7f4c7c09cb70 con 0x7f4c94107310 2024-04-24T08:21:19.871 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:19.871 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:21:10.741287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24553,24568]},"id":1}]} 2024-04-24T08:21:19.871 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 24 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c780a3e40 msgr2=0x7f4c780a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c780a3e40 0x7f4c780a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4c84006010 tx=0x7f4c84009820 comp rx=0 tx=0).stop 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94107310 msgr2=0x7f4c94074840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94107310 0x7f4c94074840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c00e9c0 tx=0x7f4c7c00ee80 comp rx=0 tx=0).stop 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 shutdown_connections 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c780a3e40 0x7f4c780a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c94074d80 0x7f4c941a6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c94107310 0x7f4c94074840 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 --2- 172.21.15.146:0/3626058735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c94102240 0x7f4c94074300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 >> 172.21.15.146:0/3626058735 conn(0x7f4c940793c0 msgr2=0x7f4c941001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 shutdown_connections 2024-04-24T08:21:19.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:19.872+0000 7f4c9aa00640 1 -- 172.21.15.146:0/3626058735 wait complete. 2024-04-24T08:21:19.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:19.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1638143852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2629545294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:19.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:19 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: Updating smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: Updating smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1638143852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2629545294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:20.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:19 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 -- 172.21.15.146:0/83868584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56641084a0 msgr2=0x7f5664108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 --2- 172.21.15.146:0/83868584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56641084a0 0x7f5664108900 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5654009900 tx=0x7f565402a480 comp rx=0 tx=0).stop 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 -- 172.21.15.146:0/83868584 shutdown_connections 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 --2- 172.21.15.146:0/83868584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5664108e40 0x7f56641152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 --2- 172.21.15.146:0/83868584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56641084a0 0x7f5664108900 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 --2- 172.21.15.146:0/83868584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f566410d3c0 0x7f566410d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 -- 172.21.15.146:0/83868584 >> 172.21.15.146:0/83868584 conn(0x7f5664077b80 msgr2=0x7f5664079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 -- 172.21.15.146:0/83868584 shutdown_connections 2024-04-24T08:21:20.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.074+0000 7f566aa00640 1 -- 172.21.15.146:0/83868584 wait complete. 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.075+0000 7f566aa00640 1 Processor -- start 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566aa00640 1 -- start start 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56641084a0 0x7f56641a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5664108e40 0x7f56641a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566410d3c0 0x7f566419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566aa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5664118830 con 0x7f5664108e40 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566aa00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f56641186b0 con 0x7f566410d3c0 2024-04-24T08:21:20.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f56641189b0 con 0x7f56641084a0 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f5669600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56641084a0 0x7f56641a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f5669600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56641084a0 0x7f56641a4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44764/0 (socket says 172.21.15.146:44764) 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f5669600640 1 -- 172.21.15.146:0/4083932523 learned_addr learned my addr 172.21.15.146:0/4083932523 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f5668c00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5664108e40 0x7f56641a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.076+0000 7f566a000640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566410d3c0 0x7f566419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5669600640 1 -- 172.21.15.146:0/4083932523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566410d3c0 msgr2=0x7f566419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5669600640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566410d3c0 0x7f566419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5669600640 1 -- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5664108e40 msgr2=0x7f56641a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5669600640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5664108e40 0x7f56641a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5669600640 1 -- 172.21.15.146:0/4083932523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566419f250 con 0x7f56641084a0 2024-04-24T08:21:20.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5669600640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56641084a0 0x7f56641a4920 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f565800e9f0 tx=0x7f565800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5662000640 1 -- 172.21.15.146:0/4083932523 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565800cd80 con 0x7f56641084a0 2024-04-24T08:21:20.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566419f540 con 0x7f56641084a0 2024-04-24T08:21:20.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5662000640 1 -- 172.21.15.146:0/4083932523 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5658010910 con 0x7f56641084a0 2024-04-24T08:21:20.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.077+0000 7f5662000640 1 -- 172.21.15.146:0/4083932523 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580188f0 con 0x7f56641084a0 2024-04-24T08:21:20.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.078+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641ab7e0 con 0x7f56641084a0 2024-04-24T08:21:20.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.079+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5664069190 con 0x7f56641084a0 2024-04-24T08:21:20.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.081+0000 7f5662000640 1 -- 172.21.15.146:0/4083932523 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f5658010ac0 con 0x7f56641084a0 2024-04-24T08:21:20.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.082+0000 7f5662000640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f564c0a3e40 0x7f564c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.082+0000 7f5662000640 1 -- 172.21.15.146:0/4083932523 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f56580d1250 con 0x7f56641084a0 2024-04-24T08:21:20.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.082+0000 7f5668c00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f564c0a3e40 0x7f564c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.083+0000 7f5668c00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f564c0a3e40 0x7f564c0a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5654000c00 tx=0x7f5654005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.084+0000 7f5662000640 1 -- 172.21.15.146:0/4083932523 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f565809d000 con 0x7f56641084a0 2024-04-24T08:21:20.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.291+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5664110900 con 0x7f56641084a0 2024-04-24T08:21:20.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.293+0000 7f5662000640 1 -- 172.21.15.146:0/4083932523 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23068 (secure 0 0 0) 0x7f56580a1300 con 0x7f56641084a0 2024-04-24T08:21:20.295 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:20.295 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":91,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:21:11.077519+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:0/4151754658":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2355630587":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6800/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/393585939":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/4023745007":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:20.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f564c0a3e40 msgr2=0x7f564c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f564c0a3e40 0x7f564c0a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5654000c00 tx=0x7f5654005b50 comp rx=0 tx=0).stop 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56641084a0 msgr2=0x7f56641a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56641084a0 0x7f56641a4920 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f565800e9f0 tx=0x7f565800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 shutdown_connections 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f564c0a3e40 0x7f564c0a6300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f566410d3c0 0x7f566419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5664108e40 0x7f56641a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 --2- 172.21.15.146:0/4083932523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f56641084a0 0x7f56641a4920 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 >> 172.21.15.146:0/4083932523 conn(0x7f5664077b80 msgr2=0x7f5664078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 shutdown_connections 2024-04-24T08:21:20.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.296+0000 7f566aa00640 1 -- 172.21.15.146:0/4083932523 wait complete. 2024-04-24T08:21:20.317 DEBUG:tasks.fwd_scrub:Yielding 2024-04-24T08:21:20.318 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T08:21:20.330 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:21:20.330 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:21:20.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-24T08:21:20.330 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-24T08:21:20.330 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:21:20.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:20.331 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:21:20.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.513+0000 7f08f2000640 1 -- 172.21.15.146:0/4251976640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 msgr2=0x7f08ec104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.513+0000 7f08f2000640 1 --2- 172.21.15.146:0/4251976640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec104830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009860 tx=0x7f08dc02a340 comp rx=0 tx=0).stop 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.513+0000 7f08f2000640 1 -- 172.21.15.146:0/4251976640 shutdown_connections 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.513+0000 7f08f2000640 1 --2- 172.21.15.146:0/4251976640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08ec109520 0x7f08ec1110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.513+0000 7f08f2000640 1 --2- 172.21.15.146:0/4251976640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08ec104e00 0x7f08ec108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.513+0000 7f08f2000640 1 --2- 172.21.15.146:0/4251976640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec104830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.513+0000 7f08f2000640 1 -- 172.21.15.146:0/4251976640 >> 172.21.15.146:0/4251976640 conn(0x7f08ec0ffcf0 msgr2=0x7f08ec102110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.514+0000 7f08f2000640 1 -- 172.21.15.146:0/4251976640 shutdown_connections 2024-04-24T08:21:20.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.514+0000 7f08f2000640 1 -- 172.21.15.146:0/4251976640 wait complete. 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 Processor -- start 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 -- start start 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec1a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08ec109520 0x7f08ec1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08ec19a910 0x7f08ec19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec1145b0 con 0x7f08ec109520 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec114430 con 0x7f08ec19a910 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08f2000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec114730 con 0x7f08ec104450 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08e3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08ec109520 0x7f08ec1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.515+0000 7f08eb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec1a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec1a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44788/0 (socket says 172.21.15.146:44788) 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 -- 172.21.15.146:0/3792754279 learned_addr learned my addr 172.21.15.146:0/3792754279 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08ebe00640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08ec19a910 0x7f08ec19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 -- 172.21.15.146:0/3792754279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08ec19a910 msgr2=0x7f08ec19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08ec19a910 0x7f08ec19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 -- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08ec109520 msgr2=0x7f08ec1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08ec109520 0x7f08ec1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 -- 172.21.15.146:0/3792754279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec19b430 con 0x7f08ec104450 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08e3e00640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08ec109520 0x7f08ec1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:20.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.516+0000 7f08eb400640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec1a0710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f08dc035040 tx=0x7f08dc033b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.517+0000 7f08e8c00640 1 -- 172.21.15.146:0/3792754279 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc037040 con 0x7f08ec104450 2024-04-24T08:21:20.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.517+0000 7f08e8c00640 1 -- 172.21.15.146:0/3792754279 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08dc040070 con 0x7f08ec104450 2024-04-24T08:21:20.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.517+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec19b6c0 con 0x7f08ec104450 2024-04-24T08:21:20.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.517+0000 7f08e8c00640 1 -- 172.21.15.146:0/3792754279 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0039b0 con 0x7f08ec104450 2024-04-24T08:21:20.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.517+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec1a78c0 con 0x7f08ec104450 2024-04-24T08:21:20.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.518+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ec04fd90 con 0x7f08ec104450 2024-04-24T08:21:20.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.519+0000 7f4a98a00640 1 -- 172.21.15.146:0/3389945129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94100c70 msgr2=0x7f4a94101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.519+0000 7f4a98a00640 1 --2- 172.21.15.146:0/3389945129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94100c70 0x7f4a94101050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009900 tx=0x7f4a8402a410 comp rx=0 tx=0).stop 2024-04-24T08:21:20.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.520+0000 7f4a98a00640 1 -- 172.21.15.146:0/3389945129 shutdown_connections 2024-04-24T08:21:20.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.520+0000 7f4a98a00640 1 --2- 172.21.15.146:0/3389945129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94101620 0x7f4a9410e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.520+0000 7f4a98a00640 1 --2- 172.21.15.146:0/3389945129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a9410e820 0x7f4a94110c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.520+0000 7f4a98a00640 1 --2- 172.21.15.146:0/3389945129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94100c70 0x7f4a94101050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.520+0000 7f4a98a00640 1 -- 172.21.15.146:0/3389945129 >> 172.21.15.146:0/3389945129 conn(0x7f4a940fc880 msgr2=0x7f4a940feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.520+0000 7f4a98a00640 1 -- 172.21.15.146:0/3389945129 shutdown_connections 2024-04-24T08:21:20.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.520+0000 7f4a98a00640 1 -- 172.21.15.146:0/3389945129 wait complete. 2024-04-24T08:21:20.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f08e8c00640 1 -- 172.21.15.146:0/3792754279 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f08dc0305e0 con 0x7f08ec104450 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 Processor -- start 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 -- start start 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94100c70 0x7f4a941a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94101620 0x7f4a941a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a9410e820 0x7f4a9419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94114510 con 0x7f4a94101620 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94114390 con 0x7f4a94100c70 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a98a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94114690 con 0x7f4a9410e820 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.521+0000 7f4a92a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94101620 0x7f4a941a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a9410e820 0x7f4a9419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94100c70 0x7f4a941a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a9410e820 0x7f4a9419a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44800/0 (socket says 172.21.15.146:44800) 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a92a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94101620 0x7f4a941a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39306/0 (socket says 172.21.15.146:39306) 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 -- 172.21.15.146:0/2956410467 learned_addr learned my addr 172.21.15.146:0/2956410467 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 -- 172.21.15.146:0/2956410467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94100c70 msgr2=0x7f4a941a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94100c70 0x7f4a941a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 -- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94101620 msgr2=0x7f4a941a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94101620 0x7f4a941a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 -- 172.21.15.146:0/2956410467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9419b000 con 0x7f4a9410e820 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a92a00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94101620 0x7f4a941a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:20.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f08e8c00640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08d00a3f60 0x7f08d00a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.522+0000 7f4a93e00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a9410e820 0x7f4a9419a830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00ec50 tx=0x7f4a8c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.523+0000 7f08e8c00640 1 -- 172.21.15.146:0/3792754279 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f08dc0f0f20 con 0x7f08ec104450 2024-04-24T08:21:20.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.523+0000 7f08e3e00640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08d00a3f60 0x7f08d00a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.523+0000 7f08e3e00640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08d00a3f60 0x7f08d00a6420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f08ec10ab80 tx=0x7f08d4005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.524+0000 7f4a8be00640 1 -- 172.21.15.146:0/2956410467 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c003b70 con 0x7f4a9410e820 2024-04-24T08:21:20.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.524+0000 7f4a8be00640 1 -- 172.21.15.146:0/2956410467 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a8c003d10 con 0x7f4a9410e820 2024-04-24T08:21:20.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.524+0000 7f4a8be00640 1 -- 172.21.15.146:0/2956410467 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c0109e0 con 0x7f4a9410e820 2024-04-24T08:21:20.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.524+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9419b2f0 con 0x7f4a9410e820 2024-04-24T08:21:20.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.524+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941a7510 con 0x7f4a9410e820 2024-04-24T08:21:20.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.524+0000 7f08e8c00640 1 -- 172.21.15.146:0/3792754279 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08dc0bd430 con 0x7f08ec104450 2024-04-24T08:21:20.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.527+0000 7f4a8be00640 1 -- 172.21.15.146:0/2956410467 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f4a8c003ec0 con 0x7f4a9410e820 2024-04-24T08:21:20.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.527+0000 7f4a9cf9a740 1 Processor -- start 2024-04-24T08:21:20.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a9cf9a740 1 -- start start 2024-04-24T08:21:20.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810ab650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f181010e70 con 0x55f1810ab270 2024-04-24T08:21:20.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f180fb7380 con 0x55f1810b6260 2024-04-24T08:21:20.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f18100ce30 con 0x55f1810abb90 2024-04-24T08:21:20.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a92a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810ab650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a92a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810ab650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39320/0 (socket says 172.21.15.146:39320) 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.528+0000 7f4a92a00640 1 -- 172.21.15.146:0/2023000988 learned_addr learned my addr 172.21.15.146:0/2023000988 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a92a00640 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a93400640 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a92a00640 1 -- 172.21.15.146:0/2023000988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 msgr2=0x55f1810b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a92a00640 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a92a00640 1 -- 172.21.15.146:0/2023000988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 msgr2=0x55f1810b8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a92a00640 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a92a00640 1 -- 172.21.15.146:0/2023000988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f180deb7e0 con 0x55f1810ab270 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a8be00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4a780a3ec0 0x7f4a780a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a92a00640 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810ab650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009710 tx=0x7f4a80009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744c995ad09fa4d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a8be00640 1 -- 172.21.15.146:0/2956410467 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4a8c0d1060 con 0x7f4a9410e820 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.529+0000 7f4a93400640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4a780a3ec0 0x7f4a780a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.530+0000 7f4a7fe00640 1 -- 172.21.15.146:0/2023000988 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80013d80 con 0x55f1810ab270 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.530+0000 7f4a7fe00640 1 -- 172.21.15.146:0/2023000988 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a80011c70 con 0x55f1810ab270 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.530+0000 7f4a7fe00640 1 -- 172.21.15.146:0/2023000988 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80003bc0 con 0x55f1810ab270 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2023000988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 msgr2=0x55f1810ab650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810ab650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009710 tx=0x7f4a80009b30 comp rx=0 tx=0).stop 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a93400640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4a780a3ec0 0x7f4a780a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a84000c00 tx=0x7f4a84005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2023000988 shutdown_connections 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2023000988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810ab650 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2023000988 >> 172.21.15.146:0/2023000988 conn(0x55f1810a7590 msgr2=0x55f1810a7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2023000988 shutdown_connections 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2023000988 wait complete. 2024-04-24T08:21:20.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.531+0000 7f4a9cf9a740 1 -- start start 2024-04-24T08:21:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 1 Processor -- start 2024-04-24T08:21:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810b3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f1810b9050 con 0x55f1810ab270 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f180fca660 con 0x55f1810b6260 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f1810b91d0 con 0x55f1810abb90 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.532+0000 7f4a9cf9a740 10 client.0 _ll_register_callbacks cb 0x55f180fb8aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a93400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44816/0 (socket says 172.21.15.146:44816) 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 -- 172.21.15.146:0/1203844849 learned_addr learned my addr 172.21.15.146:0/1203844849 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a93e00640 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 -- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 msgr2=0x55f1810b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 -- 172.21.15.146:0/1203844849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 msgr2=0x55f1810b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810b3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 -- 172.21.15.146:0/1203844849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f180e4bf70 con 0x55f1810abb90 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a92a00640 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b3750 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a80005660 tx=0x7f4a80003700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dfb856b1ffc5d0c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1203844849 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8002d060 con 0x55f1810abb90 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.533+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1203844849 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4a800046c0 con 0x55f1810abb90 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a80011c70 con 0x55f1810abb90 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1203844849 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4a800048c0 con 0x55f1810abb90 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 10 client.25402 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f180ce3170 con 0x55f1810abb90 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1203844849 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a80011c70 con 0x55f1810abb90 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1203844849 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4a80028410 con 0x55f1810abb90 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 10 client.25402 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 10 client.25402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 20 client.25402 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f1810c64f0 0x55f1810c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 4 client.25402 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.534+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f18101e890 con 0x55f1810c64f0 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.535+0000 7f4a93400640 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f1810c64f0 0x55f1810c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.535+0000 7f4a93400640 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f1810c64f0 0x55f1810c88e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f5901687b75404a server_cookie=f7eeb97553ec8ceb in_seq=0 out_seq=0 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.535+0000 7f4a7cc00640 10 client.25402 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1203844849 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f18101e890 con 0x55f1810c64f0 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a7cc00640 10 client.25402 handle_command_reply: tid=0 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 client.25402 shutdown 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 msgr2=0x55f1810b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b3750 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a80005660 tx=0x7f4a80003700 comp rx=0 tx=0).stop 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 shutdown_connections 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f1810c64f0 0x55f1810c88e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810b3750 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810b6260 0x55f1810b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1203844849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810ab270 0x55f1810b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a7cc00640 0 client.25402 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.536+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 >> 172.21.15.146:0/1203844849 conn(0x55f1810a7590 msgr2=0x55f1810ad390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.537+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 shutdown_connections 2024-04-24T08:21:20.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.537+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1203844849 wait complete. 2024-04-24T08:21:20.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.537+0000 7f4a9cf9a740 20 client.25402 trim_cache size 0 max 16384 2024-04-24T08:21:20.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.555+0000 7f4a9cf9a740 1 Processor -- start 2024-04-24T08:21:20.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a9cf9a740 1 -- start start 2024-04-24T08:21:20.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f1810cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941b2620 con 0x55f1810b6260 2024-04-24T08:21:20.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941afef0 con 0x55f1810ab270 2024-04-24T08:21:20.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941add40 con 0x55f1810abb90 2024-04-24T08:21:20.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a93e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f1810cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a93e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f1810cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39340/0 (socket says 172.21.15.146:39340) 2024-04-24T08:21:20.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a93e00640 1 -- 172.21.15.146:0/2531039131 learned_addr learned my addr 172.21.15.146:0/2531039131 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a93400640 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.556+0000 7f4a92a00640 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a93400640 1 -- 172.21.15.146:0/2531039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 msgr2=0x55f1810ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a93400640 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a93400640 1 -- 172.21.15.146:0/2531039131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 msgr2=0x55f1810cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a93400640 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f1810cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a93400640 1 -- 172.21.15.146:0/2531039131 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1810b21c0 con 0x55f1810ab270 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a93e00640 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f1810cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a92a00640 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a93400640 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810c1f10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4a8402aa00 tx=0x7f4a8403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0aa39d94cd219d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a7f400640 1 -- 172.21.15.146:0/2531039131 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a84035040 con 0x55f1810ab270 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a7f400640 1 -- 172.21.15.146:0/2531039131 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a84054810 con 0x55f1810ab270 2024-04-24T08:21:20.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.557+0000 7f4a7f400640 1 -- 172.21.15.146:0/2531039131 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a840325c0 con 0x55f1810ab270 2024-04-24T08:21:20.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.558+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2531039131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 msgr2=0x55f1810c1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.558+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810c1f10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4a8402aa00 tx=0x7f4a8403f000 comp rx=0 tx=0).stop 2024-04-24T08:21:20.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2531039131 shutdown_connections 2024-04-24T08:21:20.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f1810cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/2531039131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810c1f10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2531039131 >> 172.21.15.146:0/2531039131 conn(0x55f1810a7590 msgr2=0x55f1810b52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2531039131 shutdown_connections 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/2531039131 wait complete. 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.559+0000 7f4a9cf9a740 1 -- start start 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 1 Processor -- start 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810ba770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810bacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f181125880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941ae3b0 con 0x55f1810b6260 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941af820 con 0x55f1810ab270 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941aea80 con 0x55f1810abb90 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a9cf9a740 10 client.0 _ll_register_callbacks cb 0x7f4a800136b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:20.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.560+0000 7f4a93e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f181125880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f181125880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39342/0 (socket says 172.21.15.146:39342) 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 -- 172.21.15.146:0/1598861409 learned_addr learned my addr 172.21.15.146:0/1598861409 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a92a00640 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810bacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93400640 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810ba770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 -- 172.21.15.146:0/1598861409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 msgr2=0x55f1810bacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810bacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 -- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 msgr2=0x55f1810ba770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810ba770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 -- 172.21.15.146:0/1598861409 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f181125e70 con 0x55f1810b6260 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a93e00640 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f181125880 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0a54a0 tx=0x7f4a8c09cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8543f57c4dfaf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1598861409 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c0bb060 con 0x55f1810b6260 2024-04-24T08:21:20.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1598861409 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4a8c0ce8c0 con 0x55f1810b6260 2024-04-24T08:21:20.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.561+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a84054810 con 0x55f1810b6260 2024-04-24T08:21:20.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1598861409 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4a8c082050 con 0x55f1810b6260 2024-04-24T08:21:20.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 10 client.15279 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:20.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f18112d2b0 con 0x55f1810b6260 2024-04-24T08:21:20.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1598861409 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a84054810 con 0x55f1810b6260 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1598861409 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4a8c010b80 con 0x55f1810b6260 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 10 client.15279 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 10 client.15279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 20 client.15279 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f18112d9b0 0x55f18112fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 4 client.15279 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.562+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f181018700 con 0x55f18112d9b0 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.563+0000 7f4a93400640 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f18112d9b0 0x55f18112fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.563+0000 7f4a93400640 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f18112d9b0 0x55f18112fdd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf4f56b6665145ae server_cookie=803dae76a21d60d7 in_seq=0 out_seq=0 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.563+0000 7f4a7cc00640 10 client.15279 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.564+0000 7f4a7cc00640 1 -- 172.21.15.146:0/1598861409 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f181018700 con 0x55f18112d9b0 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.564+0000 7f4a7cc00640 10 client.15279 handle_command_reply: tid=0 2024-04-24T08:21:20.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.564+0000 7f4a9cf9a740 1 client.15279 shutdown 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.564+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 msgr2=0x55f181125880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.564+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f181125880 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0a54a0 tx=0x7f4a8c09cc20 comp rx=0 tx=0).stop 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.564+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 shutdown_connections 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.564+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f18112d9b0 0x55f18112fdd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1810abb90 0x55f1810bacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f1810b6260 0x55f181125880 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a9cf9a740 1 --2- 172.21.15.146:0/1598861409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f1810ab270 0x55f1810ba770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a7cc00640 0 client.15279 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 >> 172.21.15.146:0/1598861409 conn(0x55f1810a7590 msgr2=0x55f1810b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 shutdown_connections 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a9cf9a740 1 -- 172.21.15.146:0/1598861409 wait complete. 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.565+0000 7f4a9cf9a740 20 client.15279 trim_cache size 0 max 16384 2024-04-24T08:21:20.569 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:21:20.570 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:21:20.570 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "e3445712-33f1-4f76-b1e9-c3e084309244", 2024-04-24T08:21:20.570 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:21:20.570 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4a780a3ec0 msgr2=0x7f4a780a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4a780a3ec0 0x7f4a780a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a84000c00 tx=0x7f4a84005b50 comp rx=0 tx=0).stop 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a9410e820 msgr2=0x7f4a9419a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a9410e820 0x7f4a9419a830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00ec50 tx=0x7f4a8c00c6a0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 shutdown_connections 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4a780a3ec0 0x7f4a780a6380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a9410e820 0x7f4a9419a830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a94101620 0x7f4a941a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 --2- 172.21.15.146:0/2956410467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4a94100c70 0x7f4a941a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 >> 172.21.15.146:0/2956410467 conn(0x7f4a940fc880 msgr2=0x7f4a9410faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 shutdown_connections 2024-04-24T08:21:20.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.566+0000 7f4a98a00640 1 -- 172.21.15.146:0/2956410467 wait complete. 2024-04-24T08:21:20.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:21:20.587 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:21:20.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.760+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f08ec104e00 con 0x7f08d00a3f60 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 -- 172.21.15.146:0/3435105354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102ac0 msgr2=0x7faaf0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3435105354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102ac0 0x7faaf0106b00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faadc009860 tx=0x7faadc02a340 comp rx=0 tx=0).stop 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 -- 172.21.15.146:0/3435105354 shutdown_connections 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3435105354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faaf01071e0 0x7faaf010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3435105354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102ac0 0x7faaf0106b00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3435105354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf0102110 0x7faaf01024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 -- 172.21.15.146:0/3435105354 >> 172.21.15.146:0/3435105354 conn(0x7faaf00fd9d0 msgr2=0x7faaf00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 -- 172.21.15.146:0/3435105354 shutdown_connections 2024-04-24T08:21:20.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.772+0000 7faaf4a00640 1 -- 172.21.15.146:0/3435105354 wait complete. 2024-04-24T08:21:20.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.773+0000 7faaf4a00640 1 Processor -- start 2024-04-24T08:21:20.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.773+0000 7faaf4a00640 1 -- start start 2024-04-24T08:21:20.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaf4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102110 0x7faaf0074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaf4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faaf01071e0 0x7faaf0074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaf4a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf0074d80 0x7faaf01a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaf4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faaf0112250 con 0x7faaf0102110 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaf4a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01120d0 con 0x7faaf0074d80 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaf4a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01123d0 con 0x7faaf01071e0 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaee800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102110 0x7faaf0074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaee800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102110 0x7faaf0074300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39358/0 (socket says 172.21.15.146:39358) 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaee800640 1 -- 172.21.15.146:0/3636481207 learned_addr learned my addr 172.21.15.146:0/3636481207 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaef200640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf0074d80 0x7faaf01a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.774+0000 7faaede00640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faaf01071e0 0x7faaf0074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaee800640 1 -- 172.21.15.146:0/3636481207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faaf01071e0 msgr2=0x7faaf0074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaee800640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faaf01071e0 0x7faaf0074840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaee800640 1 -- 172.21.15.146:0/3636481207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf0074d80 msgr2=0x7faaf01a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaee800640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf0074d80 0x7faaf01a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaee800640 1 -- 172.21.15.146:0/3636481207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf01a70a0 con 0x7faaf0102110 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaef200640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf0074d80 0x7faaf01a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:20.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaee800640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102110 0x7faaf0074300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faae000ea60 tx=0x7faae000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faae7400640 1 -- 172.21.15.146:0/3636481207 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae000cde0 con 0x7faaf0102110 2024-04-24T08:21:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faae7400640 1 -- 172.21.15.146:0/3636481207 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae00035f0 con 0x7faaf0102110 2024-04-24T08:21:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf01a7390 con 0x7faaf0102110 2024-04-24T08:21:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.775+0000 7faae7400640 1 -- 172.21.15.146:0/3636481207 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae0010670 con 0x7faaf0102110 2024-04-24T08:21:20.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.776+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf01a78b0 con 0x7faaf0102110 2024-04-24T08:21:20.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 Processor -- start 2024-04-24T08:21:20.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 -- start start 2024-04-24T08:21:20.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d5a30 0x5557184d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x5557184e04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184e0a20 0x5557184e2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55571843b1f0 con 0x5557184e0a20 2024-04-24T08:21:20.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5557183e1350 con 0x5557184d5a30 2024-04-24T08:21:20.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaf8270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5557184371b0 con 0x5557184d6350 2024-04-24T08:21:20.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faae7400640 1 -- 172.21.15.146:0/3636481207 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7faae00037a0 con 0x7faaf0102110 2024-04-24T08:21:20.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.780+0000 7faaef200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d5a30 0x5557184d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d5a30 0x5557184d5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:51378/0 (socket says 172.21.15.146:51378) 2024-04-24T08:21:20.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 -- 172.21.15.146:0/4110406839 learned_addr learned my addr 172.21.15.146:0/4110406839 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaede00640 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184e0a20 0x5557184e2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaee800640 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x5557184e04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 -- 172.21.15.146:0/4110406839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 msgr2=0x5557184e04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x5557184e04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 -- 172.21.15.146:0/4110406839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184e0a20 msgr2=0x5557184e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184e0a20 0x5557184e2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 -- 172.21.15.146:0/4110406839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555718435f50 con 0x5557184d5a30 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaede00640 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184e0a20 0x5557184e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaee800640 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x5557184e04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.781+0000 7faaef200640 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d5a30 0x5557184d5e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faae8009fc0 tx=0x7faae800ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2145fa5e6fd494 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.782+0000 7faadb400640 1 -- 172.21.15.146:0/4110406839 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae801d070 con 0x5557184d5a30 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.782+0000 7faadb400640 1 -- 172.21.15.146:0/4110406839 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae8003e50 con 0x5557184d5a30 2024-04-24T08:21:20.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.782+0000 7faadb400640 1 -- 172.21.15.146:0/4110406839 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8004770 con 0x5557184d5a30 2024-04-24T08:21:20.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faaf8270740 1 -- 172.21.15.146:0/4110406839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d5a30 msgr2=0x5557184d5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faaf8270740 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d5a30 0x5557184d5e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faae8009fc0 tx=0x7faae800ec30 comp rx=0 tx=0).stop 2024-04-24T08:21:20.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faae7400640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faad40a3fc0 0x7faad40a6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faaf8270740 1 -- 172.21.15.146:0/4110406839 shutdown_connections 2024-04-24T08:21:20.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faae7400640 1 -- 172.21.15.146:0/3636481207 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faae00d0c90 con 0x7faaf0102110 2024-04-24T08:21:20.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faaf8270740 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x5557184e04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faaf8270740 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184e0a20 0x5557184e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faaf8270740 1 --2- 172.21.15.146:0/4110406839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d5a30 0x5557184d5e10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.783+0000 7faaf8270740 1 -- 172.21.15.146:0/4110406839 >> 172.21.15.146:0/4110406839 conn(0x5557184d1d50 msgr2=0x5557184d2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.784+0000 7faaee800640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faad40a3fc0 0x7faad40a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.784+0000 7faaf8270740 1 -- 172.21.15.146:0/4110406839 shutdown_connections 2024-04-24T08:21:20.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.784+0000 7faaf8270740 1 -- 172.21.15.146:0/4110406839 wait complete. 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.785+0000 7faaf8270740 1 -- start start 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.785+0000 7faaf8270740 1 Processor -- start 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.785+0000 7faaee800640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faad40a3fc0 0x7faad40a6480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faae00ce270 tx=0x7faae000ef90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaf8270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184dda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaf8270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184ddf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaf8270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184de490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaf8270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5557184e3810 con 0x5557184d5a30 2024-04-24T08:21:20.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaf8270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5557183f4690 con 0x5557184d6350 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaf8270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5557184e3990 con 0x5557184e0a20 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaf8270740 10 client.0 _ll_register_callbacks cb 0x555718430650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184ddf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaef200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184dda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184ddf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:51390/0 (socket says 172.21.15.146:51390) 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 -- 172.21.15.146:0/1912163447 learned_addr learned my addr 172.21.15.146:0/1912163447 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaef200640 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184de490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 -- 172.21.15.146:0/1912163447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 msgr2=0x5557184de490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184de490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 -- 172.21.15.146:0/1912163447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 msgr2=0x5557184dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184dda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaede00640 1 -- 172.21.15.146:0/1912163447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557183f3920 con 0x5557184d6350 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.786+0000 7faaef200640 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184dda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faaede00640 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184ddf50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faaf006b560 tx=0x7faadc009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f30e2b510246c3dd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faacfe00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faacfe00640 1 -- 172.21.15.146:0/1912163447 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc037040 con 0x5557184d6350 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faae8003e50 con 0x5557184d6350 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faacfe00640 1 -- 172.21.15.146:0/1912163447 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faadc004450 con 0x5557184d6350 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faacfe00640 1 -- 172.21.15.146:0/1912163447 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faadc04a050 con 0x5557184d6350 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faaf8270740 10 client.25421 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.787+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5557182177d0 con 0x5557184d6350 2024-04-24T08:21:20.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faacfe00640 1 -- 172.21.15.146:0/1912163447 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae8003e50 con 0x5557184d6350 2024-04-24T08:21:20.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faacfe00640 1 -- 172.21.15.146:0/1912163447 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7faadc045440 con 0x5557184d6350 2024-04-24T08:21:20.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faaf8270740 10 client.25421 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:20.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faaf8270740 10 client.25421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:20.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faaf8270740 20 client.25421 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:20.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faaf8270740 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5557184f0d10 0x5557184f3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faaf8270740 4 client.25421 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:20.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.788+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555718448c10 con 0x5557184f0d10 2024-04-24T08:21:20.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.789+0000 7faaef200640 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5557184f0d10 0x5557184f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.789+0000 7faaef200640 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5557184f0d10 0x5557184f3120 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18048bda44e196f3 server_cookie=c8b86415e0878ebd in_seq=0 out_seq=0 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.789+0000 7faacfe00640 10 client.25421 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.790+0000 7faacfe00640 1 -- 172.21.15.146:0/1912163447 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555718448c10 con 0x5557184f0d10 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.790+0000 7faacfe00640 10 client.25421 handle_command_reply: tid=0 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.790+0000 7faaf8270740 1 client.25421 shutdown 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.790+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 msgr2=0x5557184ddf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.790+0000 7faaf8270740 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184ddf50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faaf006b560 tx=0x7faadc009510 comp rx=0 tx=0).stop 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 shutdown_connections 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5557184f0d10 0x5557184f3120 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184de490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184ddf50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 --2- 172.21.15.146:0/1912163447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184dda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faacfe00640 0 client.25421 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 >> 172.21.15.146:0/1912163447 conn(0x5557184d1d50 msgr2=0x5557184d7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 shutdown_connections 2024-04-24T08:21:20.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 1 -- 172.21.15.146:0/1912163447 wait complete. 2024-04-24T08:21:20.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.791+0000 7faaf8270740 20 client.25421 trim_cache size 0 max 16384 2024-04-24T08:21:20.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.812+0000 7faaf8270740 1 Processor -- start 2024-04-24T08:21:20.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaf8270740 1 -- start start 2024-04-24T08:21:20.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaf8270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaf8270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184de260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaf8270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaf8270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01bf320 con 0x5557184d5a30 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaf8270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01bcbf0 con 0x5557184d6350 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaf8270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01baa80 con 0x5557184e0a20 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaef200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaede00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaef200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184ec6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:39398/0 (socket says 172.21.15.146:39398) 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaef200640 1 -- 172.21.15.146:0/2562343949 learned_addr learned my addr 172.21.15.146:0/2562343949 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.813+0000 7faaef200640 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faaede00640 1 -- 172.21.15.146:0/2562343949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 msgr2=0x5557184f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faaede00640 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faaede00640 1 -- 172.21.15.146:0/2562343949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 msgr2=0x5557184ec6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faaede00640 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184ec6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faaede00640 1 -- 172.21.15.146:0/2562343949 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55571814d3f0 con 0x5557184d6350 2024-04-24T08:21:20.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faaede00640 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184de260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faadc045a50 tx=0x7faadc02a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6338a2262959ff01 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faadaa00640 1 -- 172.21.15.146:0/2562343949 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc053070 con 0x5557184d6350 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faadaa00640 1 -- 172.21.15.146:0/2562343949 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faadc045440 con 0x5557184d6350 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.814+0000 7faadaa00640 1 -- 172.21.15.146:0/2562343949 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc037070 con 0x5557184d6350 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 -- 172.21.15.146:0/2562343949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 msgr2=0x5557184de260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184de260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faadc045a50 tx=0x7faadc02a340 comp rx=0 tx=0).stop 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 -- 172.21.15.146:0/2562343949 shutdown_connections 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184e0a20 0x5557184f89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184d6350 0x5557184de260 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 --2- 172.21.15.146:0/2562343949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x5557184ec6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 -- 172.21.15.146:0/2562343949 >> 172.21.15.146:0/2562343949 conn(0x5557184d1d50 msgr2=0x5557184d7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 -- 172.21.15.146:0/2562343949 shutdown_connections 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.815+0000 7faaf8270740 1 -- 172.21.15.146:0/2562343949 wait complete. 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 -- start start 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 Processor -- start 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x55571854f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x55571854faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184e0a20 0x5557184e4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01bb0d0 con 0x5557184d5a30 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01bc520 con 0x5557184e0a20 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.816+0000 7faaf8270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01bb780 con 0x5557184d6350 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaf8270740 10 client.0 _ll_register_callbacks cb 0x7faae0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:20.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x55571854faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x55571854faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44892/0 (socket says 172.21.15.146:44892) 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 -- 172.21.15.146:0/1770726881 learned_addr learned my addr 172.21.15.146:0/1770726881 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaef200640 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x55571854f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 -- 172.21.15.146:0/1770726881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184e0a20 msgr2=0x5557184e4f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184e0a20 0x5557184e4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 -- 172.21.15.146:0/1770726881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 msgr2=0x55571854f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x55571854f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.817+0000 7faaede00640 1 -- 172.21.15.146:0/1770726881 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55571810d0f0 con 0x5557184d6350 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faaede00640 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x55571854faa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faadc005e00 tx=0x7faadc005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f01e0574c59fe07 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:20.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faacfe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:21:20.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faacfe00640 1 -- 172.21.15.146:0/1770726881 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc053030 con 0x5557184d6350 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faadc045440 con 0x5557184d6350 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faacfe00640 1 -- 172.21.15.146:0/1770726881 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faadc004250 con 0x5557184d6350 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faacfe00640 1 -- 172.21.15.146:0/1770726881 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7faadc037040 con 0x5557184d6350 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faaf8270740 10 client.25423 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.818+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555718400810 con 0x5557184d6350 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faacfe00640 1 -- 172.21.15.146:0/1770726881 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faadc045440 con 0x5557184d6350 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faacfe00640 1 -- 172.21.15.146:0/1770726881 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7faadc003920 con 0x5557184d6350 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faaf8270740 10 client.25423 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faaf8270740 10 client.25423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faaf8270740 20 client.25423 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faaf8270740 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555718558120 0x55571855a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faaf8270740 4 client.25423 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:21:20.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555718442a80 con 0x555718558120 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.819+0000 7faaef200640 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555718558120 0x55571855a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaef200640 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555718558120 0x55571855a540 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e061ccbae83144 server_cookie=7d4d0e3b26905879 in_seq=0 out_seq=0 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faacfe00640 10 client.25423 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faacfe00640 1 -- 172.21.15.146:0/1770726881 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555718442a80 con 0x555718558120 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faacfe00640 10 client.25423 handle_command_reply: tid=0 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 client.25423 shutdown 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 msgr2=0x55571854faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x55571854faa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faadc005e00 tx=0x7faadc005970 comp rx=0 tx=0).stop 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 shutdown_connections 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555718558120 0x55571855a540 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5557184d6350 0x55571854faa0 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faadc005e00 tx=0x7faadc005970 comp rx=0 tx=0).stop 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5557184e0a20 0x5557184e4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 --2- 172.21.15.146:0/1770726881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5557184d5a30 0x55571854f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faacfe00640 0 client.25423 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 >> 172.21.15.146:0/1770726881 conn(0x5557184d1d50 msgr2=0x5557184d7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.820+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 shutdown_connections 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.821+0000 7faaf8270740 1 -- 172.21.15.146:0/1770726881 wait complete. 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.821+0000 7faaf8270740 20 client.25423 trim_cache size 0 max 16384 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stdout: "e3445712-33f1-4f76-b1e9-c3e084309244": { 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:21:20.826 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "e3445712-33f1-4f76-b1e9-c3e084309244", 2024-04-24T08:21:20.827 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:21:20.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:21:20.827 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:21:20.827 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:21:20.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.821+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faad40a3fc0 msgr2=0x7faad40a6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.821+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faad40a3fc0 0x7faad40a6480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faae00ce270 tx=0x7faae000ef90 comp rx=0 tx=0).stop 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.821+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102110 msgr2=0x7faaf0074300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.821+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102110 0x7faaf0074300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faae000ea60 tx=0x7faae000ef20 comp rx=0 tx=0).stop 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 shutdown_connections 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faad40a3fc0 0x7faad40a6480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faaf01071e0 0x7faaf0074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faaf0074d80 0x7faaf01a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 --2- 172.21.15.146:0/3636481207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faaf0102110 0x7faaf0074300 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 >> 172.21.15.146:0/3636481207 conn(0x7faaf00fd9d0 msgr2=0x7faaf00fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 shutdown_connections 2024-04-24T08:21:20.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.822+0000 7faaf4a00640 1 -- 172.21.15.146:0/3636481207 wait complete. 2024-04-24T08:21:20.839 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:20.839 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.0 KiB/s wr, 22 op/s 2024-04-24T08:21:20.839 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:20.839 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:20.840 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:20.840 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:20.840 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3626058735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:20.840 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:20 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4083932523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:20.842 INFO:tasks.cephfs.filesystem:scrub status for tag:e3445712-33f1-4f76-b1e9-c3e084309244 - {'path': '/', 'tag': 'e3445712-33f1-4f76-b1e9-c3e084309244', 'options': 'recursive,force'} 2024-04-24T08:21:20.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.843+0000 7f08e8c00640 1 -- 172.21.15.146:0/3792754279 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f08ec104e00 con 0x7f08d00a3f60 2024-04-24T08:21:20.845 INFO:teuthology.orchestra.run.smithi146.stdout:Schedule set for path / 2024-04-24T08:21:20.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.845+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08d00a3f60 msgr2=0x7f08d00a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.845+0000 7f08f2000640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08d00a3f60 0x7f08d00a6420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f08ec10ab80 tx=0x7f08d4005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.845+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 msgr2=0x7f08ec1a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.845+0000 7f08f2000640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec1a0710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f08dc035040 tx=0x7f08dc033b60 comp rx=0 tx=0).stop 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 shutdown_connections 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08d00a3f60 0x7f08d00a6420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08ec19a910 0x7f08ec19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08ec109520 0x7f08ec1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 --2- 172.21.15.146:0/3792754279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08ec104450 0x7f08ec1a0710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 >> 172.21.15.146:0/3792754279 conn(0x7f08ec0ffcf0 msgr2=0x7f08ec101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 shutdown_connections 2024-04-24T08:21:20.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:20.846+0000 7f08f2000640 1 -- 172.21.15.146:0/3792754279 wait complete. 2024-04-24T08:21:20.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-24T08:21:20.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.0 KiB/s wr, 22 op/s 2024-04-24T08:21:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3626058735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:20 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4083932523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:21.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: Updating smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:21:21.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.0 KiB/s wr, 22 op/s 2024-04-24T08:21:21.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:21.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:21.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:21.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:21:21.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3626058735' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:21.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:20 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4083932523' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 -- 172.21.15.146:0/2880857019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076e50 msgr2=0x7f0dbc112770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/2880857019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076e50 0x7f0dbc112770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009930 tx=0x7f0da402a420 comp rx=0 tx=0).stop 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 -- 172.21.15.146:0/2880857019 shutdown_connections 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/2880857019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076e50 0x7f0dbc112770 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/2880857019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dbc112cb0 0x7f0dbc1150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/2880857019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dbc076530 0x7f0dbc076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 -- 172.21.15.146:0/2880857019 >> 172.21.15.146:0/2880857019 conn(0x7f0dbc0ffb20 msgr2=0x7f0dbc101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:21.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 -- 172.21.15.146:0/2880857019 shutdown_connections 2024-04-24T08:21:21.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.097+0000 7f0dbbe00640 1 -- 172.21.15.146:0/2880857019 wait complete. 2024-04-24T08:21:21.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.098+0000 7f0dbbe00640 1 Processor -- start 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.098+0000 7f0dbbe00640 1 -- start start 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076530 0x7f0dbc19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dbc076e50 0x7f0dbc19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dbc112cb0 0x7f0dbc1a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc118870 con 0x7f0dbc112cb0 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbbe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1186f0 con 0x7f0dbc076e50 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0dbc1189f0 con 0x7f0dbc076530 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbb400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dbc112cb0 0x7f0dbc1a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0db2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dbc076e50 0x7f0dbc19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076530 0x7f0dbc19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0db2000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dbc076e50 0x7f0dbc19f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:47816/0 (socket says 172.21.15.146:47816) 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0dbaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076530 0x7f0dbc19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54438/0 (socket says 172.21.15.146:54438) 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.099+0000 7f0db2000640 1 -- 172.21.15.146:0/1085952477 learned_addr learned my addr 172.21.15.146:0/1085952477 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbaa00640 1 -- 172.21.15.146:0/1085952477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dbc076e50 msgr2=0x7f0dbc19f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbaa00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dbc076e50 0x7f0dbc19f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbaa00640 1 -- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dbc112cb0 msgr2=0x7f0dbc1a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbaa00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dbc112cb0 0x7f0dbc1a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbaa00640 1 -- 172.21.15.146:0/1085952477 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dbc1a3e10 con 0x7f0dbc076530 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbaa00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076530 0x7f0dbc19ee70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00e9f0 tx=0x7f0dac00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0db3e00640 1 -- 172.21.15.146:0/1085952477 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00ccc0 con 0x7f0dbc076530 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dbc1a4100 con 0x7f0dbc076530 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0db3e00640 1 -- 172.21.15.146:0/1085952477 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dac005020 con 0x7f0dbc076530 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0db3e00640 1 -- 172.21.15.146:0/1085952477 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac004a00 con 0x7f0dbc076530 2024-04-24T08:21:21.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.100+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dbc1ab990 con 0x7f0dbc076530 2024-04-24T08:21:21.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.101+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dbc0505e0 con 0x7f0dbc076530 2024-04-24T08:21:21.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.104+0000 7f0db3e00640 1 -- 172.21.15.146:0/1085952477 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f0dac0051d0 con 0x7f0dbc076530 2024-04-24T08:21:21.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.105+0000 7f0db3e00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0da00a3e30 0x7f0da00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.105+0000 7f0db3e00640 1 -- 172.21.15.146:0/1085952477 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0dac0d0b20 con 0x7f0dbc076530 2024-04-24T08:21:21.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.105+0000 7f0db2000640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0da00a3e30 0x7f0da00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.106+0000 7f0db2000640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0da00a3e30 0x7f0da00a62f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0da4005c40 tx=0x7f0da4005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:21.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.107+0000 7f0db3e00640 1 -- 172.21.15.146:0/1085952477 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dac09cfb0 con 0x7f0dbc076530 2024-04-24T08:21:21.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.338+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f0dbc062440 con 0x7f0da00a3e30 2024-04-24T08:21:21.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.475+0000 7f0db3e00640 1 -- 172.21.15.146:0/1085952477 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f0dbc062440 con 0x7f0da00a3e30 2024-04-24T08:21:21.477 INFO:teuthology.orchestra.run.smithi146.stdout:Retention added to path / 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0da00a3e30 msgr2=0x7f0da00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0da00a3e30 0x7f0da00a62f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0da4005c40 tx=0x7f0da4005b50 comp rx=0 tx=0).stop 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076530 msgr2=0x7f0dbc19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076530 0x7f0dbc19ee70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00e9f0 tx=0x7f0dac00edf0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 shutdown_connections 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0da00a3e30 0x7f0da00a62f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dbc112cb0 0x7f0dbc1a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0dbc076e50 0x7f0dbc19f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 --2- 172.21.15.146:0/1085952477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dbc076530 0x7f0dbc19ee70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.478+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 >> 172.21.15.146:0/1085952477 conn(0x7f0dbc0ffb20 msgr2=0x7f0dbc101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:21.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.479+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 shutdown_connections 2024-04-24T08:21:21.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.479+0000 7f0dbbe00640 1 -- 172.21.15.146:0/1085952477 wait complete. 2024-04-24T08:21:21.499 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 -- 172.21.15.146:0/2365677988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 msgr2=0x7f3224112770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 --2- 172.21.15.146:0/2365677988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 0x7f3224112770 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3210009930 tx=0x7f321002a420 comp rx=0 tx=0).stop 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 -- 172.21.15.146:0/2365677988 shutdown_connections 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 --2- 172.21.15.146:0/2365677988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 0x7f3224112770 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 --2- 172.21.15.146:0/2365677988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224112cb0 0x7f32241150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 --2- 172.21.15.146:0/2365677988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3224076530 0x7f3224076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 -- 172.21.15.146:0/2365677988 >> 172.21.15.146:0/2365677988 conn(0x7f32240ffb20 msgr2=0x7f3224101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 -- 172.21.15.146:0/2365677988 shutdown_connections 2024-04-24T08:21:21.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.690+0000 7f322ae00640 1 -- 172.21.15.146:0/2365677988 wait complete. 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.691+0000 7f322ae00640 1 Processor -- start 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.691+0000 7f322ae00640 1 -- start start 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f322ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224076530 0x7f322419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f322ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 0x7f322419f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f322ae00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3224112cb0 0x7f32241a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f322ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3224118870 con 0x7f3224076530 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f322ae00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32241186f0 con 0x7f3224112cb0 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f322ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32241189f0 con 0x7f3224076e50 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224076530 0x7f322419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 0x7f322419f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224076530 0x7f322419ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47910/0 (socket says 172.21.15.146:47910) 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 -- 172.21.15.146:0/3766607815 learned_addr learned my addr 172.21.15.146:0/3766607815 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f322a400640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3224112cb0 0x7f32241a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 -- 172.21.15.146:0/3766607815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 msgr2=0x7f322419f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 0x7f322419f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 -- 172.21.15.146:0/3766607815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3224112cb0 msgr2=0x7f32241a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3224112cb0 0x7f32241a35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229a00640 1 -- 172.21.15.146:0/3766607815 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32241a3d20 con 0x7f3224076530 2024-04-24T08:21:21.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.692+0000 7f3229000640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 0x7f322419f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:21.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.693+0000 7f3229a00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224076530 0x7f322419ed30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f321800e9f0 tx=0x7f321800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:21.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.693+0000 7f3222000640 1 -- 172.21.15.146:0/3766607815 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321800cd80 con 0x7f3224076530 2024-04-24T08:21:21.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.693+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32241a4010 con 0x7f3224076530 2024-04-24T08:21:21.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.693+0000 7f3222000640 1 -- 172.21.15.146:0/3766607815 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3218005500 con 0x7f3224076530 2024-04-24T08:21:21.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.693+0000 7f3222000640 1 -- 172.21.15.146:0/3766607815 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3218010690 con 0x7f3224076530 2024-04-24T08:21:21.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.693+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32241ab8a0 con 0x7f3224076530 2024-04-24T08:21:21.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.694+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3224077c70 con 0x7f3224076530 2024-04-24T08:21:21.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.696+0000 7f3222000640 1 -- 172.21.15.146:0/3766607815 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f32180046c0 con 0x7f3224076530 2024-04-24T08:21:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.698+0000 7f3222000640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f320c0a3eb0 0x7f320c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.698+0000 7f3222000640 1 -- 172.21.15.146:0/3766607815 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f32180d1080 con 0x7f3224076530 2024-04-24T08:21:21.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.698+0000 7f3229000640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f320c0a3eb0 0x7f320c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:21.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.699+0000 7f3229000640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f320c0a3eb0 0x7f320c0a6370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3210006010 tx=0x7f3210031510 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:21.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.703+0000 7f3222000640 1 -- 172.21.15.146:0/3766607815 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3218053ff0 con 0x7f3224076530 2024-04-24T08:21:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:21 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:21:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:21 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:21 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:21:21.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:21 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:21.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:21 smithi146 ceph-mon[27805]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:21.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.933+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f322410deb0 con 0x7f320c0a3eb0 2024-04-24T08:21:21.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.947+0000 7f3222000640 1 -- 172.21.15.146:0/3766607815 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f32100325c0 con 0x7f320c0a3eb0 2024-04-24T08:21:21.948 INFO:teuthology.orchestra.run.smithi146.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-24T00:00:00", "created": "2024-04-24T08:21:20", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-24T08:21:21.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f320c0a3eb0 msgr2=0x7f320c0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f320c0a3eb0 0x7f320c0a6370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3210006010 tx=0x7f3210031510 comp rx=0 tx=0).stop 2024-04-24T08:21:21.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224076530 msgr2=0x7f322419ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:21.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224076530 0x7f322419ed30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f321800e9f0 tx=0x7f321800edf0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 shutdown_connections 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f320c0a3eb0 0x7f320c0a6370 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3224076e50 0x7f322419f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3224112cb0 0x7f32241a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 --2- 172.21.15.146:0/3766607815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3224076530 0x7f322419ed30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.949+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 >> 172.21.15.146:0/3766607815 conn(0x7f32240ffb20 msgr2=0x7f3224101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.950+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 shutdown_connections 2024-04-24T08:21:21.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:21.950+0000 7f322ae00640 1 -- 172.21.15.146:0/3766607815 wait complete. 2024-04-24T08:21:21.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:21 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:21:21.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:21 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:21.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:21 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:21:21.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:21 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:21.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:21 smithi153 ceph-mon[32746]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:21.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-24T08:21:22.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:21 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:21:22.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:21 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:22.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:21 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:21:22.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:21 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:22.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:21 smithi177 ceph-mon[33046]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:22.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.169+0000 7f8e22400640 1 -- 172.21.15.146:0/262583246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c104470 msgr2=0x7f8e1c104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:22.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.169+0000 7f8e22400640 1 --2- 172.21.15.146:0/262583246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c104470 0x7f8e1c104850 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009be0 tx=0x7f8e0c02a6d0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.170+0000 7f8e22400640 1 -- 172.21.15.146:0/262583246 shutdown_connections 2024-04-24T08:21:22.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.170+0000 7f8e22400640 1 --2- 172.21.15.146:0/262583246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 0x7f8e1c111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.170+0000 7f8e22400640 1 --2- 172.21.15.146:0/262583246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104e20 0x7f8e1c108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.170+0000 7f8e22400640 1 --2- 172.21.15.146:0/262583246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c104470 0x7f8e1c104850 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.170+0000 7f8e22400640 1 -- 172.21.15.146:0/262583246 >> 172.21.15.146:0/262583246 conn(0x7f8e1c0ffcd0 msgr2=0x7f8e1c1020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:22.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.170+0000 7f8e22400640 1 -- 172.21.15.146:0/262583246 shutdown_connections 2024-04-24T08:21:22.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.170+0000 7f8e22400640 1 -- 172.21.15.146:0/262583246 wait complete. 2024-04-24T08:21:22.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.171+0000 7f8e22400640 1 Processor -- start 2024-04-24T08:21:22.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.171+0000 7f8e22400640 1 -- start start 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e22400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104470 0x7f8e1c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e22400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 0x7f8e1c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e22400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c19a860 0x7f8e1c19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e22400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c114600 con 0x7f8e1c104470 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e22400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c114480 con 0x7f8e1c109540 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e22400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c114780 con 0x7f8e1c19a860 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e1b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104470 0x7f8e1c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 0x7f8e1c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e1b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104470 0x7f8e1c1a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47926/0 (socket says 172.21.15.146:47926) 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e1b400640 1 -- 172.21.15.146:0/63077606 learned_addr learned my addr 172.21.15.146:0/63077606 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 0x7f8e1c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:47850/0 (socket says 172.21.15.146:47850) 2024-04-24T08:21:22.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e1be00640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c19a860 0x7f8e1c19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e1b400640 1 -- 172.21.15.146:0/63077606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c19a860 msgr2=0x7f8e1c19ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.172+0000 7f8e1b400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c19a860 0x7f8e1c19ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e1b400640 1 -- 172.21.15.146:0/63077606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 msgr2=0x7f8e1c1a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e1b400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 0x7f8e1c1a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e1b400640 1 -- 172.21.15.146:0/63077606 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c19b380 con 0x7f8e1c104470 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e12a00640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 0x7f8e1c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e1b400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104470 0x7f8e1c1a06b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c02acc0 tx=0x7f8e0c0305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e18c00640 1 -- 172.21.15.146:0/63077606 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c041020 con 0x7f8e1c104470 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e18c00640 1 -- 172.21.15.146:0/63077606 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e0c003e50 con 0x7f8e1c104470 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e18c00640 1 -- 172.21.15.146:0/63077606 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c036040 con 0x7f8e1c104470 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1c19b610 con 0x7f8e1c104470 2024-04-24T08:21:22.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.173+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1c1a7860 con 0x7f8e1c104470 2024-04-24T08:21:22.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.175+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e1c04fd90 con 0x7f8e1c104470 2024-04-24T08:21:22.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.177+0000 7f8e18c00640 1 -- 172.21.15.146:0/63077606 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f8e0c044020 con 0x7f8e1c104470 2024-04-24T08:21:22.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.178+0000 7f8e18c00640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8e000a3f60 0x7f8e000a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:22.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.179+0000 7f8e18c00640 1 -- 172.21.15.146:0/63077606 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8e0c0f1510 con 0x7f8e1c104470 2024-04-24T08:21:22.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.179+0000 7f8e12a00640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8e000a3f60 0x7f8e000a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:22.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.179+0000 7f8e12a00640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8e000a3f60 0x7f8e000a6420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c10aae0 tx=0x7f8e04009260 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:22.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.182+0000 7f8e18c00640 1 -- 172.21.15.146:0/63077606 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e0c04b020 con 0x7f8e1c104470 2024-04-24T08:21:22.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.414+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8e1c10c7e0 con 0x7f8e000a3f60 2024-04-24T08:21:22.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.428+0000 7f8e18c00640 1 -- 172.21.15.146:0/63077606 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8e04005020 con 0x7f8e000a3f60 2024-04-24T08:21:22.430 INFO:teuthology.orchestra.run.smithi146.stdout:/ 1m 6m3h 2024-04-24T08:21:22.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.430+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8e000a3f60 msgr2=0x7f8e000a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:22.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.430+0000 7f8e22400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8e000a3f60 0x7f8e000a6420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c10aae0 tx=0x7f8e04009260 comp rx=0 tx=0).stop 2024-04-24T08:21:22.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.430+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104470 msgr2=0x7f8e1c1a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:22.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104470 0x7f8e1c1a06b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c02acc0 tx=0x7f8e0c0305d0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 shutdown_connections 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8e000a3f60 0x7f8e000a6420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e1c19a860 0x7f8e1c19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8e1c109540 0x7f8e1c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 --2- 172.21.15.146:0/63077606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e1c104470 0x7f8e1c1a06b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 >> 172.21.15.146:0/63077606 conn(0x7f8e1c0ffcd0 msgr2=0x7f8e1c100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 shutdown_connections 2024-04-24T08:21:22.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:22.431+0000 7f8e22400640 1 -- 172.21.15.146:0/63077606 wait complete. 2024-04-24T08:21:22.453 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-24T08:21:22.530 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-24T08:21:22.541 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T08:21:22.622 INFO:tasks.workunit:Pulling workunits from ref 1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:21:22.622 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T08:21:22.622 INFO:tasks.workunit:timeout=3h 2024-04-24T08:21:22.622 INFO:tasks.workunit:cleanup=True 2024-04-24T08:21:22.623 DEBUG:teuthology.orchestra.run.smithi146:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-04-24 08:18:54.509093714 +0000 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-04-24 08:18:54.509093714 +0000 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-04-24 08:21:09.736890596 +0000 2024-04-24T08:21:22.681 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-04-24 08:18:54.509093714 +0000 2024-04-24T08:21:22.681 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:22.681 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-24T08:21:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:22 smithi146 ceph-mon[27805]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 798 B/s wr, 17 op/s 2024-04-24T08:21:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:22 smithi146 ceph-mon[27805]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:22.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:22 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:21:22.785 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:21:22.845 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-24T08:21:22.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:22 smithi153 ceph-mon[32746]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 798 B/s wr, 17 op/s 2024-04-24T08:21:22.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:22 smithi153 ceph-mon[32746]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:22.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:22 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:21:23.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:22 smithi177 ceph-mon[33046]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 798 B/s wr, 17 op/s 2024-04-24T08:21:23.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:22 smithi177 ceph-mon[33046]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:23.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:22 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:21:23.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:23 smithi153 ceph-mon[32746]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:23 smithi153 ceph-mon[32746]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:23.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:23 smithi146 ceph-mon[27805]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:23.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:23 smithi146 ceph-mon[27805]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:24.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:23 smithi177 ceph-mon[33046]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:24.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:23 smithi177 ceph-mon[33046]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:21:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:24 smithi177 ceph-mon[33046]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-04-24T08:21:24.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:24 smithi153 ceph-mon[32746]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-04-24T08:21:24.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:24 smithi146 ceph-mon[27805]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-04-24T08:21:26.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:26 smithi153 ceph-mon[32746]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-24T08:21:26.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:26 smithi146 ceph-mon[27805]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-24T08:21:27.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:26 smithi177 ceph-mon[33046]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-24T08:21:28.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:28 smithi153 ceph-mon[32746]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-24T08:21:28.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:28 smithi146 ceph-mon[27805]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-24T08:21:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:28 smithi177 ceph-mon[33046]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-24T08:21:30.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:30 smithi153 ceph-mon[32746]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-24T08:21:30.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:30 smithi146 ceph-mon[27805]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-24T08:21:31.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:30 smithi177 ceph-mon[33046]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-24T08:21:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:32 smithi153 ceph-mon[32746]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-24T08:21:32.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:32 smithi146 ceph-mon[27805]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-24T08:21:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:32 smithi177 ceph-mon[33046]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-24T08:21:34.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:34 smithi153 ceph-mon[32746]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-24T08:21:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:34 smithi146 ceph-mon[27805]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-24T08:21:35.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:34 smithi177 ceph-mon[33046]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-24T08:21:36.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:36 smithi153 ceph-mon[32746]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-24T08:21:36.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:36 smithi146 ceph-mon[27805]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-24T08:21:37.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:36 smithi177 ceph-mon[33046]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-24T08:21:38.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:38 smithi153 ceph-mon[32746]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-24T08:21:38.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:38 smithi146 ceph-mon[27805]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-24T08:21:39.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:38 smithi177 ceph-mon[33046]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 6 op/s 2024-04-24T08:21:40.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:40 smithi153 ceph-mon[32746]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-24T08:21:40.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:40 smithi146 ceph-mon[27805]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-24T08:21:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:40 smithi177 ceph-mon[33046]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-24T08:21:42.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:42 smithi153 ceph-mon[32746]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:42.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:42 smithi146 ceph-mon[27805]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:43.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:42 smithi177 ceph-mon[33046]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:44.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:44 smithi153 ceph-mon[32746]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:44.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:44 smithi146 ceph-mon[27805]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:45.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:44 smithi177 ceph-mon[33046]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:46.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:46 smithi146 ceph-mon[27805]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:47.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:46 smithi177 ceph-mon[33046]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:47.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:46 smithi153 ceph-mon[32746]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:48.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:48 smithi146 ceph-mon[27805]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:21:49.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:48 smithi177 ceph-mon[33046]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:21:49.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:48 smithi153 ceph-mon[32746]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:21:50.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:21:50.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:21:50.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:50 smithi146 ceph-mon[27805]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 -- 172.21.15.146:0/4251055409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2358102c40 msgr2=0x7f2358106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 --2- 172.21.15.146:0/4251055409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2358102c40 0x7f2358106c80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2344009900 tx=0x7f234402a410 comp rx=0 tx=0).stop 2024-04-24T08:21:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 -- 172.21.15.146:0/4251055409 shutdown_connections 2024-04-24T08:21:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 --2- 172.21.15.146:0/4251055409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2358107360 0x7f235810ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 --2- 172.21.15.146:0/4251055409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2358102c40 0x7f2358106c80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 --2- 172.21.15.146:0/4251055409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358102290 0x7f2358102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 -- 172.21.15.146:0/4251055409 >> 172.21.15.146:0/4251055409 conn(0x7f23580fdb10 msgr2=0x7f23580fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 -- 172.21.15.146:0/4251055409 shutdown_connections 2024-04-24T08:21:51.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.064+0000 7f235dc00640 1 -- 172.21.15.146:0/4251055409 wait complete. 2024-04-24T08:21:51.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:50 smithi177 ceph-mon[33046]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 Processor -- start 2024-04-24T08:21:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 -- start start 2024-04-24T08:21:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2358102290 0x7f23581a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358107360 0x7f23581a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f235819aa30 0x7f235819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23581123d0 con 0x7f235819aa30 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2358112250 con 0x7f2358107360 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.065+0000 7f235dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2358112550 con 0x7f2358102290 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358107360 0x7f23581a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2357400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f235819aa30 0x7f235819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358107360 0x7f23581a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37406/0 (socket says 172.21.15.146:37406) 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 -- 172.21.15.146:0/3644594354 learned_addr learned my addr 172.21.15.146:0/3644594354 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356a00640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2358102290 0x7f23581a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 -- 172.21.15.146:0/3644594354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2358102290 msgr2=0x7f23581a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2358102290 0x7f23581a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 -- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f235819aa30 msgr2=0x7f235819aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f235819aa30 0x7f235819aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2356000640 1 -- 172.21.15.146:0/3644594354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235819b550 con 0x7f2358107360 2024-04-24T08:21:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.066+0000 7f2357400640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f235819aa30 0x7f235819aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.067+0000 7f2356000640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358107360 0x7f23581a0d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2344005940 tx=0x7f23440040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.067+0000 7f234f400640 1 -- 172.21.15.146:0/3644594354 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234403f030 con 0x7f2358107360 2024-04-24T08:21:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.067+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235819b7e0 con 0x7f2358107360 2024-04-24T08:21:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.067+0000 7f234f400640 1 -- 172.21.15.146:0/3644594354 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23440044c0 con 0x7f2358107360 2024-04-24T08:21:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.067+0000 7f234f400640 1 -- 172.21.15.146:0/3644594354 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344035070 con 0x7f2358107360 2024-04-24T08:21:51.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.067+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23581a7930 con 0x7f2358107360 2024-04-24T08:21:51.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.070+0000 7f236096a740 1 Processor -- start 2024-04-24T08:21:51.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.070+0000 7f234f400640 1 -- 172.21.15.146:0/3644594354 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f2344042030 con 0x7f2358107360 2024-04-24T08:21:51.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.070+0000 7f236096a740 1 -- start start 2024-04-24T08:21:51.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f236096a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c17c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f236096a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c22310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f236096a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c24c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f236096a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557679b7d1f0 con 0x557679c17860 2024-04-24T08:21:51.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f236096a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557679b23350 con 0x557679c18180 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f236096a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557679b791b0 con 0x557679c22850 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c22310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2357400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c17c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c22310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37410/0 (socket says 172.21.15.146:37410) 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 -- 172.21.15.146:0/3387122181 learned_addr learned my addr 172.21.15.146:0/3387122181 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356000640 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c24c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 -- 172.21.15.146:0/3387122181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 msgr2=0x557679c24c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c24c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 -- 172.21.15.146:0/3387122181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 msgr2=0x557679c17c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c17c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.071+0000 7f2356a00640 1 -- 172.21.15.146:0/3387122181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557679b77f50 con 0x557679c18180 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.072+0000 7f2356a00640 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c22310 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2348009c90 tx=0x7f2348011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed39fe743ffcaf00 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.072+0000 7f2343e00640 1 -- 172.21.15.146:0/3387122181 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348005220 con 0x557679c18180 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.072+0000 7f2343e00640 1 -- 172.21.15.146:0/3387122181 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2348004c80 con 0x557679c18180 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.072+0000 7f2343e00640 1 -- 172.21.15.146:0/3387122181 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348019830 con 0x557679c18180 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.072+0000 7f234f400640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f233c0a3e40 0x7f233c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.072+0000 7f234f400640 1 -- 172.21.15.146:0/3644594354 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f23440f0db0 con 0x7f2358107360 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f2357400640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f233c0a3e40 0x7f233c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 -- 172.21.15.146:0/3387122181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 msgr2=0x557679c22310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c22310 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2348009c90 tx=0x7f2348011810 comp rx=0 tx=0).stop 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 -- 172.21.15.146:0/3387122181 shutdown_connections 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c24c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c22310 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 --2- 172.21.15.146:0/3387122181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c17c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 -- 172.21.15.146:0/3387122181 >> 172.21.15.146:0/3387122181 conn(0x557679c13b80 msgr2=0x557679c13f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 -- 172.21.15.146:0/3387122181 shutdown_connections 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f236096a740 1 -- 172.21.15.146:0/3387122181 wait complete. 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.073+0000 7f2357400640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f233c0a3e40 0x7f233c0a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2350009fd0 tx=0x7f2350009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 -- start start 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 Processor -- start 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557679c25640 con 0x557679c22850 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557679b36690 con 0x557679c18180 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557679c257c0 con 0x557679c17860 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.074+0000 7f236096a740 10 client.0 _ll_register_callbacks cb 0x557679b72650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:51.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37426/0 (socket says 172.21.15.146:37426) 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2357400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 -- 172.21.15.146:0/3482429689 learned_addr learned my addr 172.21.15.146:0/3482429689 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356a00640 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46506/0 (socket says 172.21.15.146:46506) 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 -- 172.21.15.146:0/3482429689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 msgr2=0x557679c1f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 -- 172.21.15.146:0/3482429689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 msgr2=0x557679c201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c201e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 -- 172.21.15.146:0/3482429689 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557679b35920 con 0x557679c18180 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356a00640 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2356000640 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1fca0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f23440bc930 tx=0x7f23440bc770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4767b3e6b50a026 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.075+0000 7f2340c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f2340c00640 1 -- 172.21.15.146:0/3482429689 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344049050 con 0x557679c18180 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f2340c00640 1 -- 172.21.15.146:0/3482429689 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f23440b92a0 con 0x557679c18180 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2348004c80 con 0x557679c18180 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f2340c00640 1 -- 172.21.15.146:0/3482429689 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f23440a7070 con 0x557679c18180 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f236096a740 10 client.25454 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5576799597d0 con 0x557679c18180 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f2340c00640 1 -- 172.21.15.146:0/3482429689 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2348004c80 con 0x557679c18180 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f2340c00640 1 -- 172.21.15.146:0/3482429689 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f23440bce90 con 0x557679c18180 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f236096a740 10 client.25454 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f236096a740 10 client.25454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f236096a740 20 client.25454 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:51.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.076+0000 7f236096a740 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c329e0 0x557679c34e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.077+0000 7f236096a740 4 client.25454 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.077+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557679b8ac10 con 0x557679c329e0 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.077+0000 7f2356a00640 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c329e0 0x557679c34e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.077+0000 7f2356a00640 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c329e0 0x557679c34e10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=accca9c32e0a99ce server_cookie=8e2529ae1b17c92 in_seq=0 out_seq=0 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.077+0000 7f2340c00640 10 client.25454 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.078+0000 7f2340c00640 1 -- 172.21.15.146:0/3482429689 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557679b8ac10 con 0x557679c329e0 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.078+0000 7f2340c00640 10 client.25454 handle_command_reply: tid=0 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.078+0000 7f236096a740 1 client.25454 shutdown 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 msgr2=0x557679c1fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1fca0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f23440bc930 tx=0x7f23440bc770 comp rx=0 tx=0).stop 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 shutdown_connections 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c329e0 0x557679c34e10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c201e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1fca0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f2340c00640 0 client.25454 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:51.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 --2- 172.21.15.146:0/3482429689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 >> 172.21.15.146:0/3482429689 conn(0x557679c13b80 msgr2=0x557679c19980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 shutdown_connections 2024-04-24T08:21:51.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 1 -- 172.21.15.146:0/3482429689 wait complete. 2024-04-24T08:21:51.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.079+0000 7f236096a740 20 client.25454 trim_cache size 0 max 16384 2024-04-24T08:21:51.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.098+0000 7f236096a740 1 Processor -- start 2024-04-24T08:21:51.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.098+0000 7f236096a740 1 -- start start 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.098+0000 7f236096a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.098+0000 7f236096a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c2e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f236096a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c3a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f236096a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23581b27e0 con 0x557679c22850 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f236096a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23581b00b0 con 0x557679c18180 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f236096a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f23581adf00 con 0x557679c17860 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46510/0 (socket says 172.21.15.146:46510) 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 -- 172.21.15.146:0/190501577 learned_addr learned my addr 172.21.15.146:0/190501577 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356000640 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c2e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 -- 172.21.15.146:0/190501577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 msgr2=0x557679c2e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c2e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2357400640 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c3a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 -- 172.21.15.146:0/190501577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 msgr2=0x557679c3a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c3a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2356a00640 1 -- 172.21.15.146:0/190501577 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55767988f3f0 con 0x557679c17860 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.099+0000 7f2357400640 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c3a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:51.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.100+0000 7f2356a00640 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1e6a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2348000c00 tx=0x7f23480056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=535a2826ba9cec5f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.100+0000 7f2343400640 1 -- 172.21.15.146:0/190501577 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348030070 con 0x557679c17860 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.100+0000 7f2343400640 1 -- 172.21.15.146:0/190501577 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f234802b7f0 con 0x557679c17860 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.100+0000 7f2343400640 1 -- 172.21.15.146:0/190501577 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234802bdb0 con 0x557679c17860 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.100+0000 7f236096a740 1 -- 172.21.15.146:0/190501577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 msgr2=0x557679c1e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.100+0000 7f236096a740 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1e6a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2348000c00 tx=0x7f23480056d0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 -- 172.21.15.146:0/190501577 shutdown_connections 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c22850 0x557679c3a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c2e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 --2- 172.21.15.146:0/190501577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c17860 0x557679c1e6a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 -- 172.21.15.146:0/190501577 >> 172.21.15.146:0/190501577 conn(0x557679c13b80 msgr2=0x557679c1ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 -- 172.21.15.146:0/190501577 shutdown_connections 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 -- 172.21.15.146:0/190501577 wait complete. 2024-04-24T08:21:51.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 -- start start 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.101+0000 7f236096a740 1 Processor -- start 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f236096a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c1cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f236096a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f236096a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c97f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f236096a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f23581ae570 con 0x557679c17860 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f236096a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23581af9e0 con 0x557679c18180 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f236096a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f23581aec40 con 0x557679c22850 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f236096a740 10 client.0 _ll_register_callbacks cb 0x7f2344005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c1cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c1cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35490/0 (socket says 172.21.15.146:35490) 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356a00640 1 -- 172.21.15.146:0/3532348065 learned_addr learned my addr 172.21.15.146:0/3532348065 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37442/0 (socket says 172.21.15.146:37442) 2024-04-24T08:21:51.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2357400640 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c97f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356000640 1 -- 172.21.15.146:0/3532348065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 msgr2=0x557679c97f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356000640 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c97f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356000640 1 -- 172.21.15.146:0/3532348065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 msgr2=0x557679c1cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356000640 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c1cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356000640 1 -- 172.21.15.146:0/3532348065 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55767984f0f0 con 0x557679c18180 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.102+0000 7f2356a00640 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c1cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f2356000640 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1d040 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23440baee0 tx=0x7f23440b8ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fdfa7f5499ae877 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f2340c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f2340c00640 1 -- 172.21.15.146:0/3532348065 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344049070 con 0x557679c18180 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f2340c00640 1 -- 172.21.15.146:0/3532348065 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2344031dd0 con 0x557679c18180 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f234802b7f0 con 0x557679c18180 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f2340c00640 1 -- 172.21.15.146:0/3532348065 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f23440c0f30 con 0x557679c18180 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f236096a740 10 client.25460 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:51.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557679b42810 con 0x557679c18180 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f2340c00640 1 -- 172.21.15.146:0/3532348065 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f234802b7f0 con 0x557679c18180 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f2340c00640 1 -- 172.21.15.146:0/3532348065 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f234403bc30 con 0x557679c18180 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f236096a740 10 client.25460 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f236096a740 10 client.25460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f236096a740 20 client.25460 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.103+0000 7f236096a740 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c99750 0x557679c9bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.104+0000 7f236096a740 4 client.25460 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.104+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557679b84a80 con 0x557679c99750 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.104+0000 7f2356a00640 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c99750 0x557679c9bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.104+0000 7f2356a00640 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c99750 0x557679c9bb70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23d87981a2c10c00 server_cookie=fb377d4e032cf65e in_seq=0 out_seq=0 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.104+0000 7f2340c00640 10 client.25460 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f2340c00640 1 -- 172.21.15.146:0/3532348065 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557679b84a80 con 0x557679c99750 2024-04-24T08:21:51.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f2340c00640 10 client.25460 handle_command_reply: tid=0 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 client.25460 shutdown 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 msgr2=0x557679c1d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1d040 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23440baee0 tx=0x7f23440b8ed0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 shutdown_connections 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557679c99750 0x557679c9bb70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557679c22850 0x557679c97f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557679c18180 0x557679c1d040 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 --2- 172.21.15.146:0/3532348065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557679c17860 0x557679c1cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f2340c00640 0 client.25460 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 >> 172.21.15.146:0/3532348065 conn(0x557679c13b80 msgr2=0x557679c12390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 shutdown_connections 2024-04-24T08:21:51.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 1 -- 172.21.15.146:0/3532348065 wait complete. 2024-04-24T08:21:51.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.105+0000 7f236096a740 20 client.25460 trim_cache size 0 max 16384 2024-04-24T08:21:51.111 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:21:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:21:51.111 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:21:51.111 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:21:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f233c0a3e40 msgr2=0x7f233c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f233c0a3e40 0x7f233c0a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2350009fd0 tx=0x7f2350009500 comp rx=0 tx=0).stop 2024-04-24T08:21:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358107360 msgr2=0x7f23581a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358107360 0x7f23581a0d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2344005940 tx=0x7f23440040c0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 shutdown_connections 2024-04-24T08:21:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f233c0a3e40 0x7f233c0a6300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f235819aa30 0x7f235819aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2358107360 0x7f23581a0d20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 --2- 172.21.15.146:0/3644594354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2358102290 0x7f23581a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 >> 172.21.15.146:0/3644594354 conn(0x7f23580fdb10 msgr2=0x7f23580fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 shutdown_connections 2024-04-24T08:21:51.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.106+0000 7f235dc00640 1 -- 172.21.15.146:0/3644594354 wait complete. 2024-04-24T08:21:51.125 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:21:51.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:51.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:50 smithi153 ceph-mon[32746]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 -- 172.21.15.146:0/4116879454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 msgr2=0x7faefc069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 --2- 172.21.15.146:0/4116879454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 0x7faefc069190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faeec009900 tx=0x7faeec02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 -- 172.21.15.146:0/4116879454 shutdown_connections 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 --2- 172.21.15.146:0/4116879454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faefc10e660 0x7faefc110a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 --2- 172.21.15.146:0/4116879454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 0x7faefc069b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 --2- 172.21.15.146:0/4116879454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 0x7faefc069190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 -- 172.21.15.146:0/4116879454 >> 172.21.15.146:0/4116879454 conn(0x7faefc076840 msgr2=0x7faefc076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 -- 172.21.15.146:0/4116879454 shutdown_connections 2024-04-24T08:21:51.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.323+0000 7faf03000640 1 -- 172.21.15.146:0/4116879454 wait complete. 2024-04-24T08:21:51.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.324+0000 7faf03000640 1 Processor -- start 2024-04-24T08:21:51.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf03000640 1 -- start start 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf03000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 0x7faefc1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf03000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 0x7faefc1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf03000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faefc10e660 0x7faefc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf03000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faefc1143f0 con 0x7faefc0696d0 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf03000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faefc114270 con 0x7faefc102d00 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf03000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faefc114570 con 0x7faefc10e660 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf01c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 0x7faefc1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf01c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 0x7faefc1a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35510/0 (socket says 172.21.15.146:35510) 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf01c00640 1 -- 172.21.15.146:0/1864440763 learned_addr learned my addr 172.21.15.146:0/1864440763 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf02600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faefc10e660 0x7faefc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.325+0000 7faf01200640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 0x7faefc1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.326+0000 7faf01c00640 1 -- 172.21.15.146:0/1864440763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faefc10e660 msgr2=0x7faefc19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.326+0000 7faf01c00640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faefc10e660 0x7faefc19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.326+0000 7faf01c00640 1 -- 172.21.15.146:0/1864440763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 msgr2=0x7faefc1a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.326+0000 7faf01c00640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 0x7faefc1a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.326+0000 7faf01c00640 1 -- 172.21.15.146:0/1864440763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc19aff0 con 0x7faefc0696d0 2024-04-24T08:21:51.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.326+0000 7faf01200640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 0x7faefc1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:51.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.326+0000 7faf01c00640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 0x7faefc1a0620 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faeec004770 tx=0x7faeec003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.327+0000 7faefa000640 1 -- 172.21.15.146:0/1864440763 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec03f070 con 0x7faefc0696d0 2024-04-24T08:21:51.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.327+0000 7faefa000640 1 -- 172.21.15.146:0/1864440763 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeec031ca0 con 0x7faefc0696d0 2024-04-24T08:21:51.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.327+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faefc19b280 con 0x7faefc0696d0 2024-04-24T08:21:51.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.327+0000 7faefa000640 1 -- 172.21.15.146:0/1864440763 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec035030 con 0x7faefc0696d0 2024-04-24T08:21:51.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.327+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faefc1a73c0 con 0x7faefc0696d0 2024-04-24T08:21:51.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.328+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faefc101370 con 0x7faefc0696d0 2024-04-24T08:21:51.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.330+0000 7faefa000640 1 -- 172.21.15.146:0/1864440763 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7faeec042070 con 0x7faefc0696d0 2024-04-24T08:21:51.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.332+0000 7faefa000640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3f90 0x7faee40a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.332+0000 7faefa000640 1 -- 172.21.15.146:0/1864440763 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faeec0f0bb0 con 0x7faefc0696d0 2024-04-24T08:21:51.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.332+0000 7faf01200640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3f90 0x7faee40a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.333+0000 7faf01200640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3f90 0x7faee40a6450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faef00095f0 tx=0x7faef0005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.337+0000 7faefa000640 1 -- 172.21.15.146:0/1864440763 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faeec0bcaf0 con 0x7faefc0696d0 2024-04-24T08:21:51.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.582+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faefc068d40 con 0x7faefc0696d0 2024-04-24T08:21:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.584+0000 7faefa000640 1 -- 172.21.15.146:0/1864440763 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11891 (secure 0 0 0) 0x7faeec0c0df0 con 0x7faefc0696d0 2024-04-24T08:21:51.585 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:21:51.585 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:21:10.741287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24553,24568]},"id":1}]} 2024-04-24T08:21:51.585 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 24 2024-04-24T08:21:51.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.586+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3f90 msgr2=0x7faee40a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.586+0000 7faf03000640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3f90 0x7faee40a6450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faef00095f0 tx=0x7faef0005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.586+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 msgr2=0x7faefc1a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.586+0000 7faf03000640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 0x7faefc1a0620 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faeec004770 tx=0x7faeec003c10 comp rx=0 tx=0).stop 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 shutdown_connections 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3f90 0x7faee40a6450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faefc10e660 0x7faefc19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faefc102d00 0x7faefc1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 --2- 172.21.15.146:0/1864440763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faefc0696d0 0x7faefc1a0620 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 >> 172.21.15.146:0/1864440763 conn(0x7faefc076840 msgr2=0x7faefc0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 shutdown_connections 2024-04-24T08:21:51.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.587+0000 7faf03000640 1 -- 172.21.15.146:0/1864440763 wait complete. 2024-04-24T08:21:51.607 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:21:51.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.816+0000 7f3263e00640 1 -- 172.21.15.146:0/2260355596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101250 msgr2=0x7f325c1016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.816+0000 7f3263e00640 1 --2- 172.21.15.146:0/2260355596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101250 0x7f325c1016b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3248009900 tx=0x7f324802a410 comp rx=0 tx=0).stop 2024-04-24T08:21:51.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.816+0000 7f3263e00640 1 -- 172.21.15.146:0/2260355596 shutdown_connections 2024-04-24T08:21:51.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.816+0000 7f3263e00640 1 --2- 172.21.15.146:0/2260355596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101250 0x7f325c1016b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.816+0000 7f3263e00640 1 --2- 172.21.15.146:0/2260355596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f325c101bf0 0x7f325c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.816+0000 7f3263e00640 1 --2- 172.21.15.146:0/2260355596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c1070e0 0x7f325c1074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.817+0000 7f3263e00640 1 -- 172.21.15.146:0/2260355596 >> 172.21.15.146:0/2260355596 conn(0x7f325c076d00 msgr2=0x7f325c077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.817+0000 7f3263e00640 1 -- 172.21.15.146:0/2260355596 shutdown_connections 2024-04-24T08:21:51.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.817+0000 7f3263e00640 1 -- 172.21.15.146:0/2260355596 wait complete. 2024-04-24T08:21:51.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.818+0000 7f3263e00640 1 Processor -- start 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.818+0000 7f3263e00640 1 -- start start 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.818+0000 7f3263e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c101250 0x7f325c102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3263e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101bf0 0x7f325c103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3263e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f325c1070e0 0x7f325c103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3263e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f325c1112e0 con 0x7f325c101250 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3263e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f325c111160 con 0x7f325c1070e0 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3263e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f325c111460 con 0x7f325c101bf0 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c101250 0x7f325c102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c101250 0x7f325c102c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35522/0 (socket says 172.21.15.146:35522) 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 -- 172.21.15.146:0/2317709989 learned_addr learned my addr 172.21.15.146:0/2317709989 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261e00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f325c1070e0 0x7f325c103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3260a00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101bf0 0x7f325c103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 -- 172.21.15.146:0/2317709989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101bf0 msgr2=0x7f325c103140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101bf0 0x7f325c103140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 -- 172.21.15.146:0/2317709989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f325c1070e0 msgr2=0x7f325c103680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f325c1070e0 0x7f325c103680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.819+0000 7f3261400640 1 -- 172.21.15.146:0/2317709989 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c1a6e50 con 0x7f325c101250 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.820+0000 7f3261400640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c101250 0x7f325c102c00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f325000e9f0 tx=0x7f325000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.820+0000 7f325a000640 1 -- 172.21.15.146:0/2317709989 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325000cd80 con 0x7f325c101250 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.820+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f325c1a7140 con 0x7f325c101250 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.820+0000 7f325a000640 1 -- 172.21.15.146:0/2317709989 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3250010910 con 0x7f325c101250 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.820+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f325c1a76e0 con 0x7f325c101250 2024-04-24T08:21:51.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.820+0000 7f325a000640 1 -- 172.21.15.146:0/2317709989 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250018910 con 0x7f325c101250 2024-04-24T08:21:51.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.824+0000 7f3266bb0740 1 Processor -- start 2024-04-24T08:21:51.825 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.824+0000 7f3266bb0740 1 -- start start 2024-04-24T08:21:51.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db78d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db83400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db83940 0x55687db85d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3266bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55687dade1d0 con 0x55687db78950 2024-04-24T08:21:51.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3266bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55687da84450 con 0x55687db83940 2024-04-24T08:21:51.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3266bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55687dada190 con 0x55687db79270 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f325a000640 1 -- 172.21.15.146:0/2317709989 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f3250010ac0 con 0x7f325c101250 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3261e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db78d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3261e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db78d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35530/0 (socket says 172.21.15.146:35530) 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3261e00640 1 -- 172.21.15.146:0/1078705578 learned_addr learned my addr 172.21.15.146:0/1078705578 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3260a00640 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db83940 0x55687db85d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.825+0000 7f3261400640 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db83400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f3261e00640 1 -- 172.21.15.146:0/1078705578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 msgr2=0x55687db83400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f3261e00640 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db83400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f3261e00640 1 -- 172.21.15.146:0/1078705578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db83940 msgr2=0x55687db85d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f3261e00640 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db83940 0x55687db85d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f3261e00640 1 -- 172.21.15.146:0/1078705578 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55687db41310 con 0x55687db78950 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f3261e00640 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db78d30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3254009fd0 tx=0x7f32540118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985b64694ecf6ad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f324e600640 1 -- 172.21.15.146:0/1078705578 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254004d20 con 0x55687db78950 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f324e600640 1 -- 172.21.15.146:0/1078705578 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3254004050 con 0x55687db78950 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.826+0000 7f324e600640 1 -- 172.21.15.146:0/1078705578 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254005220 con 0x55687db78950 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f325a000640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32440a3e40 0x7f32440a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3266bb0740 1 -- 172.21.15.146:0/1078705578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 msgr2=0x55687db78d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db78d30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3254009fd0 tx=0x7f32540118d0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3266bb0740 1 -- 172.21.15.146:0/1078705578 shutdown_connections 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db83400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db83940 0x55687db85d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1078705578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db78d30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3266bb0740 1 -- 172.21.15.146:0/1078705578 >> 172.21.15.146:0/1078705578 conn(0x55687db74c70 msgr2=0x55687db75050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f325a000640 1 -- 172.21.15.146:0/2317709989 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f32500d0ee0 con 0x7f325c101250 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.827+0000 7f3261400640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32440a3e40 0x7f32440a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.828+0000 7f3266bb0740 1 -- 172.21.15.146:0/1078705578 shutdown_connections 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.828+0000 7f3266bb0740 1 -- 172.21.15.146:0/1078705578 wait complete. 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.828+0000 7f3261400640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32440a3e40 0x7f32440a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32500ce4c0 tx=0x7f325009ca00 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.828+0000 7f3266bb0740 1 -- start start 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.828+0000 7f3266bb0740 1 Processor -- start 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db83370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db80660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db80da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3266bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55687db867d0 con 0x55687db78950 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3266bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55687da97810 con 0x55687db79270 2024-04-24T08:21:51.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3266bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55687db86950 con 0x55687db83940 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3266bb0740 10 client.0 _ll_register_callbacks cb 0x55687da65140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3261e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db83370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3261e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db83370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35544/0 (socket says 172.21.15.146:35544) 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3261e00640 1 -- 172.21.15.146:0/514955923 learned_addr learned my addr 172.21.15.146:0/514955923 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3260a00640 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db80660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3261e00640 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db80da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3261e00640 1 -- 172.21.15.146:0/514955923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 msgr2=0x55687db80da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3261e00640 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db80da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.829+0000 7f3261e00640 1 -- 172.21.15.146:0/514955923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 msgr2=0x55687db80660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3261e00640 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db80660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3261e00640 1 -- 172.21.15.146:0/514955923 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55687daa83f0 con 0x55687db78950 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3261e00640 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db83370 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f325401b040 tx=0x7f3254009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c24800490105170 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3242a00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:51.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3242a00640 1 -- 172.21.15.146:0/514955923 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254004780 con 0x55687db78950 2024-04-24T08:21:51.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3242a00640 1 -- 172.21.15.146:0/514955923 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3254005040 con 0x55687db78950 2024-04-24T08:21:51.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3254004050 con 0x55687db78950 2024-04-24T08:21:51.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3242a00640 1 -- 172.21.15.146:0/514955923 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3254035050 con 0x55687db78950 2024-04-24T08:21:51.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3266bb0740 10 client.15348 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:51.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.830+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55687d7f03f0 con 0x55687db78950 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3242a00640 1 -- 172.21.15.146:0/514955923 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3254004050 con 0x55687db78950 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3242a00640 1 -- 172.21.15.146:0/514955923 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f325400f8c0 con 0x55687db78950 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3266bb0740 10 client.15348 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3266bb0740 10 client.15348 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3266bb0740 20 client.15348 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3266bb0740 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687db93a80 0x55687db95e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3266bb0740 4 client.15348 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55687daebbf0 con 0x55687db93a80 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.831+0000 7f3260a00640 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687db93a80 0x55687db95e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.832+0000 7f3260a00640 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687db93a80 0x55687db95e70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=931fd996d38f8d1b server_cookie=8e4ffa1f101e877b in_seq=0 out_seq=0 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.832+0000 7f3242a00640 10 client.15348 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:21:51.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3242a00640 1 -- 172.21.15.146:0/514955923 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55687daebbf0 con 0x55687db93a80 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3242a00640 10 client.15348 handle_command_reply: tid=0 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 client.15348 shutdown 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 msgr2=0x55687db83370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db83370 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f325401b040 tx=0x7f3254009ca0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 shutdown_connections 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687db93a80 0x55687db95e70 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db80da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db80660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 --2- 172.21.15.146:0/514955923 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db83370 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3242a00640 0 client.15348 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 >> 172.21.15.146:0/514955923 conn(0x55687db74c70 msgr2=0x55687db7aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 shutdown_connections 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 1 -- 172.21.15.146:0/514955923 wait complete. 2024-04-24T08:21:51.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.833+0000 7f3266bb0740 20 client.15348 trim_cache size 0 max 16384 2024-04-24T08:21:51.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 Processor -- start 2024-04-24T08:21:51.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 -- start start 2024-04-24T08:21:51.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db78950 0x55687db74570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db82ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db83940 0x55687db833e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f325c1b28e0 con 0x55687db83940 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f325c1b01b0 con 0x55687db78950 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.848+0000 7f3266bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f325c1ae000 con 0x55687db79270 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db83940 0x55687db833e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db83940 0x55687db833e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35558/0 (socket says 172.21.15.146:35558) 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 -- 172.21.15.146:0/3395222737 learned_addr learned my addr 172.21.15.146:0/3395222737 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db78950 0x55687db74570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3260a00640 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db82ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 -- 172.21.15.146:0/3395222737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 msgr2=0x55687db82ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db82ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 -- 172.21.15.146:0/3395222737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db78950 msgr2=0x55687db74570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db78950 0x55687db74570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.849+0000 7f3261e00640 1 -- 172.21.15.146:0/3395222737 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55687d8ba7d0 con 0x55687db83940 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3261e00640 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db83940 0x55687db833e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3254004f50 tx=0x7f3254014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39b650f4fa59ef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f324dc00640 1 -- 172.21.15.146:0/3395222737 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254004580 con 0x55687db83940 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f324dc00640 1 -- 172.21.15.146:0/3395222737 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325402c070 con 0x55687db83940 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f324dc00640 1 -- 172.21.15.146:0/3395222737 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254005330 con 0x55687db83940 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3266bb0740 1 -- 172.21.15.146:0/3395222737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db83940 msgr2=0x55687db833e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3266bb0740 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db83940 0x55687db833e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3254004f50 tx=0x7f3254014040 comp rx=0 tx=0).stop 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3266bb0740 1 -- 172.21.15.146:0/3395222737 shutdown_connections 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3266bb0740 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db79270 0x55687db82ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3266bb0740 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db83940 0x55687db833e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3266bb0740 1 --2- 172.21.15.146:0/3395222737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db78950 0x55687db74570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.850+0000 7f3266bb0740 1 -- 172.21.15.146:0/3395222737 >> 172.21.15.146:0/3395222737 conn(0x55687db74c70 msgr2=0x55687db7f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.851+0000 7f3266bb0740 1 -- 172.21.15.146:0/3395222737 shutdown_connections 2024-04-24T08:21:51.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.851+0000 7f3266bb0740 1 -- 172.21.15.146:0/3395222737 wait complete. 2024-04-24T08:21:51.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.851+0000 7f3266bb0740 1 -- start start 2024-04-24T08:21:51.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.851+0000 7f3266bb0740 1 Processor -- start 2024-04-24T08:21:51.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db846f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db84c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3266bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db85170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3266bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f325c1ae670 con 0x55687db78950 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3266bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f325c1afae0 con 0x55687db79270 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3266bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f325c1aed40 con 0x55687db83940 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3266bb0740 10 client.0 _ll_register_callbacks cb 0x7f3250005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db84c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3261e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db846f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db84c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37514/0 (socket says 172.21.15.146:37514) 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 -- 172.21.15.146:0/1248022218 learned_addr learned my addr 172.21.15.146:0/1248022218 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3261e00640 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db85170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 -- 172.21.15.146:0/1248022218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 msgr2=0x55687db85170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db85170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 -- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 msgr2=0x55687db846f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db846f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3260a00640 1 -- 172.21.15.146:0/1248022218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55687db89e70 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.852+0000 7f3261e00640 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db846f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3260a00640 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db84c30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3248005940 tx=0x7f32480042c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7319b824eeabe801 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3242a00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3242a00640 1 -- 172.21.15.146:0/1248022218 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3248035040 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3242a00640 1 -- 172.21.15.146:0/1248022218 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3248005040 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f325402c070 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3242a00640 1 -- 172.21.15.146:0/1248022218 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3248005240 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3266bb0740 10 client.25484 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.853+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55687dbfa1d0 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3242a00640 1 -- 172.21.15.146:0/1248022218 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f325402c070 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3242a00640 1 -- 172.21.15.146:0/1248022218 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3248004750 con 0x55687db79270 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3266bb0740 10 client.25484 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3266bb0740 10 client.25484 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:21:51.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3266bb0740 20 client.25484 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687dbfa8d0 0x55687dbfccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3266bb0740 4 client.25484 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55687dae5a60 con 0x55687dbfa8d0 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.854+0000 7f3261e00640 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687dbfa8d0 0x55687dbfccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3261e00640 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687dbfa8d0 0x55687dbfccf0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3cb9b6aa966387e server_cookie=c80be756ead5c1d6 in_seq=0 out_seq=0 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3242a00640 10 client.25484 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3242a00640 1 -- 172.21.15.146:0/1248022218 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55687dae5a60 con 0x55687dbfa8d0 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3242a00640 10 client.25484 handle_command_reply: tid=0 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 client.25484 shutdown 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 msgr2=0x55687db84c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db84c30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3248005940 tx=0x7f32480042c0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 shutdown_connections 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55687dbfa8d0 0x55687dbfccf0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55687db83940 0x55687db85170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55687db79270 0x55687db84c30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3242a00640 0 client.25484 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:21:51.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 --2- 172.21.15.146:0/1248022218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55687db78950 0x55687db846f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.855+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 >> 172.21.15.146:0/1248022218 conn(0x55687db74c70 msgr2=0x55687db7a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 shutdown_connections 2024-04-24T08:21:51.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3266bb0740 1 -- 172.21.15.146:0/1248022218 wait complete. 2024-04-24T08:21:51.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3266bb0740 20 client.25484 trim_cache size 0 max 16384 2024-04-24T08:21:51.860 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32440a3e40 msgr2=0x7f32440a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32440a3e40 0x7f32440a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32500ce4c0 tx=0x7f325009ca00 comp rx=0 tx=0).stop 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c101250 msgr2=0x7f325c102c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c101250 0x7f325c102c00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f325000e9f0 tx=0x7f325000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 shutdown_connections 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32440a3e40 0x7f32440a6300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f325c101bf0 0x7f325c103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f325c1070e0 0x7f325c103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 --2- 172.21.15.146:0/2317709989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f325c101250 0x7f325c102c00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:51.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.856+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 >> 172.21.15.146:0/2317709989 conn(0x7f325c076d00 msgr2=0x7f325c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:51.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.857+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 shutdown_connections 2024-04-24T08:21:51.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:51.857+0000 7f3263e00640 1 -- 172.21.15.146:0/2317709989 wait complete. 2024-04-24T08:21:51.872 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:51 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1864440763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:51.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:21:52.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:51 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1864440763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 -- 172.21.15.146:0/4089582794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 msgr2=0x7f98301153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 --2- 172.21.15.146:0/4089582794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 0x7f98301153b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9828009cd0 tx=0x7f982802a8e0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 -- 172.21.15.146:0/4089582794 shutdown_connections 2024-04-24T08:21:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 --2- 172.21.15.146:0/4089582794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 0x7f98301153b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 --2- 172.21.15.146:0/4089582794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9830108580 0x7f9830108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 --2- 172.21.15.146:0/4089582794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983010d500 0x7f983010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 -- 172.21.15.146:0/4089582794 >> 172.21.15.146:0/4089582794 conn(0x7f9830077e20 msgr2=0x7f9830106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 -- 172.21.15.146:0/4089582794 shutdown_connections 2024-04-24T08:21:52.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.073+0000 7f9834c00640 1 -- 172.21.15.146:0/4089582794 wait complete. 2024-04-24T08:21:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.074+0000 7f9834c00640 1 Processor -- start 2024-04-24T08:21:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.074+0000 7f9834c00640 1 -- start start 2024-04-24T08:21:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f9834c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9830108580 0x7f983019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f9834c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 0x7f983019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f9834c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983010d500 0x7f98301a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f9834c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9830118900 con 0x7f9830108580 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f9834c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9830118780 con 0x7f983010d500 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f9834c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9830118a80 con 0x7f9830108f40 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9830108580 0x7f983019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f9827e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 0x7f983019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9830108580 0x7f983019ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35582/0 (socket says 172.21.15.146:35582) 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f982f400640 1 -- 172.21.15.146:0/793611958 learned_addr learned my addr 172.21.15.146:0/793611958 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.075+0000 7f982fe00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983010d500 0x7f98301a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9827e00640 1 -- 172.21.15.146:0/793611958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983010d500 msgr2=0x7f98301a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9827e00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983010d500 0x7f98301a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9827e00640 1 -- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9830108580 msgr2=0x7f983019ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9827e00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9830108580 0x7f983019ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9827e00640 1 -- 172.21.15.146:0/793611958 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98301a3d30 con 0x7f9830108f40 2024-04-24T08:21:52.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f982fe00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983010d500 0x7f98301a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:52.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9827e00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 0x7f983019f280 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f981800e9f0 tx=0x7f981800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f982cc00640 1 -- 172.21.15.146:0/793611958 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981800ccc0 con 0x7f9830108f40 2024-04-24T08:21:52.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98301a4020 con 0x7f9830108f40 2024-04-24T08:21:52.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f982cc00640 1 -- 172.21.15.146:0/793611958 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9818010910 con 0x7f9830108f40 2024-04-24T08:21:52.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f982cc00640 1 -- 172.21.15.146:0/793611958 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9818018960 con 0x7f9830108f40 2024-04-24T08:21:52.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.076+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98301ab8b0 con 0x7f9830108f40 2024-04-24T08:21:52.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.079+0000 7f9839015740 1 Processor -- start 2024-04-24T08:21:52.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.079+0000 7f9839015740 1 -- start start 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f982cc00640 1 -- 172.21.15.146:0/793611958 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f9818010ac0 con 0x7f9830108f40 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f9839015740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb30f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f9839015740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f9839015740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f9839015740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555c8fa961d0 con 0x555c8fb30b40 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f9839015740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c8fa3c450 con 0x555c8fb3bb30 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f9839015740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555c8fa92190 con 0x555c8fb31460 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f982fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb30f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f9827e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f982f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f982fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb30f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35586/0 (socket says 172.21.15.146:35586) 2024-04-24T08:21:52.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.080+0000 7f982fe00640 1 -- 172.21.15.146:0/2934333489 learned_addr learned my addr 172.21.15.146:0/2934333489 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982fe00640 1 -- 172.21.15.146:0/2934333489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 msgr2=0x555c8fb3b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982fe00640 1 --2- 172.21.15.146:0/2934333489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982fe00640 1 -- 172.21.15.146:0/2934333489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 msgr2=0x555c8fb3df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982fe00640 1 --2- 172.21.15.146:0/2934333489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982fe00640 1 -- 172.21.15.146:0/2934333489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c8faf9310 con 0x555c8fb30b40 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982fe00640 1 --2- 172.21.15.146:0/2934333489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb30f20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9828009cd0 tx=0x7f9828009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e230a6a8c75c205b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f981fe00640 1 -- 172.21.15.146:0/2934333489 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828048070 con 0x555c8fb30b40 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f981fe00640 1 -- 172.21.15.146:0/2934333489 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9828036040 con 0x555c8fb30b40 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f981fe00640 1 -- 172.21.15.146:0/2934333489 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98280042a0 con 0x555c8fb30b40 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982cc00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98140a3ec0 0x7f98140a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.081+0000 7f982cc00640 1 -- 172.21.15.146:0/793611958 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f98180d0e30 con 0x7f9830108f40 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f982f400640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98140a3ec0 0x7f98140a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 -- 172.21.15.146:0/2934333489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 msgr2=0x555c8fb30f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 --2- 172.21.15.146:0/2934333489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb30f20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9828009cd0 tx=0x7f9828009a20 comp rx=0 tx=0).stop 2024-04-24T08:21:52.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 -- 172.21.15.146:0/2934333489 shutdown_connections 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 --2- 172.21.15.146:0/2934333489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 --2- 172.21.15.146:0/2934333489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 --2- 172.21.15.146:0/2934333489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb30f20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 -- 172.21.15.146:0/2934333489 >> 172.21.15.146:0/2934333489 conn(0x555c8fb2ce60 msgr2=0x555c8fb2d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 -- 172.21.15.146:0/2934333489 shutdown_connections 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f9839015740 1 -- 172.21.15.146:0/2934333489 wait complete. 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.082+0000 7f982f400640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98140a3ec0 0x7f98140a6380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x555c8fb32100 tx=0x7f982000b4a0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 -- start start 2024-04-24T08:21:52.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 Processor -- start 2024-04-24T08:21:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb3b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb38fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555c8fb3e960 con 0x555c8fb30b40 2024-04-24T08:21:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555c8fa4f810 con 0x555c8fb3bb30 2024-04-24T08:21:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.083+0000 7f9839015740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555c8fb3eae0 con 0x555c8fb31460 2024-04-24T08:21:52.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f9839015740 10 client.0 _ll_register_callbacks cb 0x555c8fa1d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb3b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb3b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35596/0 (socket says 172.21.15.146:35596) 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 -- 172.21.15.146:0/631007652 learned_addr learned my addr 172.21.15.146:0/631007652 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982fe00640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb38fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f9827e00640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 -- 172.21.15.146:0/631007652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 msgr2=0x555c8fb388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 -- 172.21.15.146:0/631007652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 msgr2=0x555c8fb38fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb38fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982f400640 1 -- 172.21.15.146:0/631007652 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c8fa603f0 con 0x555c8fb30b40 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f9827e00640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.084+0000 7f982fe00640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb38fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:52.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f982f400640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb3b5b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f982001f670 tx=0x7f9820004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934eac9457104312 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f981cc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f981cc00640 1 -- 172.21.15.146:0/631007652 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982002c060 con 0x555c8fb30b40 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f981cc00640 1 -- 172.21.15.146:0/631007652 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9820017030 con 0x555c8fb30b40 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9828036040 con 0x555c8fb30b40 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f981cc00640 1 -- 172.21.15.146:0/631007652 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9820010780 con 0x555c8fb30b40 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f9839015740 10 client.15366 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c8f7a83f0 con 0x555c8fb30b40 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.085+0000 7f981cc00640 1 -- 172.21.15.146:0/631007652 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9828036040 con 0x555c8fb30b40 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f981cc00640 1 -- 172.21.15.146:0/631007652 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f982001cbe0 con 0x555c8fb30b40 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f9839015740 10 client.15366 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f9839015740 10 client.15366 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f9839015740 20 client.15366 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f9839015740 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb4bd90 0x555c8fb4e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f9839015740 4 client.15366 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:52.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c8faa3bf0 con 0x555c8fb4bd90 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.086+0000 7f9827e00640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb4bd90 0x555c8fb4e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.087+0000 7f981cc00640 10 client.15366 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.087+0000 7f9827e00640 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb4bd90 0x555c8fb4e1e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0943e2f26af6d06 server_cookie=81beca54a2b20a8e in_seq=0 out_seq=0 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.087+0000 7f981cc00640 1 -- 172.21.15.146:0/631007652 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c8faa3bf0 con 0x555c8fb4bd90 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.087+0000 7f981cc00640 10 client.15366 handle_command_reply: tid=0 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.087+0000 7f9839015740 1 client.15366 shutdown 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 msgr2=0x555c8fb3b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb3b5b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f982001f670 tx=0x7f9820004730 comp rx=0 tx=0).stop 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 shutdown_connections 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb4bd90 0x555c8fb4e1e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb38fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 --2- 172.21.15.146:0/631007652 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb3b5b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 >> 172.21.15.146:0/631007652 conn(0x555c8fb2ce60 msgr2=0x555c8fb32c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f981cc00640 0 client.15366 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:21:52.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 shutdown_connections 2024-04-24T08:21:52.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 1 -- 172.21.15.146:0/631007652 wait complete. 2024-04-24T08:21:52.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.088+0000 7f9839015740 20 client.15366 trim_cache size 0 max 16384 2024-04-24T08:21:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 Processor -- start 2024-04-24T08:21:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 -- start start 2024-04-24T08:21:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb47820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb31460 0x555c8fb38050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb3bb30 0x555c8fb32ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f98301b69d0 con 0x555c8fb30b40 2024-04-24T08:21:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98301b42a0 con 0x555c8fb31460 2024-04-24T08:21:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f9839015740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98301b20f0 con 0x555c8fb3bb30 2024-04-24T08:21:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb47820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.103+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb47820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35612/0 (socket says 172.21.15.146:35612) 2024-04-24T08:21:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982f400640 1 -- 172.21.15.146:0/3572037275 learned_addr learned my addr 172.21.15.146:0/3572037275 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f9827e00640 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb31460 0x555c8fb38050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982fe00640 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb3bb30 0x555c8fb32ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982f400640 1 -- 172.21.15.146:0/3572037275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb3bb30 msgr2=0x555c8fb32ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982f400640 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb3bb30 0x555c8fb32ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982f400640 1 -- 172.21.15.146:0/3572037275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb31460 msgr2=0x555c8fb38050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982f400640 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb31460 0x555c8fb38050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982f400640 1 -- 172.21.15.146:0/3572037275 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c8f8727d0 con 0x555c8fb30b40 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982fe00640 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb3bb30 0x555c8fb32ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f982f400640 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb47820 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f982001b7e0 tx=0x7f982001b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20415ff8c7028bb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f981f400640 1 -- 172.21.15.146:0/3572037275 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982001f240 con 0x555c8fb30b40 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f981f400640 1 -- 172.21.15.146:0/3572037275 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f982001be10 con 0x555c8fb30b40 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.104+0000 7f981f400640 1 -- 172.21.15.146:0/3572037275 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982001c650 con 0x555c8fb30b40 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 -- 172.21.15.146:0/3572037275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 msgr2=0x555c8fb47820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb47820 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f982001b7e0 tx=0x7f982001b810 comp rx=0 tx=0).stop 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 -- 172.21.15.146:0/3572037275 shutdown_connections 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb3bb30 0x555c8fb32ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb31460 0x555c8fb38050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 --2- 172.21.15.146:0/3572037275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb47820 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 -- 172.21.15.146:0/3572037275 >> 172.21.15.146:0/3572037275 conn(0x555c8fb2ce60 msgr2=0x555c8fb323a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 -- 172.21.15.146:0/3572037275 shutdown_connections 2024-04-24T08:21:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.105+0000 7f9839015740 1 -- 172.21.15.146:0/3572037275 wait complete. 2024-04-24T08:21:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 -- start start 2024-04-24T08:21:52.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 Processor -- start 2024-04-24T08:21:52.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb43fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f98301b2760 con 0x555c8fb30b40 2024-04-24T08:21:52.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98301b3bd0 con 0x555c8fb3bb30 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98301b2e30 con 0x555c8fb31460 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.106+0000 7f9839015740 10 client.0 _ll_register_callbacks cb 0x7f982000bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb43fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb43fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35624/0 (socket says 172.21.15.146:35624) 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 -- 172.21.15.146:0/3725806130 learned_addr learned my addr 172.21.15.146:0/3725806130 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f9827e00640 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982fe00640 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 -- 172.21.15.146:0/3725806130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 msgr2=0x555c8fb3d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 -- 172.21.15.146:0/3725806130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 msgr2=0x555c8fb3d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 -- 172.21.15.146:0/3725806130 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c8fb3bf10 con 0x555c8fb30b40 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f982f400640 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb43fe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9820018040 tx=0x7f9820005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb67919aab8c8b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.107+0000 7f981cc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f981cc00640 1 -- 172.21.15.146:0/3725806130 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982003f030 con 0x555c8fb30b40 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f981cc00640 1 -- 172.21.15.146:0/3725806130 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f98200109e0 con 0x555c8fb30b40 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f982001be10 con 0x555c8fb30b40 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f981cc00640 1 -- 172.21.15.146:0/3725806130 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9820049050 con 0x555c8fb30b40 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 10 client.15372 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555c8fb42fe0 con 0x555c8fb30b40 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f981cc00640 1 -- 172.21.15.146:0/3725806130 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f982001be10 con 0x555c8fb30b40 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f981cc00640 1 -- 172.21.15.146:0/3725806130 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f982001c7f0 con 0x555c8fb30b40 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 10 client.15372 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 10 client.15372 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 20 client.15372 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb437e0 0x555c8fbb4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 4 client.15372 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.108+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c8fa9da60 con 0x555c8fb437e0 2024-04-24T08:21:52.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.109+0000 7f9827e00640 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb437e0 0x555c8fbb4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.109+0000 7f9827e00640 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb437e0 0x555c8fbb4330 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f43b3d6ae9a6977 server_cookie=719dea51268eeb33 in_seq=0 out_seq=0 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.109+0000 7f981cc00640 10 client.15372 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.109+0000 7f981cc00640 1 -- 172.21.15.146:0/3725806130 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c8fa9da60 con 0x555c8fb437e0 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.109+0000 7f981cc00640 10 client.15372 handle_command_reply: tid=0 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 client.15372 shutdown 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 msgr2=0x555c8fb43fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb43fe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9820018040 tx=0x7f9820005970 comp rx=0 tx=0).stop 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 shutdown_connections 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x555c8fb437e0 0x555c8fbb4330 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555c8fb31460 0x555c8fb3d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555c8fb3bb30 0x555c8fb3d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 --2- 172.21.15.146:0/3725806130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555c8fb30b40 0x555c8fb43fe0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f981cc00640 0 client.15372 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:21:52.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 >> 172.21.15.146:0/3725806130 conn(0x555c8fb2ce60 msgr2=0x555c8fb32b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 shutdown_connections 2024-04-24T08:21:52.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 1 -- 172.21.15.146:0/3725806130 wait complete. 2024-04-24T08:21:52.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.110+0000 7f9839015740 20 client.15372 trim_cache size 0 max 16384 2024-04-24T08:21:52.115 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98140a3ec0 msgr2=0x7f98140a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98140a3ec0 0x7f98140a6380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x555c8fb32100 tx=0x7f982000b4a0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 msgr2=0x7f983019f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 0x7f983019f280 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f981800e9f0 tx=0x7f981800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 shutdown_connections 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98140a3ec0 0x7f98140a6380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9830108f40 0x7f983019f280 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f983010d500 0x7f98301a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 --2- 172.21.15.146:0/793611958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9830108580 0x7f983019ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 >> 172.21.15.146:0/793611958 conn(0x7f9830077e20 msgr2=0x7f9830078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 shutdown_connections 2024-04-24T08:21:52.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.111+0000 7f9834c00640 1 -- 172.21.15.146:0/793611958 wait complete. 2024-04-24T08:21:52.132 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:21:52.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:51 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1864440763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 -- 172.21.15.146:0/153340194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108580 msgr2=0x7f2cd0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/153340194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108580 0x7f2cd0108a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009890 tx=0x7f2cb802a360 comp rx=0 tx=0).stop 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 -- 172.21.15.146:0/153340194 shutdown_connections 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/153340194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108580 0x7f2cd0108a00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/153340194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108f40 0x7f2cd01153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/153340194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 0x7f2cd010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 -- 172.21.15.146:0/153340194 >> 172.21.15.146:0/153340194 conn(0x7f2cd0077e20 msgr2=0x7f2cd0106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 -- 172.21.15.146:0/153340194 shutdown_connections 2024-04-24T08:21:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.335+0000 7f2cd4a00640 1 -- 172.21.15.146:0/153340194 wait complete. 2024-04-24T08:21:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.336+0000 7f2cd4a00640 1 Processor -- start 2024-04-24T08:21:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.336+0000 7f2cd4a00640 1 -- start start 2024-04-24T08:21:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cd4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108580 0x7f2cd019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cd4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108f40 0x7f2cd019f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cd4a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 0x7f2cd01a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cd4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01188d0 con 0x7f2cd0108580 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cd4a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd0118750 con 0x7f2cd010d500 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cd4a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd0118a50 con 0x7f2cd0108f40 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108580 0x7f2cd019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108f40 0x7f2cd019f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2ccf200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 0x7f2cd01a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2ccf200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 0x7f2cd01a3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37592/0 (socket says 172.21.15.146:37592) 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2ccf200640 1 -- 172.21.15.146:0/4015686444 learned_addr learned my addr 172.21.15.146:0/4015686444 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108580 0x7f2cd019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35642/0 (socket says 172.21.15.146:35642) 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108f40 0x7f2cd019f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46648/0 (socket says 172.21.15.146:46648) 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2ccf200640 1 -- 172.21.15.146:0/4015686444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108f40 msgr2=0x7f2cd019f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2ccf200640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108f40 0x7f2cd019f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2ccf200640 1 -- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108580 msgr2=0x7f2cd019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.337+0000 7f2ccf200640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108580 0x7f2cd019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2ccf200640 1 -- 172.21.15.146:0/4015686444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd01a3d50 con 0x7f2cd010d500 2024-04-24T08:21:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2cce800640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108580 0x7f2cd019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108f40 0x7f2cd019f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2ccf200640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 0x7f2cd01a3620 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800ef00 tx=0x7f2cc800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2cc7e00640 1 -- 172.21.15.146:0/4015686444 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8003970 con 0x7f2cd010d500 2024-04-24T08:21:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2cc7e00640 1 -- 172.21.15.146:0/4015686444 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc8004020 con 0x7f2cd010d500 2024-04-24T08:21:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd01a4040 con 0x7f2cd010d500 2024-04-24T08:21:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2cc7e00640 1 -- 172.21.15.146:0/4015686444 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc800b910 con 0x7f2cd010d500 2024-04-24T08:21:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.338+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd01ab8d0 con 0x7f2cd010d500 2024-04-24T08:21:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.341+0000 7f2cd81a9740 1 Processor -- start 2024-04-24T08:21:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cc7e00640 1 -- 172.21.15.146:0/4015686444 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7f2cc801d020 con 0x7f2cd010d500 2024-04-24T08:21:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cd81a9740 1 -- start start 2024-04-24T08:21:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994636b40 0x55f994636f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994637460 0x55f9946415f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994641b30 0x55f994643f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f99459c1d0 con 0x55f994636b40 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f994542450 con 0x55f994641b30 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.342+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f994598190 con 0x55f994637460 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994636b40 0x55f994636f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994636b40 0x55f994636f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35644/0 (socket says 172.21.15.146:35644) 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 -- 172.21.15.146:0/2600264428 learned_addr learned my addr 172.21.15.146:0/2600264428 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cce800640 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994637460 0x55f9946415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994641b30 0x55f994643f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 -- 172.21.15.146:0/2600264428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994637460 msgr2=0x55f9946415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994637460 0x55f9946415f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 -- 172.21.15.146:0/2600264428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994641b30 msgr2=0x55f994643f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994641b30 0x55f994643f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc7e00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2cb40a3e40 0x7f2cb40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 -- 172.21.15.146:0/2600264428 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9945ff310 con 0x55f994636b40 2024-04-24T08:21:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cce800640 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994637460 0x55f9946415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc7e00640 1 -- 172.21.15.146:0/4015686444 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2cc80d09b0 con 0x7f2cd010d500 2024-04-24T08:21:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.343+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994636b40 0x55f994636f20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8005e00 tx=0x7f2cb8009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ca82a02d63a4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.344+0000 7f2ccf200640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2cb40a3e40 0x7f2cb40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.344+0000 7f2cbf400640 1 -- 172.21.15.146:0/2600264428 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8046070 con 0x55f994636b40 2024-04-24T08:21:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.344+0000 7f2cbf400640 1 -- 172.21.15.146:0/2600264428 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb80035f0 con 0x55f994636b40 2024-04-24T08:21:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.344+0000 7f2cbf400640 1 -- 172.21.15.146:0/2600264428 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8037040 con 0x55f994636b40 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 -- 172.21.15.146:0/2600264428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994636b40 msgr2=0x55f994636f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2ccf200640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2cb40a3e40 0x7f2cb40a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2cc80cdf90 tx=0x7f2cc8004ac0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994636b40 0x55f994636f20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8005e00 tx=0x7f2cb8009490 comp rx=0 tx=0).stop 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 -- 172.21.15.146:0/2600264428 shutdown_connections 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994637460 0x55f9946415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994641b30 0x55f994643f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/2600264428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994636b40 0x55f994636f20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 -- 172.21.15.146:0/2600264428 >> 172.21.15.146:0/2600264428 conn(0x55f994632e60 msgr2=0x55f994633240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 -- 172.21.15.146:0/2600264428 shutdown_connections 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 -- 172.21.15.146:0/2600264428 wait complete. 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.345+0000 7f2cd81a9740 1 -- start start 2024-04-24T08:21:52.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 1 Processor -- start 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994636b40 0x55f9946415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994637460 0x55f99463e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994641b30 0x55f99463efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f994644960 con 0x55f994637460 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f994555810 con 0x55f994636b40 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f994644ae0 con 0x55f994641b30 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994637460 0x55f99463e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cd81a9740 10 client.0 _ll_register_callbacks cb 0x55f994523140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994637460 0x55f99463e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35646/0 (socket says 172.21.15.146:35646) 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cce800640 1 -- 172.21.15.146:0/3483189541 learned_addr learned my addr 172.21.15.146:0/3483189541 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.346+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994641b30 0x55f99463efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994636b40 0x55f9946415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cce800640 1 -- 172.21.15.146:0/3483189541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994641b30 msgr2=0x55f99463efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cce800640 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994641b30 0x55f99463efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cce800640 1 -- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994636b40 msgr2=0x55f9946415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cce800640 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994636b40 0x55f9946415b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cce800640 1 -- 172.21.15.146:0/3483189541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9945663f0 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cce800640 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994637460 0x55f99463e8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0012b10 tx=0x7f2cc000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe73adbc3f5b8521 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3483189541 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc000fb40 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3483189541 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2cc000fce0 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.347+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cb80035f0 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3483189541 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2cc0004020 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 10 client.15384 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f9942ae3f0 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3483189541 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb80035f0 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3483189541 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2cc0025470 con 0x55f994637460 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 10 client.15384 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:52.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 10 client.15384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 20 client.15384 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f994651cf0 0x55f994654150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 4 client.15384 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.348+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9945a9bf0 con 0x55f994651cf0 2024-04-24T08:21:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.349+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f994651cf0 0x55f994654150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.349+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f994651cf0 0x55f994654150 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd0741825aa1864f server_cookie=debff0521b792453 in_seq=0 out_seq=0 2024-04-24T08:21:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.349+0000 7f2cb3e00640 10 client.15384 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:52.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3483189541 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f9945a9bf0 con 0x55f994651cf0 2024-04-24T08:21:52.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cb3e00640 10 client.15384 handle_command_reply: tid=0 2024-04-24T08:21:52.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 client.15384 shutdown 2024-04-24T08:21:52.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994637460 msgr2=0x55f99463e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994637460 0x55f99463e8a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0012b10 tx=0x7f2cc000f3c0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 shutdown_connections 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f994651cf0 0x55f994654150 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994641b30 0x55f99463efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994637460 0x55f99463e8a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cb3e00640 0 client.15384 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3483189541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994636b40 0x55f9946415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 >> 172.21.15.146:0/3483189541 conn(0x55f994632e60 msgr2=0x55f994638b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 shutdown_connections 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.350+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3483189541 wait complete. 2024-04-24T08:21:52.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.351+0000 7f2cd81a9740 20 client.15384 trim_cache size 0 max 16384 2024-04-24T08:21:52.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.365+0000 7f2cd81a9740 1 Processor -- start 2024-04-24T08:21:52.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.365+0000 7f2cd81a9740 1 -- start start 2024-04-24T08:21:52.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f99463da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f9946410e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994659a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01b6910 con 0x55f994641b30 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01b41e0 con 0x55f994637460 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01b2030 con 0x55f994636b40 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994659a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994659a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35650/0 (socket says 172.21.15.146:35650) 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 -- 172.21.15.146:0/3994411867 learned_addr learned my addr 172.21.15.146:0/3994411867 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cce800640 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f9946410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f99463da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 -- 172.21.15.146:0/3994411867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 msgr2=0x55f99463da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f99463da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 -- 172.21.15.146:0/3994411867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 msgr2=0x55f9946410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f9946410e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cc5e00640 1 -- 172.21.15.146:0/3994411867 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9943787d0 con 0x55f994641b30 2024-04-24T08:21:52.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.366+0000 7f2cce800640 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f9946410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:52.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.367+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994659a60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009250 tx=0x7f2cb80304e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0015947911626e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.367+0000 7f2cbea00640 1 -- 172.21.15.146:0/3994411867 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8046040 con 0x55f994641b30 2024-04-24T08:21:52.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.367+0000 7f2cbea00640 1 -- 172.21.15.146:0/3994411867 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb8047040 con 0x55f994641b30 2024-04-24T08:21:52.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.367+0000 7f2cbea00640 1 -- 172.21.15.146:0/3994411867 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8037040 con 0x55f994641b30 2024-04-24T08:21:52.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.367+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3994411867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 msgr2=0x55f994659a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.367+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994659a60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009250 tx=0x7f2cb80304e0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3994411867 shutdown_connections 2024-04-24T08:21:52.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994659a60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f9946410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3994411867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f99463da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3994411867 >> 172.21.15.146:0/3994411867 conn(0x55f994632e60 msgr2=0x55f994638290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3994411867 shutdown_connections 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3994411867 wait complete. 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 -- start start 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.368+0000 7f2cd81a9740 1 Processor -- start 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f994649da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f99464a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cd81a9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994644d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01b26a0 con 0x55f994641b30 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01b3b10 con 0x55f994637460 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cd81a9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd01b2d70 con 0x55f994636b40 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cd81a9740 10 client.0 _ll_register_callbacks cb 0x7f2cc0009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cce800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f99464a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f994649da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cce800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f99464a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37624/0 (socket says 172.21.15.146:37624) 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cce800640 1 -- 172.21.15.146:0/3028208307 learned_addr learned my addr 172.21.15.146:0/3028208307 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994644d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.369+0000 7f2cce800640 1 -- 172.21.15.146:0/3028208307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 msgr2=0x55f994649da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cce800640 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f994649da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cce800640 1 -- 172.21.15.146:0/3028208307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 msgr2=0x55f994644d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cce800640 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994644d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cce800640 1 -- 172.21.15.146:0/3028208307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f994645520 con 0x55f994637460 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cce800640 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f99464a2e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0001eb0 tx=0x7f2cc0012a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=361b8eecbe1b524f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3028208307 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc000fb40 con 0x55f994637460 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cb8047040 con 0x55f994637460 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3028208307 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2cc000fce0 con 0x55f994637460 2024-04-24T08:21:52.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3028208307 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2cc0005240 con 0x55f994637460 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cd81a9740 10 client.25520 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.370+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f9946b8d50 con 0x55f994637460 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3028208307 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb8047040 con 0x55f994637460 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3028208307 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2cc0037470 con 0x55f994637460 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cd81a9740 10 client.25520 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cd81a9740 10 client.25520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cd81a9740 20 client.25520 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f9946b9470 0x55f9946bb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cd81a9740 4 client.25520 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f9945a3a60 con 0x55f9946b9470 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.371+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f9946b9470 0x55f9946bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cb3e00640 10 client.25520 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cc5e00640 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f9946b9470 0x55f9946bb890 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e72dd25845e594b4 server_cookie=cd099b9d2f6bb530 in_seq=0 out_seq=0 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cb3e00640 1 -- 172.21.15.146:0/3028208307 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f9945a3a60 con 0x55f9946b9470 2024-04-24T08:21:52.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cb3e00640 10 client.25520 handle_command_reply: tid=0 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 client.25520 shutdown 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 msgr2=0x55f99464a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f99464a2e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0001eb0 tx=0x7f2cc0012a20 comp rx=0 tx=0).stop 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 shutdown_connections 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f9946b9470 0x55f9946bb890 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f994641b30 0x55f994644d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cb3e00640 0 client.25520 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f994637460 0x55f99464a2e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 --2- 172.21.15.146:0/3028208307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f994636b40 0x55f994649da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 >> 172.21.15.146:0/3028208307 conn(0x55f994632e60 msgr2=0x55f99463d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 shutdown_connections 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 1 -- 172.21.15.146:0/3028208307 wait complete. 2024-04-24T08:21:52.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.372+0000 7f2cd81a9740 20 client.25520 trim_cache size 0 max 16384 2024-04-24T08:21:52.378 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:21:52.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2cb40a3e40 msgr2=0x7f2cb40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2cb40a3e40 0x7f2cb40a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2cc80cdf90 tx=0x7f2cc8004ac0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 msgr2=0x7f2cd01a3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 0x7f2cd01a3620 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800ef00 tx=0x7f2cc800c740 comp rx=0 tx=0).stop 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 shutdown_connections 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2cb40a3e40 0x7f2cb40a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cd0108f40 0x7f2cd019f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2cd010d500 0x7f2cd01a3620 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 --2- 172.21.15.146:0/4015686444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2cd0108580 0x7f2cd019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 >> 172.21.15.146:0/4015686444 conn(0x7f2cd0077e20 msgr2=0x7f2cd00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 shutdown_connections 2024-04-24T08:21:52.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:52.373+0000 7f2cd4a00640 1 -- 172.21.15.146:0/4015686444 wait complete. 2024-04-24T08:21:52.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:52 smithi146 ceph-mon[27805]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:53.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:52 smithi177 ceph-mon[33046]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:53.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:52 smithi153 ceph-mon[32746]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:53.391 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:21:53.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:53.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 -- 172.21.15.146:0/4045660057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f8104cc0 msgr2=0x7fc1f8108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 --2- 172.21.15.146:0/4045660057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f8104cc0 0x7fc1f8108d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009900 tx=0x7fc1e402a410 comp rx=0 tx=0).stop 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 -- 172.21.15.146:0/4045660057 shutdown_connections 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 --2- 172.21.15.146:0/4045660057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f81093e0 0x7fc1f8110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 --2- 172.21.15.146:0/4045660057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f8104cc0 0x7fc1f8108d00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 --2- 172.21.15.146:0/4045660057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 0x7fc1f81046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 -- 172.21.15.146:0/4045660057 >> 172.21.15.146:0/4045660057 conn(0x7fc1f80ffb10 msgr2=0x7fc1f8101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 -- 172.21.15.146:0/4045660057 shutdown_connections 2024-04-24T08:21:53.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.612+0000 7fc1f7200640 1 -- 172.21.15.146:0/4045660057 wait complete. 2024-04-24T08:21:53.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.613+0000 7fc1f7200640 1 Processor -- start 2024-04-24T08:21:53.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.613+0000 7fc1f7200640 1 -- start start 2024-04-24T08:21:53.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 0x7fc1f81a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f81093e0 0x7fc1f81a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f7200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f819a800 0x7fc1f819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f81144a0 con 0x7fc1f8104310 2024-04-24T08:21:53.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f7200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f8114320 con 0x7fc1f819a800 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f8114620 con 0x7fc1f81093e0 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 0x7fc1f81a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f81093e0 0x7fc1f81a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f819a800 0x7fc1f819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 0x7fc1f81a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35680/0 (socket says 172.21.15.146:35680) 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.614+0000 7fc1f5e00640 1 -- 172.21.15.146:0/1752816776 learned_addr learned my addr 172.21.15.146:0/1752816776 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1f5e00640 1 -- 172.21.15.146:0/1752816776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f81093e0 msgr2=0x7fc1f81a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f81093e0 0x7fc1f81a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1f5e00640 1 -- 172.21.15.146:0/1752816776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f819a800 msgr2=0x7fc1f819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f819a800 0x7fc1f819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1f5e00640 1 -- 172.21.15.146:0/1752816776 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f819b390 con 0x7fc1f8104310 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 0x7fc1f81a0690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800e9f0 tx=0x7fc1e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1eea00640 1 -- 172.21.15.146:0/1752816776 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800cd80 con 0x7fc1f8104310 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f819b680 con 0x7fc1f8104310 2024-04-24T08:21:53.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1eea00640 1 -- 172.21.15.146:0/1752816776 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e8010910 con 0x7fc1f8104310 2024-04-24T08:21:53.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.615+0000 7fc1eea00640 1 -- 172.21.15.146:0/1752816776 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e80188f0 con 0x7fc1f8104310 2024-04-24T08:21:53.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.616+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f81a7880 con 0x7fc1f8104310 2024-04-24T08:21:53.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 Processor -- start 2024-04-24T08:21:53.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 -- start start 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1eea00640 1 -- 172.21.15.146:0/1752816776 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7fc1e8010ac0 con 0x7fc1f8104310 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2270 0x561a366a2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366acd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366af650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561a36607e70 con 0x561a366a2b90 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a365ae380 con 0x561a366a2270 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.620+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561a36603e30 con 0x561a366ad260 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2270 0x561a366a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2270 0x561a366a2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37644/0 (socket says 172.21.15.146:37644) 2024-04-24T08:21:53.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f6800640 1 -- 172.21.15.146:0/3729280061 learned_addr learned my addr 172.21.15.146:0/3729280061 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f5400640 1 -- 172.21.15.146:0/3729280061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2270 msgr2=0x561a366a2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f5400640 1 --2- 172.21.15.146:0/3729280061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2270 0x561a366a2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f5400640 1 -- 172.21.15.146:0/3729280061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 msgr2=0x561a366acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f5400640 1 --2- 172.21.15.146:0/3729280061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.621+0000 7fc1f5400640 1 -- 172.21.15.146:0/3729280061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a363e27e0 con 0x561a366ad260 2024-04-24T08:21:53.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.622+0000 7fc1f5400640 1 --2- 172.21.15.146:0/3729280061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366af650 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc1e402a930 tx=0x7fc1e4005bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=228ad89908f98df5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.622+0000 7fc1eea00640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc1dc0a3ec0 0x7fc1dc0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.622+0000 7fc1e3400640 1 -- 172.21.15.146:0/3729280061 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4035040 con 0x561a366ad260 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.622+0000 7fc1e3400640 1 -- 172.21.15.146:0/3729280061 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e4004480 con 0x561a366ad260 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.622+0000 7fc1e3400640 1 -- 172.21.15.146:0/3729280061 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40035f0 con 0x561a366ad260 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.622+0000 7fc1eea00640 1 -- 172.21.15.146:0/1752816776 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc1e80d0db0 con 0x7fc1f8104310 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.622+0000 7fc1f6800640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc1dc0a3ec0 0x7fc1dc0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3729280061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 msgr2=0x561a366af650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3729280061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366af650 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc1e402a930 tx=0x7fc1e4005bf0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1f6800640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc1dc0a3ec0 0x7fc1dc0a6380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0005f40 tx=0x7fc1f00096a0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3729280061 shutdown_connections 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3729280061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366af650 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3729280061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3729280061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2270 0x561a366a2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3729280061 >> 172.21.15.146:0/3729280061 conn(0x561a3669e590 msgr2=0x561a3669e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3729280061 shutdown_connections 2024-04-24T08:21:53.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.623+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3729280061 wait complete. 2024-04-24T08:21:53.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 -- start start 2024-04-24T08:21:53.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 Processor -- start 2024-04-24T08:21:53.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366aa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2b90 0x561a366aa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366aac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561a366afff0 con 0x561a366a2270 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a365c1660 con 0x561a366ad260 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.624+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561a366b0170 con 0x561a366a2b90 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1fffe1740 10 client.0 _ll_register_callbacks cb 0x561a365afaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366aa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366aa1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35700/0 (socket says 172.21.15.146:35700) 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 -- 172.21.15.146:0/2104865335 learned_addr learned my addr 172.21.15.146:0/2104865335 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2b90 0x561a366aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 -- 172.21.15.146:0/2104865335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2b90 msgr2=0x561a366aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2b90 0x561a366aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 -- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 msgr2=0x561a366aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5e00640 1 -- 172.21.15.146:0/2104865335 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a36442f70 con 0x561a366a2270 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f5400640 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2b90 0x561a366aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.625+0000 7fc1f6800640 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366aa1f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800e9c0 tx=0x7fc1e80d44b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586f7b249c3ea75c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1d7e00640 1 -- 172.21.15.146:0/2104865335 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e809c520 con 0x561a366a2270 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1d7e00640 1 -- 172.21.15.146:0/2104865335 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc1e80ce870 con 0x561a366a2270 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1e4004480 con 0x561a366a2270 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1d7e00640 1 -- 172.21.15.146:0/2104865335 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc1e809cbc0 con 0x561a366a2270 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1fffe1740 10 client.15402 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561a362da170 con 0x561a366a2270 2024-04-24T08:21:53.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.626+0000 7fc1d7e00640 1 -- 172.21.15.146:0/2104865335 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e4004480 con 0x561a366a2270 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1d7e00640 1 -- 172.21.15.146:0/2104865335 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fc1e80d4af0 con 0x561a366a2270 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1fffe1740 10 client.15402 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1fffe1740 10 client.15402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1fffe1740 20 client.15402 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a366bd4d0 0x561a366bf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1fffe1740 4 client.15402 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a36615890 con 0x561a366bd4d0 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.627+0000 7fc1f5400640 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a366bd4d0 0x561a366bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.628+0000 7fc1f5400640 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a366bd4d0 0x561a366bf900 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c956b6bf92a12a9 server_cookie=1d7131a8ef2235b4 in_seq=0 out_seq=0 2024-04-24T08:21:53.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.628+0000 7fc1d7e00640 10 client.15402 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1d7e00640 1 -- 172.21.15.146:0/2104865335 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a36615890 con 0x561a366bd4d0 2024-04-24T08:21:53.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1d7e00640 10 client.15402 handle_command_reply: tid=0 2024-04-24T08:21:53.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 client.15402 shutdown 2024-04-24T08:21:53.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 msgr2=0x561a366aa1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366aa1f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800e9c0 tx=0x7fc1e80d44b0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 shutdown_connections 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a366bd4d0 0x561a366bf900 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2b90 0x561a366aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1d7e00640 0 client.15402 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/2104865335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366aa1f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 >> 172.21.15.146:0/2104865335 conn(0x561a3669e590 msgr2=0x561a3669d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 shutdown_connections 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 1 -- 172.21.15.146:0/2104865335 wait complete. 2024-04-24T08:21:53.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.629+0000 7fc1fffe1740 20 client.15402 trim_cache size 0 max 16384 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 Processor -- start 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 -- start start 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2b90 0x561a3669d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f81b2600 con 0x561a366a2270 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f81afed0 con 0x561a366a2b90 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f81add20 con 0x561a366ad260 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35706/0 (socket says 172.21.15.146:35706) 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.649+0000 7fc1f5e00640 1 -- 172.21.15.146:0/254736454 learned_addr learned my addr 172.21.15.146:0/254736454 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5400640 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2b90 0x561a3669d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f6800640 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5e00640 1 -- 172.21.15.146:0/254736454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 msgr2=0x561a366c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366c51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5e00640 1 -- 172.21.15.146:0/254736454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2b90 msgr2=0x561a3669d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2b90 0x561a3669d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5e00640 1 -- 172.21.15.146:0/254736454 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a366a3500 con 0x561a366a2270 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5400640 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2b90 0x561a3669d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366b8eb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80097b0 tx=0x7fc1e809c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723b1e62c5a52f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1e2a00640 1 -- 172.21.15.146:0/254736454 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e80ce2d0 con 0x561a366a2270 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1e2a00640 1 -- 172.21.15.146:0/254736454 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e80a5780 con 0x561a366a2270 2024-04-24T08:21:53.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.650+0000 7fc1e2a00640 1 -- 172.21.15.146:0/254736454 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e80ce8d0 con 0x561a366a2270 2024-04-24T08:21:53.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 -- 172.21.15.146:0/254736454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 msgr2=0x561a366b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366b8eb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80097b0 tx=0x7fc1e809c7c0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 -- 172.21.15.146:0/254736454 shutdown_connections 2024-04-24T08:21:53.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366ad260 0x561a366c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366a2b90 0x561a3669d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/254736454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2270 0x561a366b8eb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 -- 172.21.15.146:0/254736454 >> 172.21.15.146:0/254736454 conn(0x561a3669e590 msgr2=0x561a3669ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 -- 172.21.15.146:0/254736454 shutdown_connections 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.651+0000 7fc1fffe1740 1 -- 172.21.15.146:0/254736454 wait complete. 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.652+0000 7fc1fffe1740 1 -- start start 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.652+0000 7fc1fffe1740 1 Processor -- start 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.652+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2270 0x561a366b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1fffe1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f81ae390 con 0x561a366a2b90 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f81af800 con 0x561a366ad260 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1fffe1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f81aea60 con 0x561a366a2270 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1fffe1740 10 client.0 _ll_register_callbacks cb 0x7fc1e8009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2270 0x561a366b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35718/0 (socket says 172.21.15.146:35718) 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 -- 172.21.15.146:0/3530216027 learned_addr learned my addr 172.21.15.146:0/3530216027 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f6800640 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 -- 172.21.15.146:0/3530216027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2270 msgr2=0x561a366b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2270 0x561a366b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 -- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 msgr2=0x561a366b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 -- 172.21.15.146:0/3530216027 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a36722a10 con 0x561a366a2b90 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f6800640 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1f5400640 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366b12f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40096c0 tx=0x7fc1e4005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201a65bec9b288cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.653+0000 7fc1d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1d7e00640 1 -- 172.21.15.146:0/3530216027 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4035050 con 0x561a366a2b90 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1d7e00640 1 -- 172.21.15.146:0/3530216027 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc1e4053370 con 0x561a366a2b90 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1e80a5780 con 0x561a366a2b90 2024-04-24T08:21:53.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1d7e00640 1 -- 172.21.15.146:0/3530216027 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc1e4053570 con 0x561a366a2b90 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1fffe1740 10 client.15408 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561a36723d50 con 0x561a366a2b90 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1d7e00640 1 -- 172.21.15.146:0/3530216027 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e80a5780 con 0x561a366a2b90 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.654+0000 7fc1d7e00640 1 -- 172.21.15.146:0/3530216027 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fc1e40035f0 con 0x561a366a2b90 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1fffe1740 10 client.15408 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1fffe1740 10 client.15408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1fffe1740 20 client.15408 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a36724490 0x561a36726880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1fffe1740 4 client.15408 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561a3660f700 con 0x561a36724490 2024-04-24T08:21:53.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a36724490 0x561a36726880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1f5e00640 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a36724490 0x561a36726880 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8403c7801859c76 server_cookie=93c0019a7654e942 in_seq=0 out_seq=0 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.655+0000 7fc1d7e00640 10 client.15408 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.656+0000 7fc1d7e00640 1 -- 172.21.15.146:0/3530216027 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561a3660f700 con 0x561a36724490 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.656+0000 7fc1d7e00640 10 client.15408 handle_command_reply: tid=0 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.656+0000 7fc1fffe1740 1 client.15408 shutdown 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.656+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 msgr2=0x561a366b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.656+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366b12f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40096c0 tx=0x7fc1e4005750 comp rx=0 tx=0).stop 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 shutdown_connections 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a36724490 0x561a36726880 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a366ad260 0x561a366b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a366a2b90 0x561a366b12f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 --2- 172.21.15.146:0/3530216027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a366a2270 0x561a366b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 >> 172.21.15.146:0/3530216027 conn(0x561a3669e590 msgr2=0x561a3669d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1d7e00640 0 client.15408 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 shutdown_connections 2024-04-24T08:21:53.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 1 -- 172.21.15.146:0/3530216027 wait complete. 2024-04-24T08:21:53.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.657+0000 7fc1fffe1740 20 client.15408 trim_cache size 0 max 16384 2024-04-24T08:21:53.661 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:21:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:21:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "e1b8a28e-5833-4556-ae94-e4fe6b9ec7c7", 2024-04-24T08:21:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:21:53.661 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc1dc0a3ec0 msgr2=0x7fc1dc0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc1dc0a3ec0 0x7fc1dc0a6380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0005f40 tx=0x7fc1f00096a0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 msgr2=0x7fc1f81a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 0x7fc1f81a0690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800e9f0 tx=0x7fc1e800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 shutdown_connections 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc1dc0a3ec0 0x7fc1dc0a6380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1f81093e0 0x7fc1f81a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1f819a800 0x7fc1f819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 --2- 172.21.15.146:0/1752816776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1f8104310 0x7fc1f81a0690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 >> 172.21.15.146:0/1752816776 conn(0x7fc1f80ffb10 msgr2=0x7fc1f8101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 shutdown_connections 2024-04-24T08:21:53.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.658+0000 7fc1f7200640 1 -- 172.21.15.146:0/1752816776 wait complete. 2024-04-24T08:21:53.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:21:53.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 -- 172.21.15.146:0/3124413562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac108580 msgr2=0x7fdeac108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/3124413562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac108580 0x7fdeac108a00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fde94009930 tx=0x7fde9402a440 comp rx=0 tx=0).stop 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 -- 172.21.15.146:0/3124413562 shutdown_connections 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/3124413562 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108f40 0x7fdeac1153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/3124413562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac108580 0x7fdeac108a00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/3124413562 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac10d500 0x7fdeac10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 -- 172.21.15.146:0/3124413562 >> 172.21.15.146:0/3124413562 conn(0x7fdeac077e20 msgr2=0x7fdeac106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 -- 172.21.15.146:0/3124413562 shutdown_connections 2024-04-24T08:21:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.888+0000 7fdeb0a00640 1 -- 172.21.15.146:0/3124413562 wait complete. 2024-04-24T08:21:53.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.889+0000 7fdeb0a00640 1 Processor -- start 2024-04-24T08:21:53.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.889+0000 7fdeb0a00640 1 -- start start 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.889+0000 7fdeb0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108580 0x7fdeac19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeb0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac108f40 0x7fdeac19f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeb0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac10d500 0x7fdeac1a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeb0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac118900 con 0x7fdeac10d500 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeb0a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac118780 con 0x7fdeac108580 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeb0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac118a80 con 0x7fdeac108f40 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac10d500 0x7fdeac1a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeab400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108580 0x7fdeac19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeab400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108580 0x7fdeac19ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37682/0 (socket says 172.21.15.146:37682) 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeab400640 1 -- 172.21.15.146:0/4072282208 learned_addr learned my addr 172.21.15.146:0/4072282208 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac10d500 0x7fdeac1a36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35740/0 (socket says 172.21.15.146:35740) 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdea3e00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac108f40 0x7fdeac19f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeabe00640 1 -- 172.21.15.146:0/4072282208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac108f40 msgr2=0x7fdeac19f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeabe00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac108f40 0x7fdeac19f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeabe00640 1 -- 172.21.15.146:0/4072282208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108580 msgr2=0x7fdeac19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeabe00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108580 0x7fdeac19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeabe00640 1 -- 172.21.15.146:0/4072282208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac1a3de0 con 0x7fdeac10d500 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.890+0000 7fdeab400640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108580 0x7fdeac19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.891+0000 7fdeabe00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac10d500 0x7fdeac1a36b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdea400ebc0 tx=0x7fdea400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.891+0000 7fdea8c00640 1 -- 172.21.15.146:0/4072282208 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4003b70 con 0x7fdeac10d500 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.891+0000 7fdea8c00640 1 -- 172.21.15.146:0/4072282208 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdea4003d10 con 0x7fdeac10d500 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.891+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeac1a40d0 con 0x7fdeac10d500 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.891+0000 7fdea8c00640 1 -- 172.21.15.146:0/4072282208 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea4010720 con 0x7fdeac10d500 2024-04-24T08:21:53.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.891+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeac1ab960 con 0x7fdeac10d500 2024-04-24T08:21:53.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.894+0000 7fdeb49fd740 1 Processor -- start 2024-04-24T08:21:53.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.894+0000 7fdea8c00640 1 -- 172.21.15.146:0/4072282208 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) ==== 138874+0+0 (secure 0 0 0) 0x7fdea4004020 con 0x7fdeac10d500 2024-04-24T08:21:53.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeb49fd740 1 -- start start 2024-04-24T08:21:53.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be2443e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2444350 0x556be244e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2450e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556be23a91f0 con 0x556be2443a30 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556be234f350 con 0x556be244ea20 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556be23a51b0 con 0x556be2444350 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeab400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2444350 0x556be244e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdea3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be2443e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeab400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2444350 0x556be244e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46778/0 (socket says 172.21.15.146:46778) 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.895+0000 7fdeab400640 1 -- 172.21.15.146:0/4087754341 learned_addr learned my addr 172.21.15.146:0/4087754341 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdea3e00640 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2450e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdeab400640 1 -- 172.21.15.146:0/4087754341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 msgr2=0x556be2450e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdeab400640 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2450e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdeab400640 1 -- 172.21.15.146:0/4087754341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 msgr2=0x556be2443e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdeab400640 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be2443e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdeab400640 1 -- 172.21.15.146:0/4087754341 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be23a3f50 con 0x556be2444350 2024-04-24T08:21:53.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdea8c00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fde900a3e40 0x7fde900a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.896+0000 7fdeab400640 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2444350 0x556be244e4e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde9c011540 tx=0x7fde9c011940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc95b17c5d41abe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fdea8c00640 1 -- 172.21.15.146:0/4072282208 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdea40d0fe0 con 0x7fdeac10d500 2024-04-24T08:21:53.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fdea3e00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fde900a3e40 0x7fde900a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fde9be00640 1 -- 172.21.15.146:0/4087754341 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c01e070 con 0x556be2444350 2024-04-24T08:21:53.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fde9be00640 1 -- 172.21.15.146:0/4087754341 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9c00fb60 con 0x556be2444350 2024-04-24T08:21:53.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fde9be00640 1 -- 172.21.15.146:0/4087754341 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c019b50 con 0x556be2444350 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fdeb49fd740 1 -- 172.21.15.146:0/4087754341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2444350 msgr2=0x556be244e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fdea3e00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fde900a3e40 0x7fde900a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fde94006010 tx=0x7fde94005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.897+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2444350 0x556be244e4e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde9c011540 tx=0x7fde9c011940 comp rx=0 tx=0).stop 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 -- 172.21.15.146:0/4087754341 shutdown_connections 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2444350 0x556be244e4e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2450e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/4087754341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be2443e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 -- 172.21.15.146:0/4087754341 >> 172.21.15.146:0/4087754341 conn(0x556be243fd50 msgr2=0x556be2440130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 -- 172.21.15.146:0/4087754341 shutdown_connections 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 -- 172.21.15.146:0/4087754341 wait complete. 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 -- start start 2024-04-24T08:21:53.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.898+0000 7fdeb49fd740 1 Processor -- start 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be244b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be244c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556be2451810 con 0x556be2443a30 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556be2362690 con 0x556be2444350 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556be2451990 con 0x556be244ea20 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeb49fd740 10 client.0 _ll_register_callbacks cb 0x556be239e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be244b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be244b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35760/0 (socket says 172.21.15.146:35760) 2024-04-24T08:21:53.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeab400640 1 -- 172.21.15.146:0/3952675901 learned_addr learned my addr 172.21.15.146:0/3952675901 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdeabe00640 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be244c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.899+0000 7fdea3e00640 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fdeab400640 1 -- 172.21.15.146:0/3952675901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 msgr2=0x556be244c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fdeab400640 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be244c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fdeab400640 1 -- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 msgr2=0x556be244bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fdeab400640 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fdeab400640 1 -- 172.21.15.146:0/3952675901 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be2361920 con 0x556be2443a30 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fdeab400640 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be244b980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0196b0 tx=0x7fde9c009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796f2e46fed6cfd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fde98c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fde98c00640 1 -- 172.21.15.146:0/3952675901 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c01e040 con 0x556be2443a30 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fde98c00640 1 -- 172.21.15.146:0/3952675901 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fde9c016030 con 0x556be2443a30 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.900+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde9c00fb60 con 0x556be2443a30 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fde98c00640 1 -- 172.21.15.146:0/3952675901 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fde9c02f050 con 0x556be2443a30 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 10 client.15420 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:53.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556be21857d0 con 0x556be2443a30 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fde98c00640 1 -- 172.21.15.146:0/3952675901 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9c00fb60 con 0x556be2443a30 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fde98c00640 1 -- 172.21.15.146:0/3952675901 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fde9c021070 con 0x556be2443a30 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 10 client.15420 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 10 client.15420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 20 client.15420 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be245ed30 0x556be2461120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 4 client.15420 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.901+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556be23b6c10 con 0x556be245ed30 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.902+0000 7fdea3e00640 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be245ed30 0x556be2461120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.902+0000 7fdea3e00640 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be245ed30 0x556be2461120 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743919c7d0b48b46 server_cookie=966d8c278a651fc3 in_seq=0 out_seq=0 2024-04-24T08:21:53.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.902+0000 7fde98c00640 10 client.15420 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.903+0000 7fde98c00640 1 -- 172.21.15.146:0/3952675901 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556be23b6c10 con 0x556be245ed30 2024-04-24T08:21:53.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.903+0000 7fde98c00640 10 client.15420 handle_command_reply: tid=0 2024-04-24T08:21:53.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.903+0000 7fdeb49fd740 1 client.15420 shutdown 2024-04-24T08:21:53.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 msgr2=0x556be244b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be244b980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0196b0 tx=0x7fde9c009e10 comp rx=0 tx=0).stop 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 shutdown_connections 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be245ed30 0x556be2461120 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be244c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/3952675901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be244b980 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fde98c00640 0 client.15420 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 >> 172.21.15.146:0/3952675901 conn(0x556be243fd50 msgr2=0x556be243efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 shutdown_connections 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 1 -- 172.21.15.146:0/3952675901 wait complete. 2024-04-24T08:21:53.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.904+0000 7fdeb49fd740 20 client.15420 trim_cache size 0 max 16384 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.923+0000 7fdeb49fd740 1 Processor -- start 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.923+0000 7fdeb49fd740 1 -- start start 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.923+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be245a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.923+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be243ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac1b6a90 con 0x556be2443a30 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac1b4360 con 0x556be2444350 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac1b21b0 con 0x556be244ea20 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be245a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be245a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35766/0 (socket says 172.21.15.146:35766) 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 -- 172.21.15.146:0/1423364372 learned_addr learned my addr 172.21.15.146:0/1423364372 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeabe00640 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be243ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdea3e00640 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 -- 172.21.15.146:0/1423364372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 msgr2=0x556be243ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be243ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 -- 172.21.15.146:0/1423364372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 msgr2=0x556be244b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.924+0000 7fdeab400640 1 -- 172.21.15.146:0/1423364372 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be20bb3f0 con 0x556be2443a30 2024-04-24T08:21:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.925+0000 7fdeab400640 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be245a700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fde9c000c00 tx=0x7fde9c009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae98513469ad5fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.925+0000 7fde9b400640 1 -- 172.21.15.146:0/1423364372 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c01e030 con 0x556be2443a30 2024-04-24T08:21:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.925+0000 7fde9b400640 1 -- 172.21.15.146:0/1423364372 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9c00f8c0 con 0x556be2443a30 2024-04-24T08:21:53.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.925+0000 7fde9b400640 1 -- 172.21.15.146:0/1423364372 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c019ab0 con 0x556be2443a30 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.925+0000 7fdeb49fd740 1 -- 172.21.15.146:0/1423364372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 msgr2=0x556be245a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.925+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be245a700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fde9c000c00 tx=0x7fde9c009550 comp rx=0 tx=0).stop 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 -- 172.21.15.146:0/1423364372 shutdown_connections 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be244ea20 0x556be243ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be2444350 0x556be244b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/1423364372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2443a30 0x556be245a700 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 -- 172.21.15.146:0/1423364372 >> 172.21.15.146:0/1423364372 conn(0x556be243fd50 msgr2=0x556be243e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 -- 172.21.15.146:0/1423364372 shutdown_connections 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 -- 172.21.15.146:0/1423364372 wait complete. 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 -- start start 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.926+0000 7fdeb49fd740 1 Processor -- start 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2443a30 0x556be2451bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2444350 0x556be2452110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeb49fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2452650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac1b2820 con 0x556be2444350 2024-04-24T08:21:53.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac1b3c90 con 0x556be244ea20 2024-04-24T08:21:53.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeb49fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac1b2ef0 con 0x556be2443a30 2024-04-24T08:21:53.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeb49fd740 10 client.0 _ll_register_callbacks cb 0x7fde9c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeabe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2452650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdea3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2444350 0x556be2452110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeabe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2452650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37714/0 (socket says 172.21.15.146:37714) 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeabe00640 1 -- 172.21.15.146:0/2898969586 learned_addr learned my addr 172.21.15.146:0/2898969586 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeab400640 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2443a30 0x556be2451bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeabe00640 1 -- 172.21.15.146:0/2898969586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2443a30 msgr2=0x556be2451bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeabe00640 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2443a30 0x556be2451bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeabe00640 1 -- 172.21.15.146:0/2898969586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2444350 msgr2=0x556be2452110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:53.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.927+0000 7fdeabe00640 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2444350 0x556be2452110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fdeabe00640 1 -- 172.21.15.146:0/2898969586 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be207b0f0 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fdea3e00640 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2444350 0x556be2452110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fdeabe00640 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2452650 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdea40991a0 tx=0x7fdea4009200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb3d943dc2abb33d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fde98c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fde98c00640 1 -- 172.21.15.146:0/2898969586 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea40ba070 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fde98c00640 1 -- 172.21.15.146:0/2898969586 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdea40ce8a0 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde9c00f8c0 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fde98c00640 1 -- 172.21.15.146:0/2898969586 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdea409cd40 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fdeb49fd740 10 client.25556 fetch_fsmap learned FSMap version 24 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.928+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556be236e810 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fde98c00640 1 -- 172.21.15.146:0/2898969586 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9c00f8c0 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fde98c00640 1 -- 172.21.15.146:0/2898969586 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fdea407bbf0 con 0x556be244ea20 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fdeb49fd740 10 client.25556 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:21:53.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fdeb49fd740 10 client.25556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fdeb49fd740 20 client.25556 populate_metadata read hostname 'smithi146' 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be24c4a00 0x556be24c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fdeb49fd740 4 client.25556 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556be23b0a80 con 0x556be24c4a00 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.929+0000 7fdeab400640 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be24c4a00 0x556be24c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeab400640 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be24c4a00 0x556be24c6e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac52df1bb8224e66 server_cookie=a16fb94fd621f243 in_seq=0 out_seq=0 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fde98c00640 10 client.25556 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fde98c00640 1 -- 172.21.15.146:0/2898969586 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556be23b0a80 con 0x556be24c4a00 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fde98c00640 10 client.25556 handle_command_reply: tid=0 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 client.25556 shutdown 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 msgr2=0x556be2452650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2452650 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdea40991a0 tx=0x7fdea4009200 comp rx=0 tx=0).stop 2024-04-24T08:21:53.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 shutdown_connections 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556be24c4a00 0x556be24c6e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556be244ea20 0x556be2452650 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fde98c00640 0 client.25556 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556be2444350 0x556be2452110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 --2- 172.21.15.146:0/2898969586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556be2443a30 0x556be2451bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 >> 172.21.15.146:0/2898969586 conn(0x556be243fd50 msgr2=0x556be243efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 shutdown_connections 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 1 -- 172.21.15.146:0/2898969586 wait complete. 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.930+0000 7fdeb49fd740 20 client.25556 trim_cache size 0 max 16384 2024-04-24T08:21:53.935 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "e1b8a28e-5833-4556-ae94-e4fe6b9ec7c7": { 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "e1b8a28e-5833-4556-ae94-e4fe6b9ec7c7", 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:21:53.936 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:21:53.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fde900a3e40 msgr2=0x7fde900a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fde900a3e40 0x7fde900a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fde94006010 tx=0x7fde94005b50 comp rx=0 tx=0).stop 2024-04-24T08:21:53.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac10d500 msgr2=0x7fdeac1a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:53.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac10d500 0x7fdeac1a36b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdea400ebc0 tx=0x7fdea400c6a0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 shutdown_connections 2024-04-24T08:21:53.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fde900a3e40 0x7fde900a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdeac108f40 0x7fdeac19f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeac10d500 0x7fdeac1a36b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 --2- 172.21.15.146:0/4072282208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdeac108580 0x7fdeac19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:53.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 >> 172.21.15.146:0/4072282208 conn(0x7fdeac077e20 msgr2=0x7fdeac078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:53.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 shutdown_connections 2024-04-24T08:21:53.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:21:53.931+0000 7fdeb0a00640 1 -- 172.21.15.146:0/4072282208 wait complete. 2024-04-24T08:21:53.949 INFO:tasks.cephfs.filesystem:scrub status for tag:e1b8a28e-5833-4556-ae94-e4fe6b9ec7c7 - {'path': '/', 'tag': 'e1b8a28e-5833-4556-ae94-e4fe6b9ec7c7', 'options': 'recursive,force'} 2024-04-24T08:21:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:54 smithi146 ceph-mon[27805]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:54 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:21:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:54 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:54 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:21:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:54 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:54 smithi177 ceph-mon[33046]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:54 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:21:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:54 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:54 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:21:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:54 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:55.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:54 smithi153 ceph-mon[32746]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:21:55.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:54 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:21:55.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:54 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:55.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:54 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:21:55.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:54 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:21:57.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:56 smithi177 ceph-mon[33046]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:57.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:56 smithi153 ceph-mon[32746]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:57.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:56 smithi146 ceph-mon[27805]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:21:58.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:57 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:21:58.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:57 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:21:58.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:57 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:21:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:21:58 smithi177 ceph-mon[33046]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:21:59.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:21:58 smithi153 ceph-mon[32746]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:21:59.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:21:58 smithi146 ceph-mon[27805]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:22:00.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:22:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:22:00.503+0000 7f15e4000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:22:00.767 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:22:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:22:00.503+0000 7f15e4000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:22:00.768 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:22:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:22:00.503+0000 7f15e4000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:22:00.768 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:22:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:22:00.503+0000 7f15e4000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:22:00.768 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:22:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:22:00.503+0000 7f15e4000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:22:01.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:00 smithi153 ceph-mon[32746]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-24T08:22:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:00 smithi177 ceph-mon[33046]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-24T08:22:01.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:00 smithi146 ceph-mon[27805]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 6 op/s 2024-04-24T08:22:03.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:02 smithi177 ceph-mon[33046]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-24T08:22:03.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:02 smithi177 ceph-mon[33046]: mgrmap e26: x(active, since 50s), standbys: y, z 2024-04-24T08:22:03.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:02 smithi153 ceph-mon[32746]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-24T08:22:03.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:02 smithi153 ceph-mon[32746]: mgrmap e26: x(active, since 50s), standbys: y, z 2024-04-24T08:22:03.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:02 smithi146 ceph-mon[27805]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-24T08:22:03.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:02 smithi146 ceph-mon[27805]: mgrmap e26: x(active, since 50s), standbys: y, z 2024-04-24T08:22:05.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:04 smithi177 ceph-mon[33046]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:22:05.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:04 smithi153 ceph-mon[32746]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:22:05.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:04 smithi146 ceph-mon[27805]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:22:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:06 smithi177 ceph-mon[33046]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-24T08:22:07.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:06 smithi153 ceph-mon[32746]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-24T08:22:07.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:06 smithi146 ceph-mon[27805]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-24T08:22:09.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:08 smithi177 ceph-mon[33046]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:22:09.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:08 smithi153 ceph-mon[32746]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:22:09.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:08 smithi146 ceph-mon[27805]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:22:11.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:10 smithi177 ceph-mon[33046]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:11.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:10 smithi153 ceph-mon[32746]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:11.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:10 smithi146 ceph-mon[27805]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:13.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:12 smithi177 ceph-mon[33046]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:13.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:12 smithi153 ceph-mon[32746]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:13.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:12 smithi146 ceph-mon[27805]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:15.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:14 smithi177 ceph-mon[33046]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:22:15.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:14 smithi153 ceph-mon[32746]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:22:15.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:14 smithi146 ceph-mon[27805]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:22:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:16 smithi153 ceph-mon[32746]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:22:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:16 smithi146 ceph-mon[27805]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:22:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:16 smithi177 ceph-mon[33046]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:22:19.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:18 smithi153 ceph-mon[32746]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:22:19.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:18 smithi146 ceph-mon[27805]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:22:19.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:18 smithi177 ceph-mon[33046]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:22:20.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:19 smithi153 ceph-mon[32746]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:22:20.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:19 smithi146 ceph-mon[27805]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:22:20.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:19 smithi177 ceph-mon[33046]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:22:21.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:20 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:22:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:20 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:20 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:20 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:20 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:20 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:22:21.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:20 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:20 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:20 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:20 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:20 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:22:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:20 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:20 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:20 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:20 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:22.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:21 smithi153 ceph-mon[32746]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:22:22.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:21 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:22:22.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:21 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:22:22.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:21 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:22.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:21 smithi146 ceph-mon[27805]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:22:22.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:21 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:22:22.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:21 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:22:22.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:21 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:22.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:21 smithi177 ceph-mon[33046]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:22:22.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:21 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:22:22.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:21 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:22:22.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:21 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:22:23.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:22:23.950 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 -- 172.21.15.146:0/2662112120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 msgr2=0x7f9ed01024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/2662112120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01024f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009900 tx=0x7f9ec002a410 comp rx=0 tx=0).stop 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 -- 172.21.15.146:0/2662112120 shutdown_connections 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/2662112120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ed0102ac0 0x7f9ed0106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/2662112120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed01071e0 0x7f9ed010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/2662112120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01024f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 -- 172.21.15.146:0/2662112120 >> 172.21.15.146:0/2662112120 conn(0x7f9ed00fd9d0 msgr2=0x7f9ed00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 -- 172.21.15.146:0/2662112120 shutdown_connections 2024-04-24T08:22:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.153+0000 7f9ecfe00640 1 -- 172.21.15.146:0/2662112120 wait complete. 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.154+0000 7f9ecfe00640 1 Processor -- start 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.154+0000 7f9ecfe00640 1 -- start start 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.154+0000 7f9ecfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed01071e0 0x7f9ed01a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ed019a860 0x7f9ed019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01122e0 con 0x7f9ed0102110 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecfe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed0112160 con 0x7f9ed01071e0 2024-04-24T08:22:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecfe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed0112460 con 0x7f9ed019a860 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54676/0 (socket says 172.21.15.146:54676) 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 -- 172.21.15.146:0/643520951 learned_addr learned my addr 172.21.15.146:0/643520951 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ece000640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed01071e0 0x7f9ed01a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecf400640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ed019a860 0x7f9ed019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 -- 172.21.15.146:0/643520951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ed019a860 msgr2=0x7f9ed019ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ed019a860 0x7f9ed019ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 -- 172.21.15.146:0/643520951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed01071e0 msgr2=0x7f9ed01a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed01071e0 0x7f9ed01a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ecea00640 1 -- 172.21.15.146:0/643520951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed019b380 con 0x7f9ed0102110 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.155+0000 7f9ece000640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed01071e0 0x7f9ed01a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.156+0000 7f9ecea00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01a06b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0004770 tx=0x7f9ec0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.156+0000 7f9ec7400640 1 -- 172.21.15.146:0/643520951 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec003f070 con 0x7f9ed0102110 2024-04-24T08:22:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.156+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed019b610 con 0x7f9ed0102110 2024-04-24T08:22:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.156+0000 7f9ec7400640 1 -- 172.21.15.146:0/643520951 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec0031ca0 con 0x7f9ed0102110 2024-04-24T08:22:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.156+0000 7f9ec7400640 1 -- 172.21.15.146:0/643520951 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0035030 con 0x7f9ed0102110 2024-04-24T08:22:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.156+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed01a7800 con 0x7f9ed0102110 2024-04-24T08:22:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.160+0000 7f9ed8a14740 1 Processor -- start 2024-04-24T08:22:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.160+0000 7f9ed8a14740 1 -- start start 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed185e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1904e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed192e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f5ed0eb1f0 con 0x55f5ed186350 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f5ed091350 con 0x55f5ed190a20 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f5ed0e71b0 con 0x55f5ed185a30 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ec7400640 1 -- 172.21.15.146:0/643520951 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 26) ==== 138931+0+0 (secure 0 0 0) 0x7f9ec0042070 con 0x7f9ed0102110 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed185e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1904e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed185e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56140/0 (socket says 172.21.15.146:56140) 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ecf400640 1 -- 172.21.15.146:0/896070128 learned_addr learned my addr 172.21.15.146:0/896070128 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.161+0000 7f9ece000640 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed192e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ecf400640 1 -- 172.21.15.146:0/896070128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 msgr2=0x55f5ed192e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ecf400640 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed192e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ecf400640 1 -- 172.21.15.146:0/896070128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 msgr2=0x55f5ed1904e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ecf400640 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1904e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ecf400640 1 -- 172.21.15.146:0/896070128 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5ed0e5f50 con 0x55f5ed185a30 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ecf400640 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed185e10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009f50 tx=0x7f9ec80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ff406d4bc0a7201 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ebbe00640 1 -- 172.21.15.146:0/896070128 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8004d20 con 0x55f5ed185a30 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ebbe00640 1 -- 172.21.15.146:0/896070128 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec8004050 con 0x55f5ed185a30 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.162+0000 7f9ebbe00640 1 -- 172.21.15.146:0/896070128 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8005240 con 0x55f5ed185a30 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.163+0000 7f9ec7400640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9eb40a3ed0 0x7f9eb40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.163+0000 7f9ec7400640 1 -- 172.21.15.146:0/643520951 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9ec00f0d40 con 0x7f9ed0102110 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.163+0000 7f9ecea00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9eb40a3ed0 0x7f9eb40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.163+0000 7f9ed8a14740 1 -- 172.21.15.146:0/896070128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 msgr2=0x55f5ed185e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.163+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed185e10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009f50 tx=0x7f9ec80118d0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.163+0000 7f9ed8a14740 1 -- 172.21.15.146:0/896070128 shutdown_connections 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed192e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1904e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/896070128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed185e10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 -- 172.21.15.146:0/896070128 >> 172.21.15.146:0/896070128 conn(0x55f5ed181d50 msgr2=0x55f5ed182130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 -- 172.21.15.146:0/896070128 shutdown_connections 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 -- 172.21.15.146:0/896070128 wait complete. 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ecea00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9eb40a3ed0 0x7f9eb40a6390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00ee2c0 tx=0x7f9ec00f47f0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 -- start start 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.164+0000 7f9ed8a14740 1 Processor -- start 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed18d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed18de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed18e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f5ed193810 con 0x55f5ed186350 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f5ed0a4690 con 0x55f5ed190a20 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f5ed193990 con 0x55f5ed185a30 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ed8a14740 10 client.0 _ll_register_callbacks cb 0x55f5ed0e0650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed18d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed18d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56156/0 (socket says 172.21.15.146:56156) 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ecf400640 1 -- 172.21.15.146:0/2621500818 learned_addr learned my addr 172.21.15.146:0/2621500818 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.165+0000 7f9ece000640 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed18de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ecf400640 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed18e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ecf400640 1 -- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 msgr2=0x55f5ed18e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ecf400640 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed18e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ecf400640 1 -- 172.21.15.146:0/2621500818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 msgr2=0x55f5ed18de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ecf400640 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed18de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ecf400640 1 -- 172.21.15.146:0/2621500818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5ed0a3920 con 0x55f5ed185a30 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ecf400640 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed18d900 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009d60 tx=0x7f9ec8005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca05ce3b5e396f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9eb8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:22:24.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9eb8c00640 1 -- 172.21.15.146:0/2621500818 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec802b020 con 0x55f5ed185a30 2024-04-24T08:22:24.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9eb8c00640 1 -- 172.21.15.146:0/2621500818 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9ec8004cd0 con 0x55f5ed185a30 2024-04-24T08:22:24.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.166+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ec8004050 con 0x55f5ed185a30 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9eb8c00640 1 -- 172.21.15.146:0/2621500818 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9ec8027470 con 0x55f5ed185a30 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 10 client.25573 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f5ecec77d0 con 0x55f5ed185a30 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9eb8c00640 1 -- 172.21.15.146:0/2621500818 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec8004050 con 0x55f5ed185a30 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9eb8c00640 1 -- 172.21.15.146:0/2621500818 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9ec80257c0 con 0x55f5ed185a30 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 10 client.25573 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 10 client.25573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 20 client.25573 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed1a0c80 0x55f5ed1a3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 4 client.25573 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.167+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5ed0f8c10 con 0x55f5ed1a0c80 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.168+0000 7f9ece000640 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed1a0c80 0x55f5ed1a3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.168+0000 7f9eb8c00640 10 client.25573 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.168+0000 7f9ece000640 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed1a0c80 0x55f5ed1a3070 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33b1ac475c851b7e server_cookie=82d96fc2170ce5d7 in_seq=0 out_seq=0 2024-04-24T08:22:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9eb8c00640 1 -- 172.21.15.146:0/2621500818 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f5ed0f8c10 con 0x55f5ed1a0c80 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9eb8c00640 10 client.25573 handle_command_reply: tid=0 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 client.25573 shutdown 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 msgr2=0x55f5ed18d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed18d900 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009d60 tx=0x7f9ec8005bd0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 shutdown_connections 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed1a0c80 0x55f5ed1a3070 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed18e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed18de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.169+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/2621500818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed18d900 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.170+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 >> 172.21.15.146:0/2621500818 conn(0x55f5ed181d50 msgr2=0x55f5ed187b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.170+0000 7f9eb8c00640 0 client.25573 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.170+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 shutdown_connections 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.170+0000 7f9ed8a14740 1 -- 172.21.15.146:0/2621500818 wait complete. 2024-04-24T08:22:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.170+0000 7f9ed8a14740 20 client.25573 trim_cache size 0 max 16384 2024-04-24T08:22:24.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.188+0000 7f9ed8a14740 1 Processor -- start 2024-04-24T08:22:24.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.188+0000 7f9ed8a14740 1 -- start start 2024-04-24T08:22:24.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1a8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed1a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01b2670 con 0x55f5ed186350 2024-04-24T08:22:24.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01aff40 con 0x55f5ed190a20 2024-04-24T08:22:24.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01add90 con 0x55f5ed185a30 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ece000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1a8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed19c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56158/0 (socket says 172.21.15.146:56158) 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ecf400640 1 -- 172.21.15.146:0/1997580505 learned_addr learned my addr 172.21.15.146:0/1997580505 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.189+0000 7f9ecf400640 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed1a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ecf400640 1 -- 172.21.15.146:0/1997580505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 msgr2=0x55f5ed1a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ecf400640 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed1a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ecf400640 1 -- 172.21.15.146:0/1997580505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 msgr2=0x55f5ed1a8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ecf400640 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1a8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ecf400640 1 -- 172.21.15.146:0/1997580505 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5ecdfd3f0 con 0x55f5ed185a30 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ece000640 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:24.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ecf400640 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed19c6d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8025b90 tx=0x7f9ec8009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abd80a8e39dc7481 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ebb400640 1 -- 172.21.15.146:0/1997580505 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec802b040 con 0x55f5ed185a30 2024-04-24T08:22:24.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ebb400640 1 -- 172.21.15.146:0/1997580505 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec8025960 con 0x55f5ed185a30 2024-04-24T08:22:24.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.190+0000 7f9ebb400640 1 -- 172.21.15.146:0/1997580505 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8027ad0 con 0x55f5ed185a30 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 -- 172.21.15.146:0/1997580505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 msgr2=0x55f5ed19c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed19c6d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8025b90 tx=0x7f9ec8009230 comp rx=0 tx=0).stop 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 -- 172.21.15.146:0/1997580505 shutdown_connections 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed190a20 0x55f5ed1a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed186350 0x55f5ed1a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/1997580505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed185a30 0x55f5ed19c6d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 -- 172.21.15.146:0/1997580505 >> 172.21.15.146:0/1997580505 conn(0x55f5ed181d50 msgr2=0x55f5ed18c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 -- 172.21.15.146:0/1997580505 shutdown_connections 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 -- 172.21.15.146:0/1997580505 wait complete. 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.191+0000 7f9ed8a14740 1 -- start start 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 1 Processor -- start 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed185a30 0x55f5ed1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed186350 0x55f5ed18acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed190a20 0x55f5ed18b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01ae400 con 0x55f5ed185a30 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01af870 con 0x55f5ed186350 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01aead0 con 0x55f5ed190a20 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ed8a14740 10 client.0 _ll_register_callbacks cb 0x7f9ec00f46c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed185a30 0x55f5ed1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed185a30 0x55f5ed1990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54714/0 (socket says 172.21.15.146:54714) 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.192+0000 7f9ecf400640 1 -- 172.21.15.146:0/3296363698 learned_addr learned my addr 172.21.15.146:0/3296363698 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ece000640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed186350 0x55f5ed18acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ecf400640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed190a20 0x55f5ed18b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ece000640 1 -- 172.21.15.146:0/3296363698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed190a20 msgr2=0x55f5ed18b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ece000640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed190a20 0x55f5ed18b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ece000640 1 -- 172.21.15.146:0/3296363698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed185a30 msgr2=0x55f5ed1990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ece000640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed185a30 0x55f5ed1990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ece000640 1 -- 172.21.15.146:0/3296363698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5ecdbd0f0 con 0x55f5ed186350 2024-04-24T08:22:24.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ecf400640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed185a30 0x55f5ed1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:24.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9ece000640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed186350 0x55f5ed18acd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55f5ed18d160 tx=0x7f9ebc004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=499289e4c0ed1b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.193+0000 7f9eb8c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.194+0000 7f9eb8c00640 1 -- 172.21.15.146:0/3296363698 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc012930 con 0x55f5ed186350 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.194+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ec8025960 con 0x55f5ed186350 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.194+0000 7f9eb8c00640 1 -- 172.21.15.146:0/3296363698 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9ebc012ad0 con 0x55f5ed186350 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.194+0000 7f9eb8c00640 1 -- 172.21.15.146:0/3296363698 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9ebc012cd0 con 0x55f5ed186350 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.194+0000 7f9ed8a14740 10 client.25580 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.194+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f5ed0b0810 con 0x55f5ed186350 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.194+0000 7f9eb8c00640 1 -- 172.21.15.146:0/3296363698 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec8025960 con 0x55f5ed186350 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.195+0000 7f9eb8c00640 1 -- 172.21.15.146:0/3296363698 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9ebc031590 con 0x55f5ed186350 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.195+0000 7f9ed8a14740 10 client.25580 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.195+0000 7f9ed8a14740 10 client.25580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.195+0000 7f9ed8a14740 20 client.25580 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.195+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed2066c0 0x55f5ed208ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.195+0000 7f9ed8a14740 4 client.25580 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:22:24.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.195+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f5ed0f2a80 con 0x55f5ed2066c0 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9ecf400640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed2066c0 0x55f5ed208ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9ecf400640 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed2066c0 0x55f5ed208ae0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d50c349ed6caef server_cookie=1a75c7cd6b4a46b2 in_seq=0 out_seq=0 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9eb8c00640 10 client.25580 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9eb8c00640 1 -- 172.21.15.146:0/3296363698 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f5ed0f2a80 con 0x55f5ed2066c0 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9eb8c00640 10 client.25580 handle_command_reply: tid=0 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9ed8a14740 1 client.25580 shutdown 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed186350 msgr2=0x55f5ed18acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.196+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed186350 0x55f5ed18acd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55f5ed18d160 tx=0x7f9ebc004410 comp rx=0 tx=0).stop 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 shutdown_connections 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f5ed2066c0 0x55f5ed208ae0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5ed190a20 0x55f5ed18b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5ed186350 0x55f5ed18acd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9eb8c00640 0 client.25580 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 --2- 172.21.15.146:0/3296363698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5ed185a30 0x55f5ed1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 >> 172.21.15.146:0/3296363698 conn(0x55f5ed181d50 msgr2=0x55f5ed18d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 shutdown_connections 2024-04-24T08:22:24.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 1 -- 172.21.15.146:0/3296363698 wait complete. 2024-04-24T08:22:24.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ed8a14740 20 client.25580 trim_cache size 0 max 16384 2024-04-24T08:22:24.201 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:22:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:22:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:22:24.201 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9eb40a3ed0 msgr2=0x7f9eb40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9eb40a3ed0 0x7f9eb40a6390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00ee2c0 tx=0x7f9ec00f47f0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 msgr2=0x7f9ed01a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.197+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01a06b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0004770 tx=0x7f9ec0003c10 comp rx=0 tx=0).stop 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 shutdown_connections 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9eb40a3ed0 0x7f9eb40a6390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ed019a860 0x7f9ed019ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed01071e0 0x7f9ed01a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 --2- 172.21.15.146:0/643520951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed0102110 0x7f9ed01a06b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 >> 172.21.15.146:0/643520951 conn(0x7f9ed00fd9d0 msgr2=0x7f9ed00fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 shutdown_connections 2024-04-24T08:22:24.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.198+0000 7f9ecfe00640 1 -- 172.21.15.146:0/643520951 wait complete. 2024-04-24T08:22:24.217 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:22:24.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:22:24.270 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:24 smithi146 ceph-mon[27805]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 -- 172.21.15.146:0/3091585733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c074cf0 msgr2=0x7f733c10e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 --2- 172.21.15.146:0/3091585733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c074cf0 0x7f733c10e350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7328009900 tx=0x7f732802a480 comp rx=0 tx=0).stop 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 -- 172.21.15.146:0/3091585733 shutdown_connections 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 --2- 172.21.15.146:0/3091585733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f733c10e890 0x7f733c110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 --2- 172.21.15.146:0/3091585733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c074cf0 0x7f733c10e350 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 --2- 172.21.15.146:0/3091585733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c0743d0 0x7f733c0747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 -- 172.21.15.146:0/3091585733 >> 172.21.15.146:0/3091585733 conn(0x7f733c0fdaa0 msgr2=0x7f733c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 -- 172.21.15.146:0/3091585733 shutdown_connections 2024-04-24T08:22:24.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.399+0000 7f7341600640 1 -- 172.21.15.146:0/3091585733 wait complete. 2024-04-24T08:22:24.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 Processor -- start 2024-04-24T08:22:24.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 -- start start 2024-04-24T08:22:24.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c0743d0 0x7f733c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c074cf0 0x7f733c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f733c10e890 0x7f733c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1145f0 con 0x7f733c0743d0 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f733c114470 con 0x7f733c10e890 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.400+0000 7f7341600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f733c114770 con 0x7f733c074cf0 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c0743d0 0x7f733c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c0743d0 0x7f733c1a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54730/0 (socket says 172.21.15.146:54730) 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 -- 172.21.15.146:0/3143734695 learned_addr learned my addr 172.21.15.146:0/3143734695 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733a000640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c074cf0 0x7f733c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733b400640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f733c10e890 0x7f733c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 -- 172.21.15.146:0/3143734695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c074cf0 msgr2=0x7f733c1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c074cf0 0x7f733c1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 -- 172.21.15.146:0/3143734695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f733c10e890 msgr2=0x7f733c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f733c10e890 0x7f733c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733aa00640 1 -- 172.21.15.146:0/3143734695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733c19b0f0 con 0x7f733c0743d0 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.401+0000 7f733a000640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c074cf0 0x7f733c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.402+0000 7f733aa00640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c0743d0 0x7f733c1a07c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f732c00e9f0 tx=0x7f732c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.402+0000 7f7333400640 1 -- 172.21.15.146:0/3143734695 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c00cd80 con 0x7f733c0743d0 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.402+0000 7f7333400640 1 -- 172.21.15.146:0/3143734695 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f732c010910 con 0x7f733c0743d0 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.402+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f733c19b3e0 con 0x7f733c0743d0 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.402+0000 7f7333400640 1 -- 172.21.15.146:0/3143734695 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c018990 con 0x7f733c0743d0 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.402+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f733c1a7680 con 0x7f733c0743d0 2024-04-24T08:22:24.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.403+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f733c068e20 con 0x7f733c0743d0 2024-04-24T08:22:24.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.405+0000 7f7333400640 1 -- 172.21.15.146:0/3143734695 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 26) ==== 138931+0+0 (secure 0 0 0) 0x7f732c010ac0 con 0x7f733c0743d0 2024-04-24T08:22:24.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.406+0000 7f7333400640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f73200a3f50 0x7f73200a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.407+0000 7f7333400640 1 -- 172.21.15.146:0/3143734695 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f732c0d0d50 con 0x7f733c0743d0 2024-04-24T08:22:24.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.407+0000 7f733a000640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f73200a3f50 0x7f73200a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.408+0000 7f733a000640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f73200a3f50 0x7f73200a6410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7328009230 tx=0x7f73280058d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.416+0000 7f7333400640 1 -- 172.21.15.146:0/3143734695 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f732c09cb70 con 0x7f733c0743d0 2024-04-24T08:22:24.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:24 smithi153 ceph-mon[32746]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:22:24.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:24 smithi177 ceph-mon[33046]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:22:24.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.694+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f733c19bea0 con 0x7f733c0743d0 2024-04-24T08:22:24.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.695+0000 7f7333400640 1 -- 172.21.15.146:0/3143734695 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11891 (secure 0 0 0) 0x7f732c0a0e70 con 0x7f733c0743d0 2024-04-24T08:22:24.697 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:22:24.697 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:21:10.741287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24553,24568]},"id":1}]} 2024-04-24T08:22:24.697 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 24 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f73200a3f50 msgr2=0x7f73200a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f73200a3f50 0x7f73200a6410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7328009230 tx=0x7f73280058d0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c0743d0 msgr2=0x7f733c1a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c0743d0 0x7f733c1a07c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f732c00e9f0 tx=0x7f732c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 shutdown_connections 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f73200a3f50 0x7f73200a6410 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c074cf0 0x7f733c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f733c10e890 0x7f733c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 --2- 172.21.15.146:0/3143734695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f733c0743d0 0x7f733c1a07c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 >> 172.21.15.146:0/3143734695 conn(0x7f733c0fdaa0 msgr2=0x7f733c0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 shutdown_connections 2024-04-24T08:22:24.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.698+0000 7f7341600640 1 -- 172.21.15.146:0/3143734695 wait complete. 2024-04-24T08:22:24.718 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:22:24.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.914+0000 7f4b0a600640 1 -- 172.21.15.146:0/1519672973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 msgr2=0x7f4b0410bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.914+0000 7f4b0a600640 1 --2- 172.21.15.146:0/1519672973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 0x7f4b0410bf00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009900 tx=0x7f4af002a480 comp rx=0 tx=0).stop 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.915+0000 7f4b0a600640 1 -- 172.21.15.146:0/1519672973 shutdown_connections 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.915+0000 7f4b0a600640 1 --2- 172.21.15.146:0/1519672973 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b0410c4d0 0x7f4b0410e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.915+0000 7f4b0a600640 1 --2- 172.21.15.146:0/1519672973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 0x7f4b0410bf00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.915+0000 7f4b0a600640 1 --2- 172.21.15.146:0/1519672973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b04103100 0x7f4b041034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.915+0000 7f4b0a600640 1 -- 172.21.15.146:0/1519672973 >> 172.21.15.146:0/1519672973 conn(0x7f4b040fc880 msgr2=0x7f4b040feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.915+0000 7f4b0a600640 1 -- 172.21.15.146:0/1519672973 shutdown_connections 2024-04-24T08:22:24.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.915+0000 7f4b0a600640 1 -- 172.21.15.146:0/1519672973 wait complete. 2024-04-24T08:22:24.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.916+0000 7f4b0a600640 1 Processor -- start 2024-04-24T08:22:24.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.916+0000 7f4b0a600640 1 -- start start 2024-04-24T08:22:24.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b0a600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04103100 0x7f4b041a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b0a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 0x7f4b041a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b0a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b0410c4d0 0x7f4b0419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b0a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4b04112290 con 0x7f4b04103a20 2024-04-24T08:22:24.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b0a600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b04112110 con 0x7f4b04103100 2024-04-24T08:22:24.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b0a600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4b04112410 con 0x7f4b0410c4d0 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b03e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 0x7f4b041a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b09200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04103100 0x7f4b041a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b09c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b0410c4d0 0x7f4b0419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b03e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 0x7f4b041a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54760/0 (socket says 172.21.15.146:54760) 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.917+0000 7f4b03e00640 1 -- 172.21.15.146:0/4147460597 learned_addr learned my addr 172.21.15.146:0/4147460597 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b09c00640 1 -- 172.21.15.146:0/4147460597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04103100 msgr2=0x7f4b041a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b09c00640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04103100 0x7f4b041a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b09c00640 1 -- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 msgr2=0x7f4b041a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b09c00640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 0x7f4b041a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b09c00640 1 -- 172.21.15.146:0/4147460597 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0419afa0 con 0x7f4b0410c4d0 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b09200640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04103100 0x7f4b041a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b09c00640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b0410c4d0 0x7f4b0419a7d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00ec50 tx=0x7f4afc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b01600640 1 -- 172.21.15.146:0/4147460597 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc003b70 con 0x7f4b0410c4d0 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0419b290 con 0x7f4b0410c4d0 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b01600640 1 -- 172.21.15.146:0/4147460597 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4afc003d10 con 0x7f4b0410c4d0 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.918+0000 7f4b01600640 1 -- 172.21.15.146:0/4147460597 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc010890 con 0x7f4b0410c4d0 2024-04-24T08:22:24.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.919+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b041a74b0 con 0x7f4b0410c4d0 2024-04-24T08:22:24.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.923+0000 7f4b0ebae740 1 Processor -- start 2024-04-24T08:22:24.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.923+0000 7f4b01600640 1 -- 172.21.15.146:0/4147460597 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138931+0+0 (secure 0 0 0) 0x7f4afc003fe0 con 0x7f4b0410c4d0 2024-04-24T08:22:24.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b0ebae740 1 -- start start 2024-04-24T08:22:24.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b28f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b33b30 0x5558e8b35f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5558e8a8e1d0 con 0x5558e8b29460 2024-04-24T08:22:24.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5558e8a34450 con 0x5558e8b33b30 2024-04-24T08:22:24.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5558e8a8a190 con 0x5558e8b28b40 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.924+0000 7f4b03e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b28f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b28f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56214/0 (socket says 172.21.15.146:56214) 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 -- 172.21.15.146:0/1447153361 learned_addr learned my addr 172.21.15.146:0/1447153361 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b33b30 0x5558e8b35f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b09200640 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 -- 172.21.15.146:0/1447153361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b33b30 msgr2=0x5558e8b35f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b33b30 0x5558e8b35f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 -- 172.21.15.146:0/1447153361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 msgr2=0x5558e8b335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.925+0000 7f4b03e00640 1 -- 172.21.15.146:0/1447153361 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e8af1310 con 0x5558e8b28b40 2024-04-24T08:22:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.926+0000 7f4b03e00640 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b28f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4af002aa70 tx=0x7f4af002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ade649d51c7744e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.926+0000 7f4b01600640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4aec0a3ed0 0x7f4aec0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.926+0000 7f4af6000640 1 -- 172.21.15.146:0/1447153361 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0035030 con 0x5558e8b28b40 2024-04-24T08:22:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.926+0000 7f4af6000640 1 -- 172.21.15.146:0/1447153361 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4af00047a0 con 0x5558e8b28b40 2024-04-24T08:22:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.926+0000 7f4af6000640 1 -- 172.21.15.146:0/1447153361 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0003850 con 0x5558e8b28b40 2024-04-24T08:22:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.926+0000 7f4b01600640 1 -- 172.21.15.146:0/4147460597 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4afc0d0ff0 con 0x7f4b0410c4d0 2024-04-24T08:22:24.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.926+0000 7f4b09200640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4aec0a3ed0 0x7f4aec0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1447153361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 msgr2=0x5558e8b28f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b28f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4af002aa70 tx=0x7f4af002a480 comp rx=0 tx=0).stop 2024-04-24T08:22:24.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1447153361 shutdown_connections 2024-04-24T08:22:24.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b33b30 0x5558e8b35f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1447153361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b28f20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1447153361 >> 172.21.15.146:0/1447153361 conn(0x5558e8b24e60 msgr2=0x5558e8b25240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1447153361 shutdown_connections 2024-04-24T08:22:24.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b09200640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4aec0a3ed0 0x7f4aec0a6390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x5558e8b2a100 tx=0x7f4af8005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.927+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1447153361 wait complete. 2024-04-24T08:22:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.928+0000 7f4b0ebae740 1 -- start start 2024-04-24T08:22:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.928+0000 7f4b0ebae740 1 Processor -- start 2024-04-24T08:22:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b28b40 0x5558e8b33560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b30850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b33b30 0x5558e8b30f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b09200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b28b40 0x5558e8b33560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5558e8b36960 con 0x5558e8b29460 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5558e8a47810 con 0x5558e8b28b40 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5558e8b36ae0 con 0x5558e8b33b30 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b09200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b28b40 0x5558e8b33560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:42898/0 (socket says 172.21.15.146:42898) 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b09200640 1 -- 172.21.15.146:0/2727197730 learned_addr learned my addr 172.21.15.146:0/2727197730 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b0ebae740 10 client.0 _ll_register_callbacks cb 0x5558e8a15140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.929+0000 7f4b03e00640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b30850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09c00640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b33b30 0x5558e8b30f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09200640 1 -- 172.21.15.146:0/2727197730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b33b30 msgr2=0x5558e8b30f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09200640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b33b30 0x5558e8b30f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09200640 1 -- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 msgr2=0x5558e8b30850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09200640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b30850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09200640 1 -- 172.21.15.146:0/2727197730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e8a583f0 con 0x5558e8b28b40 2024-04-24T08:22:24.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09c00640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b33b30 0x5558e8b30f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:24.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4b09200640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b28b40 0x5558e8b33560 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4af800fb70 tx=0x7f4af80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4afcec3b1a3ddc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.930+0000 7f4aeaa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.931+0000 7f4aeaa00640 1 -- 172.21.15.146:0/2727197730 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af802e060 con 0x5558e8b28b40 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.931+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4af00047a0 con 0x5558e8b28b40 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.931+0000 7f4aeaa00640 1 -- 172.21.15.146:0/2727197730 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4af8018030 con 0x5558e8b28b40 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.931+0000 7f4aeaa00640 1 -- 172.21.15.146:0/2727197730 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4af800fde0 con 0x5558e8b28b40 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.931+0000 7f4b0ebae740 10 client.25598 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.931+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5558e87a03f0 con 0x5558e8b28b40 2024-04-24T08:22:24.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.931+0000 7f4aeaa00640 1 -- 172.21.15.146:0/2727197730 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4af00047a0 con 0x5558e8b28b40 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.932+0000 7f4aeaa00640 1 -- 172.21.15.146:0/2727197730 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4af80319e0 con 0x5558e8b28b40 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.933+0000 7f4b0ebae740 10 client.25598 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.933+0000 7f4b0ebae740 10 client.25598 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.933+0000 7f4b0ebae740 20 client.25598 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.933+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8b43c70 0x5558e8b460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.933+0000 7f4b0ebae740 4 client.25598 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.933+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558e8a9bbf0 con 0x5558e8b43c70 2024-04-24T08:22:24.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.933+0000 7f4b03e00640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8b43c70 0x5558e8b460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.934+0000 7f4b03e00640 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8b43c70 0x5558e8b460c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=10babdd2e51e2ace server_cookie=b74e9a9a35d94e6 in_seq=0 out_seq=0 2024-04-24T08:22:24.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.934+0000 7f4aeaa00640 10 client.25598 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:24.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4aeaa00640 1 -- 172.21.15.146:0/2727197730 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5558e8a9bbf0 con 0x5558e8b43c70 2024-04-24T08:22:24.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4aeaa00640 10 client.25598 handle_command_reply: tid=0 2024-04-24T08:22:24.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 client.25598 shutdown 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b28b40 msgr2=0x5558e8b33560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b28b40 0x5558e8b33560 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4af800fb70 tx=0x7f4af80046c0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 shutdown_connections 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8b43c70 0x5558e8b460c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b33b30 0x5558e8b30f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b29460 0x5558e8b30850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2727197730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b28b40 0x5558e8b33560 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.935+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 >> 172.21.15.146:0/2727197730 conn(0x5558e8b24e60 msgr2=0x5558e8b2ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.936+0000 7f4aeaa00640 0 client.25598 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.936+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 shutdown_connections 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.936+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2727197730 wait complete. 2024-04-24T08:22:24.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.936+0000 7f4b0ebae740 20 client.25598 trim_cache size 0 max 16384 2024-04-24T08:22:24.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.957+0000 7f4b0ebae740 1 Processor -- start 2024-04-24T08:22:24.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.957+0000 7f4b0ebae740 1 -- start start 2024-04-24T08:22:24.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.957+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b2b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.957+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b30130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.957+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b23dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041b6a80 con 0x5558e8b33b30 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041b4350 con 0x5558e8b29460 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041b21c0 con 0x5558e8b28b40 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b23dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b23dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54774/0 (socket says 172.21.15.146:54774) 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 -- 172.21.15.146:0/2431237981 learned_addr learned my addr 172.21.15.146:0/2431237981 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b03e00640 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b30130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09200640 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b2b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 -- 172.21.15.146:0/2431237981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 msgr2=0x5558e8b2b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b2b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 -- 172.21.15.146:0/2431237981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 msgr2=0x5558e8b30130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b30130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09c00640 1 -- 172.21.15.146:0/2431237981 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e886a7d0 con 0x5558e8b33b30 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b09200640 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b2b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.958+0000 7f4b03e00640 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b30130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.959+0000 7f4b09c00640 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b23dd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4afc09ccf0 tx=0x7f4afc09c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2ac4a564fef08e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.959+0000 7f4af5600640 1 -- 172.21.15.146:0/2431237981 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc09d460 con 0x5558e8b33b30 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.959+0000 7f4af5600640 1 -- 172.21.15.146:0/2431237981 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4afc09cd60 con 0x5558e8b33b30 2024-04-24T08:22:24.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.959+0000 7f4af5600640 1 -- 172.21.15.146:0/2431237981 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc0bd0c0 con 0x5558e8b33b30 2024-04-24T08:22:24.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2431237981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 msgr2=0x5558e8b23dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b23dd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4afc09ccf0 tx=0x7f4afc09c8a0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2431237981 shutdown_connections 2024-04-24T08:22:24.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b23dd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b30130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/2431237981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b2b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2431237981 >> 172.21.15.146:0/2431237981 conn(0x5558e8b24e60 msgr2=0x5558e8b236c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2431237981 shutdown_connections 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.960+0000 7f4b0ebae740 1 -- 172.21.15.146:0/2431237981 wait complete. 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.961+0000 7f4b0ebae740 1 -- start start 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.961+0000 7f4b0ebae740 1 Processor -- start 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b3bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b3c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b0ebae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b2dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041b2810 con 0x5558e8b33b30 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041b3c80 con 0x5558e8b29460 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b0ebae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041b2ee0 con 0x5558e8b28b40 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b0ebae740 10 client.0 _ll_register_callbacks cb 0x7f4af800f4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b03e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b3c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b3bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b2dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b3bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56234/0 (socket says 172.21.15.146:56234) 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b2dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54780/0 (socket says 172.21.15.146:54780) 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09c00640 1 -- 172.21.15.146:0/1368809088 learned_addr learned my addr 172.21.15.146:0/1368809088 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b03e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b3c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:42922/0 (socket says 172.21.15.146:42922) 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09200640 1 -- 172.21.15.146:0/1368809088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 msgr2=0x5558e8b3c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09200640 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b3c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09200640 1 -- 172.21.15.146:0/1368809088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 msgr2=0x5558e8b2dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.962+0000 7f4b09200640 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b2dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.963+0000 7f4b09200640 1 -- 172.21.15.146:0/1368809088 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e8b2e300 con 0x5558e8b28b40 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.963+0000 7f4b09200640 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b3bd30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4af8019040 tx=0x7f4af8005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31759c46faf8378e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:24.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.963+0000 7f4aeaa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.963+0000 7f4aeaa00640 1 -- 172.21.15.146:0/1368809088 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8040060 con 0x5558e8b28b40 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.963+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4afc09cd60 con 0x5558e8b28b40 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.963+0000 7f4aeaa00640 1 -- 172.21.15.146:0/1368809088 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4af8012e40 con 0x5558e8b28b40 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.963+0000 7f4aeaa00640 1 -- 172.21.15.146:0/1368809088 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4af801ea00 con 0x5558e8b28b40 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 10 client.25603 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5558e8b34c10 con 0x5558e8b28b40 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4aeaa00640 1 -- 172.21.15.146:0/1368809088 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4afc09cd60 con 0x5558e8b28b40 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4aeaa00640 1 -- 172.21.15.146:0/1368809088 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f4af80313a0 con 0x5558e8b28b40 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 10 client.25603 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 10 client.25603 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:22:24.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 20 client.25603 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8baa420 0x5558e8bac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 4 client.25603 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558e8a95a60 con 0x5558e8baa420 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.964+0000 7f4b03e00640 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8baa420 0x5558e8bac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.965+0000 7f4b03e00640 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8baa420 0x5558e8bac810 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e1adba42228b56b4 server_cookie=8c0bbd9ffd700b72 in_seq=0 out_seq=0 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.965+0000 7f4aeaa00640 10 client.25603 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4aeaa00640 1 -- 172.21.15.146:0/1368809088 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558e8a95a60 con 0x5558e8baa420 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4aeaa00640 10 client.25603 handle_command_reply: tid=0 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 client.25603 shutdown 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 msgr2=0x5558e8b3bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b3bd30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4af8019040 tx=0x7f4af8005db0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 shutdown_connections 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5558e8baa420 0x5558e8bac810 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558e8b33b30 0x5558e8b2dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5558e8b29460 0x5558e8b3c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 --2- 172.21.15.146:0/1368809088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558e8b28b40 0x5558e8b3bd30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 >> 172.21.15.146:0/1368809088 conn(0x5558e8b24e60 msgr2=0x5558e8b23f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4aeaa00640 0 client.25603 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 shutdown_connections 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.966+0000 7f4b0ebae740 1 -- 172.21.15.146:0/1368809088 wait complete. 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0ebae740 20 client.25603 trim_cache size 0 max 16384 2024-04-24T08:22:24.971 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4aec0a3ed0 msgr2=0x7f4aec0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4aec0a3ed0 0x7f4aec0a6390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x5558e8b2a100 tx=0x7f4af8005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b0410c4d0 msgr2=0x7f4b0419a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b0410c4d0 0x7f4b0419a7d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00ec50 tx=0x7f4afc00c6a0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 shutdown_connections 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4aec0a3ed0 0x7f4aec0a6390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b0410c4d0 0x7f4b0419a7d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b04103a20 0x7f4b041a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 --2- 172.21.15.146:0/4147460597 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b04103100 0x7f4b041a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:24.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.967+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 >> 172.21.15.146:0/4147460597 conn(0x7f4b040fc880 msgr2=0x7f4b04100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.968+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 shutdown_connections 2024-04-24T08:22:24.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:24.968+0000 7f4b0a600640 1 -- 172.21.15.146:0/4147460597 wait complete. 2024-04-24T08:22:24.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:22:25.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.211+0000 7f3ede400640 1 -- 172.21.15.146:0/759583960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed81093f0 msgr2=0x7f3ed8110fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.211+0000 7f3ede400640 1 --2- 172.21.15.146:0/759583960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed81093f0 0x7f3ed8110fc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009ca0 tx=0x7f3ed002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.212+0000 7f3ede400640 1 -- 172.21.15.146:0/759583960 shutdown_connections 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.212+0000 7f3ede400640 1 --2- 172.21.15.146:0/759583960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed8104cd0 0x7f3ed8108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.212+0000 7f3ede400640 1 --2- 172.21.15.146:0/759583960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed81093f0 0x7f3ed8110fc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.212+0000 7f3ede400640 1 --2- 172.21.15.146:0/759583960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ed8104320 0x7f3ed8104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.212+0000 7f3ede400640 1 -- 172.21.15.146:0/759583960 >> 172.21.15.146:0/759583960 conn(0x7f3ed80ffb40 msgr2=0x7f3ed8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.212+0000 7f3ede400640 1 -- 172.21.15.146:0/759583960 shutdown_connections 2024-04-24T08:22:25.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.212+0000 7f3ede400640 1 -- 172.21.15.146:0/759583960 wait complete. 2024-04-24T08:22:25.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.213+0000 7f3ede400640 1 Processor -- start 2024-04-24T08:22:25.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.213+0000 7f3ede400640 1 -- start start 2024-04-24T08:22:25.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.213+0000 7f3ede400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed8104320 0x7f3ed81a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.213+0000 7f3ede400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed81093f0 0x7f3ed81a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3ede400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ed819a7b0 0x7f3ed819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3ede400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed8114490 con 0x7f3ed819a7b0 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3ede400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed8114310 con 0x7f3ed8104320 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3ede400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed8114610 con 0x7f3ed81093f0 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed8104320 0x7f3ed81a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed8104320 0x7f3ed81a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:42936/0 (socket says 172.21.15.146:42936) 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 -- 172.21.15.146:0/2307200414 learned_addr learned my addr 172.21.15.146:0/2307200414 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edda00640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ed819a7b0 0x7f3ed819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed81093f0 0x7f3ed81a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 -- 172.21.15.146:0/2307200414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed81093f0 msgr2=0x7f3ed81a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed81093f0 0x7f3ed81a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 -- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ed819a7b0 msgr2=0x7f3ed819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ed819a7b0 0x7f3ed819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.214+0000 7f3edd000640 1 -- 172.21.15.146:0/2307200414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed819b340 con 0x7f3ed8104320 2024-04-24T08:22:25.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.215+0000 7f3edd000640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed8104320 0x7f3ed81a0640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00ea60 tx=0x7f3ecc00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.215+0000 7f3ed5600640 1 -- 172.21.15.146:0/2307200414 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00cde0 con 0x7f3ed8104320 2024-04-24T08:22:25.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.215+0000 7f3ed5600640 1 -- 172.21.15.146:0/2307200414 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ecc0035f0 con 0x7f3ed8104320 2024-04-24T08:22:25.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.215+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed819b630 con 0x7f3ed8104320 2024-04-24T08:22:25.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.215+0000 7f3ed5600640 1 -- 172.21.15.146:0/2307200414 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc010670 con 0x7f3ed8104320 2024-04-24T08:22:25.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.215+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed81a7860 con 0x7f3ed8104320 2024-04-24T08:22:25.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 Processor -- start 2024-04-24T08:22:25.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ed5600640 1 -- 172.21.15.146:0/2307200414 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 138931+0+0 (secure 0 0 0) 0x7f3ecc0037a0 con 0x7f3ed8104320 2024-04-24T08:22:25.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 -- start start 2024-04-24T08:22:25.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906feff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ff0460 0x55e906ffa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ffab30 0x55e906ffcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e906f551d0 con 0x55e906ffab30 2024-04-24T08:22:25.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e906efb450 con 0x55e906fefb40 2024-04-24T08:22:25.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.220+0000 7f3ee2858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e906f51190 con 0x55e906ff0460 2024-04-24T08:22:25.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ff0460 0x55e906ffa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edda00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906feff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ff0460 0x55e906ffa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56268/0 (socket says 172.21.15.146:56268) 2024-04-24T08:22:25.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edd000640 1 -- 172.21.15.146:0/2316684247 learned_addr learned my addr 172.21.15.146:0/2316684247 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ffab30 0x55e906ffcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edda00640 1 -- 172.21.15.146:0/2316684247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ff0460 msgr2=0x55e906ffa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edda00640 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ff0460 0x55e906ffa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edda00640 1 -- 172.21.15.146:0/2316684247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ffab30 msgr2=0x55e906ffcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edda00640 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ffab30 0x55e906ffcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.221+0000 7f3edda00640 1 -- 172.21.15.146:0/2316684247 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e906fb8310 con 0x55e906fefb40 2024-04-24T08:22:25.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.222+0000 7f3edda00640 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906feff20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006010 tx=0x7f3ed0005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f88261fb0dd79499 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.222+0000 7f3ed5600640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ec00a3f50 0x7f3ec00a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.222+0000 7f3ed5600640 1 -- 172.21.15.146:0/2307200414 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3ecc0d0d70 con 0x7f3ed8104320 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.222+0000 7f3eca000640 1 -- 172.21.15.146:0/2316684247 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0031db0 con 0x55e906fefb40 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.222+0000 7f3edd000640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ec00a3f50 0x7f3ec00a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.222+0000 7f3eca000640 1 -- 172.21.15.146:0/2316684247 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed0036040 con 0x55e906fefb40 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.222+0000 7f3eca000640 1 -- 172.21.15.146:0/2316684247 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed00042a0 con 0x55e906fefb40 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 -- 172.21.15.146:0/2316684247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 msgr2=0x55e906feff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3edd000640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ec00a3f50 0x7f3ec00a6410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0a5220 tx=0x7f3ecc0d4840 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906feff20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0006010 tx=0x7f3ed0005f10 comp rx=0 tx=0).stop 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 -- 172.21.15.146:0/2316684247 shutdown_connections 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ff0460 0x55e906ffa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ffab30 0x55e906ffcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2316684247 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906feff20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 -- 172.21.15.146:0/2316684247 >> 172.21.15.146:0/2316684247 conn(0x55e906febe60 msgr2=0x55e906fec240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 -- 172.21.15.146:0/2316684247 shutdown_connections 2024-04-24T08:22:25.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.223+0000 7f3ee2858740 1 -- 172.21.15.146:0/2316684247 wait complete. 2024-04-24T08:22:25.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.224+0000 7f3ee2858740 1 -- start start 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.224+0000 7f3ee2858740 1 Processor -- start 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.224+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.224+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ff78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.224+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ff8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3ee2858740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e906ffd7e0 con 0x55e906ff0460 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3ee2858740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e906f0e810 con 0x55e906fefb40 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3ee2858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e906ffd960 con 0x55e906ffab30 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3ee2858740 10 client.0 _ll_register_callbacks cb 0x55e906edc140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3edda00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3edda00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffa5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:42958/0 (socket says 172.21.15.146:42958) 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3edda00640 1 -- 172.21.15.146:0/2936431132 learned_addr learned my addr 172.21.15.146:0/2936431132 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3edda00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ff8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3edda00640 1 -- 172.21.15.146:0/2936431132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 msgr2=0x55e906ff8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3edda00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ff8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.225+0000 7f3edda00640 1 -- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 msgr2=0x55e906ff78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ff78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3edda00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ff78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3edda00640 1 -- 172.21.15.146:0/2936431132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e906f1f3f0 con 0x55e906fefb40 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ff78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3edda00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffa5e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0031ac0 tx=0x7f3ed00319c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9466fabe98165a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3ebea00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3ebea00640 1 -- 172.21.15.146:0/2936431132 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0058060 con 0x55e906fefb40 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3ebea00640 1 -- 172.21.15.146:0/2936431132 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3ed002f6d0 con 0x55e906fefb40 2024-04-24T08:22:25.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.226+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ed0036040 con 0x55e906fefb40 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ebea00640 1 -- 172.21.15.146:0/2936431132 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ed002f8d0 con 0x55e906fefb40 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 10 client.25616 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e906c673f0 con 0x55e906fefb40 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ebea00640 1 -- 172.21.15.146:0/2936431132 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed00305c0 con 0x55e906fefb40 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ebea00640 1 -- 172.21.15.146:0/2936431132 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3ed004d030 con 0x55e906fefb40 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 10 client.25616 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 10 client.25616 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 20 client.25616 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e90700aa10 0x55e90700ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 4 client.25616 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3ed0036040 con 0x55e90700aa10 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.227+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e90700aa10 0x55e90700ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.228+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e90700aa10 0x55e90700ce00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31a7a11aa1e5d764 server_cookie=e81516bf60c0520d in_seq=0 out_seq=0 2024-04-24T08:22:25.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.228+0000 7f3ebea00640 10 client.25616 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.229+0000 7f3ebea00640 1 -- 172.21.15.146:0/2936431132 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f3ed0036040 con 0x55e90700aa10 2024-04-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.229+0000 7f3ebea00640 10 client.25616 handle_command_reply: tid=0 2024-04-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.229+0000 7f3ee2858740 1 client.25616 shutdown 2024-04-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 msgr2=0x55e906ffa5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffa5e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0031ac0 tx=0x7f3ed00319c0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 shutdown_connections 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e90700aa10 0x55e90700ce00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ff8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ff78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 --2- 172.21.15.146:0/2936431132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffa5e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 >> 172.21.15.146:0/2936431132 conn(0x55e906febe60 msgr2=0x55e906ff1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ebea00640 0 client.25616 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 shutdown_connections 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 1 -- 172.21.15.146:0/2936431132 wait complete. 2024-04-24T08:22:25.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.230+0000 7f3ee2858740 20 client.25616 trim_cache size 0 max 16384 2024-04-24T08:22:25.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:25 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3143734695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:25.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.250+0000 7f3ee2858740 1 Processor -- start 2024-04-24T08:22:25.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.250+0000 7f3ee2858740 1 -- start start 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906fefb40 0x55e906ff7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e9070066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906ffab30 0x55e906ffa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ee2858740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81b2690 con 0x55e906ff0460 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ee2858740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81aff60 con 0x55e906ffab30 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ee2858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81addb0 con 0x55e906fefb40 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e9070066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3edda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906fefb40 0x55e906ff7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e9070066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54820/0 (socket says 172.21.15.146:54820) 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3edda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906fefb40 0x55e906ff7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56292/0 (socket says 172.21.15.146:56292) 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3ed7e00640 1 -- 172.21.15.146:0/1642585364 learned_addr learned my addr 172.21.15.146:0/1642585364 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.251+0000 7f3edda00640 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906ffab30 0x55e906ffa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3ed7e00640 1 -- 172.21.15.146:0/1642585364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906fefb40 msgr2=0x55e906ff7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906fefb40 0x55e906ff7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3ed7e00640 1 -- 172.21.15.146:0/1642585364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906ffab30 msgr2=0x55e906ffa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906ffab30 0x55e906ffa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3ed7e00640 1 -- 172.21.15.146:0/1642585364 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e906d317d0 con 0x55e906ff0460 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3edda00640 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906fefb40 0x55e906ff7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e9070066a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3ed810a9c0 tx=0x7f3ec4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dab9c73e4aaf59e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.252+0000 7f3ec9600640 1 -- 172.21.15.146:0/1642585364 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4004050 con 0x55e906ff0460 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ec9600640 1 -- 172.21.15.146:0/1642585364 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec40041f0 con 0x55e906ff0460 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ec9600640 1 -- 172.21.15.146:0/1642585364 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4003b30 con 0x55e906ff0460 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ee2858740 1 -- 172.21.15.146:0/1642585364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 msgr2=0x55e9070066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e9070066a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3ed810a9c0 tx=0x7f3ec4012820 comp rx=0 tx=0).stop 2024-04-24T08:22:25.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ee2858740 1 -- 172.21.15.146:0/1642585364 shutdown_connections 2024-04-24T08:22:25.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906ffab30 0x55e906ffa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e9070066a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1642585364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906fefb40 0x55e906ff7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.253+0000 7f3ee2858740 1 -- 172.21.15.146:0/1642585364 >> 172.21.15.146:0/1642585364 conn(0x55e906febe60 msgr2=0x55e906ff7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.254+0000 7f3ee2858740 1 -- 172.21.15.146:0/1642585364 shutdown_connections 2024-04-24T08:22:25.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.254+0000 7f3ee2858740 1 -- 172.21.15.146:0/1642585364 wait complete. 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.254+0000 7f3ee2858740 1 -- start start 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.254+0000 7f3ee2858740 1 Processor -- start 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ffbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ee2858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ffc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ee2858740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81ae420 con 0x55e906ff0460 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ee2858740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81af890 con 0x55e906fefb40 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ee2858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed81aeaf0 con 0x55e906ffab30 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ee2858740 10 client.0 _ll_register_callbacks cb 0x7f3ecc0d4710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3edda00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ffbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3edda00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:42974/0 (socket says 172.21.15.146:42974) 2024-04-24T08:22:25.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.255+0000 7f3edda00640 1 -- 172.21.15.146:0/1330562452 learned_addr learned my addr 172.21.15.146:0/1330562452 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3edda00640 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ffc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3ed7e00640 1 -- 172.21.15.146:0/1330562452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 msgr2=0x55e906ffc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ffc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3ed7e00640 1 -- 172.21.15.146:0/1330562452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 msgr2=0x55e906ffb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3ed7e00640 1 -- 172.21.15.146:0/1330562452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e907000d00 con 0x55e906ff0460 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3ed7e00640 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ffbe90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4003630 tx=0x7f3ec40096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08790f90662a8f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.256+0000 7f3ebea00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.257+0000 7f3ebea00640 1 -- 172.21.15.146:0/1330562452 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec401d460 con 0x55e906ff0460 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.257+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ec40041f0 con 0x55e906ff0460 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.257+0000 7f3ebea00640 1 -- 172.21.15.146:0/1330562452 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3ec401d600 con 0x55e906ff0460 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.257+0000 7f3ebea00640 1 -- 172.21.15.146:0/1330562452 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ec401d840 con 0x55e906ff0460 2024-04-24T08:22:25.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.257+0000 7f3ee2858740 10 client.15486 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:25.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.257+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e907071020 con 0x55e906ff0460 2024-04-24T08:22:25.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.257+0000 7f3ebea00640 1 -- 172.21.15.146:0/1330562452 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec40041f0 con 0x55e906ff0460 2024-04-24T08:22:25.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3ebea00640 1 -- 172.21.15.146:0/1330562452 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3ec402e060 con 0x55e906ff0460 2024-04-24T08:22:25.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3ee2858740 10 client.15486 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3ee2858740 10 client.15486 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3ee2858740 20 client.15486 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e9070716f0 0x55e907073b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3ee2858740 4 client.15486 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e906f62bf0 con 0x55e9070716f0 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.258+0000 7f3edda00640 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e9070716f0 0x55e907073b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.259+0000 7f3edda00640 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e9070716f0 0x55e907073b10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7266c2e8a7730e4c server_cookie=5f308d185c3dbad2 in_seq=0 out_seq=0 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.259+0000 7f3ebea00640 10 client.15486 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.259+0000 7f3ebea00640 1 -- 172.21.15.146:0/1330562452 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e906f62bf0 con 0x55e9070716f0 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.259+0000 7f3ebea00640 10 client.15486 handle_command_reply: tid=0 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.259+0000 7f3ee2858740 1 client.15486 shutdown 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.259+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 msgr2=0x55e906ffbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.259+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ffbe90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4003630 tx=0x7f3ec40096d0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 shutdown_connections 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55e9070716f0 0x55e907073b10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e906ffab30 0x55e906ffc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e906ff0460 0x55e906ffbe90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 --2- 172.21.15.146:0/1330562452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e906fefb40 0x55e906ffb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 >> 172.21.15.146:0/1330562452 conn(0x55e906febe60 msgr2=0x55e906fea6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ebea00640 0 client.15486 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 shutdown_connections 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 1 -- 172.21.15.146:0/1330562452 wait complete. 2024-04-24T08:22:25.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.260+0000 7f3ee2858740 20 client.15486 trim_cache size 0 max 16384 2024-04-24T08:22:25.265 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:22:25.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ec00a3f50 msgr2=0x7f3ec00a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ec00a3f50 0x7f3ec00a6410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0a5220 tx=0x7f3ecc0d4840 comp rx=0 tx=0).stop 2024-04-24T08:22:25.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed8104320 msgr2=0x7f3ed81a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed8104320 0x7f3ed81a0640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc00ea60 tx=0x7f3ecc00ef20 comp rx=0 tx=0).stop 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 shutdown_connections 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ec00a3f50 0x7f3ec00a6410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed81093f0 0x7f3ed81a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ed819a7b0 0x7f3ed819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 --2- 172.21.15.146:0/2307200414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3ed8104320 0x7f3ed81a0640 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 >> 172.21.15.146:0/2307200414 conn(0x7f3ed80ffb40 msgr2=0x7f3ed8101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 shutdown_connections 2024-04-24T08:22:25.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.261+0000 7f3ede400640 1 -- 172.21.15.146:0/2307200414 wait complete. 2024-04-24T08:22:25.281 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:22:25.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:25 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3143734695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.486+0000 7f1503200640 1 -- 172.21.15.146:0/3052295525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1504102ac0 msgr2=0x7f1504106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.486+0000 7f1503200640 1 --2- 172.21.15.146:0/3052295525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1504102ac0 0x7f1504106b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009860 tx=0x7f14ec02a340 comp rx=0 tx=0).stop 2024-04-24T08:22:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.486+0000 7f1503200640 1 -- 172.21.15.146:0/3052295525 shutdown_connections 2024-04-24T08:22:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.486+0000 7f1503200640 1 --2- 172.21.15.146:0/3052295525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1504102ac0 0x7f1504106b00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.486+0000 7f1503200640 1 --2- 172.21.15.146:0/3052295525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15041071e0 0x7f150410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.486+0000 7f1503200640 1 --2- 172.21.15.146:0/3052295525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 0x7f15041024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.486+0000 7f1503200640 1 -- 172.21.15.146:0/3052295525 >> 172.21.15.146:0/3052295525 conn(0x7f15040fd9d0 msgr2=0x7f15040ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.487+0000 7f1503200640 1 -- 172.21.15.146:0/3052295525 shutdown_connections 2024-04-24T08:22:25.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.487+0000 7f1503200640 1 -- 172.21.15.146:0/3052295525 wait complete. 2024-04-24T08:22:25.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 Processor -- start 2024-04-24T08:22:25.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 -- start start 2024-04-24T08:22:25.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 0x7f15041a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15041071e0 0x7f15041a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f150419a830 0x7f150419ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1504112250 con 0x7f150419a830 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15041120d0 con 0x7f1504102110 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.488+0000 7f1503200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15041123d0 con 0x7f15041071e0 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 0x7f15041a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 0x7f15041a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:42990/0 (socket says 172.21.15.146:42990) 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 -- 172.21.15.146:0/2342630766 learned_addr learned my addr 172.21.15.146:0/2342630766 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1502800640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f150419a830 0x7f150419ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501400640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15041071e0 0x7f15041a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 -- 172.21.15.146:0/2342630766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15041071e0 msgr2=0x7f15041a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15041071e0 0x7f15041a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 -- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f150419a830 msgr2=0x7f150419ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f150419a830 0x7f150419ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501e00640 1 -- 172.21.15.146:0/2342630766 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150419b3c0 con 0x7f1504102110 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.489+0000 7f1501400640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15041071e0 0x7f15041a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.490+0000 7f1501e00640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 0x7f15041a06c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f14f400db80 tx=0x7f14f400df80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.490+0000 7f14faa00640 1 -- 172.21.15.146:0/2342630766 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f4017070 con 0x7f1504102110 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.490+0000 7f14faa00640 1 -- 172.21.15.146:0/2342630766 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14f400be30 con 0x7f1504102110 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.490+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150419b6b0 con 0x7f1504102110 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.490+0000 7f14faa00640 1 -- 172.21.15.146:0/2342630766 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f400f880 con 0x7f1504102110 2024-04-24T08:22:25.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.490+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15041a7860 con 0x7f1504102110 2024-04-24T08:22:25.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.493+0000 7f150be5a740 1 Processor -- start 2024-04-24T08:22:25.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.493+0000 7f14faa00640 1 -- 172.21.15.146:0/2342630766 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 138931+0+0 (secure 0 0 0) 0x7f14f401a070 con 0x7f1504102110 2024-04-24T08:22:25.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.493+0000 7f150be5a740 1 -- start start 2024-04-24T08:22:25.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f150be5a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9d3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f150be5a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f150be5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9e0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f150be5a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb9391d0 con 0x55dbeb9d3b40 2024-04-24T08:22:25.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f150be5a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb8df450 con 0x55dbeb9d4460 2024-04-24T08:22:25.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f150be5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb935190 con 0x55dbeb9deb30 2024-04-24T08:22:25.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f1501400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9e0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f1501e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f1501400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9e0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56316/0 (socket says 172.21.15.146:56316) 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.494+0000 7f1501400640 1 -- 172.21.15.146:0/3851260687 learned_addr learned my addr 172.21.15.146:0/3851260687 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.495+0000 7f1501400640 1 -- 172.21.15.146:0/3851260687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d4460 msgr2=0x55dbeb9de5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.495+0000 7f1501400640 1 --2- 172.21.15.146:0/3851260687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.495+0000 7f1501400640 1 -- 172.21.15.146:0/3851260687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d3b40 msgr2=0x55dbeb9d3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.495+0000 7f1501400640 1 --2- 172.21.15.146:0/3851260687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9d3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.495+0000 7f1501400640 1 -- 172.21.15.146:0/3851260687 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbeb99c310 con 0x55dbeb9deb30 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.495+0000 7f1501400640 1 --2- 172.21.15.146:0/3851260687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9e0f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009860 tx=0x7f14ec0303b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d500c2de75f39acb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.495+0000 7f14faa00640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14e80a3ed0 0x7f14e80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.496+0000 7f14faa00640 1 -- 172.21.15.146:0/2342630766 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f14f40d0ae0 con 0x7f1504102110 2024-04-24T08:22:25.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.496+0000 7f14f3400640 1 -- 172.21.15.146:0/3851260687 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec037040 con 0x55dbeb9deb30 2024-04-24T08:22:25.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.496+0000 7f1502800640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14e80a3ed0 0x7f14e80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.496+0000 7f14f3400640 1 -- 172.21.15.146:0/3851260687 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec030550 con 0x55dbeb9deb30 2024-04-24T08:22:25.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.496+0000 7f14f3400640 1 -- 172.21.15.146:0/3851260687 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec005430 con 0x55dbeb9deb30 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 -- 172.21.15.146:0/3851260687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 msgr2=0x55dbeb9e0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 --2- 172.21.15.146:0/3851260687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9e0f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009860 tx=0x7f14ec0303b0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f1502800640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14e80a3ed0 0x7f14e80a6390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14fc005fd0 tx=0x7f14fc011810 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 -- 172.21.15.146:0/3851260687 shutdown_connections 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 --2- 172.21.15.146:0/3851260687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9e0f20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 --2- 172.21.15.146:0/3851260687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 --2- 172.21.15.146:0/3851260687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9d3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 -- 172.21.15.146:0/3851260687 >> 172.21.15.146:0/3851260687 conn(0x55dbeb9cfe60 msgr2=0x55dbeb9d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 -- 172.21.15.146:0/3851260687 shutdown_connections 2024-04-24T08:22:25.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.497+0000 7f150be5a740 1 -- 172.21.15.146:0/3851260687 wait complete. 2024-04-24T08:22:25.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.498+0000 7f150be5a740 1 -- start start 2024-04-24T08:22:25.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.498+0000 7f150be5a740 1 Processor -- start 2024-04-24T08:22:25.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f150be5a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f150be5a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9db830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f150be5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f150be5a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb9ea170 con 0x55dbeb9d4460 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f150be5a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb8f2810 con 0x55dbeb9d3b40 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f150be5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dbeb9ea2f0 con 0x55dbeb9deb30 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f150be5a740 10 client.0 _ll_register_callbacks cb 0x55dbeb8c0140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9de540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43004/0 (socket says 172.21.15.146:43004) 2024-04-24T08:22:25.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 -- 172.21.15.146:0/2604591527 learned_addr learned my addr 172.21.15.146:0/2604591527 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501400640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1502800640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 -- 172.21.15.146:0/2604591527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 msgr2=0x55dbeb9dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 -- 172.21.15.146:0/2604591527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 msgr2=0x55dbeb9db830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9db830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501e00640 1 -- 172.21.15.146:0/2604591527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbeb9033f0 con 0x55dbeb9d3b40 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.499+0000 7f1501400640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9db830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f1501e00640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9de540 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f14f400b6f0 tx=0x7f14f4009a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92160471fea60058 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f14e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f14e7e00640 1 -- 172.21.15.146:0/2604591527 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f4021030 con 0x55dbeb9d3b40 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f14e7e00640 1 -- 172.21.15.146:0/2604591527 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f14f40870e0 con 0x55dbeb9d3b40 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14ec030550 con 0x55dbeb9d3b40 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f14e7e00640 1 -- 172.21.15.146:0/2604591527 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f14f409c1b0 con 0x55dbeb9d3b40 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f150be5a740 10 client.25634 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:25.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dbeb64b3f0 con 0x55dbeb9d3b40 2024-04-24T08:22:25.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.500+0000 7f14e7e00640 1 -- 172.21.15.146:0/2604591527 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec030550 con 0x55dbeb9d3b40 2024-04-24T08:22:25.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f14e7e00640 1 -- 172.21.15.146:0/2604591527 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f14f40b2020 con 0x55dbeb9d3b40 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f150be5a740 10 client.25634 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f150be5a740 10 client.25634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f150be5a740 20 client.25634 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f150be5a740 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeb9f74e0 0x55dbeb9f98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f150be5a740 4 client.25634 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbeb946bf0 con 0x55dbeb9f74e0 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.501+0000 7f1501400640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeb9f74e0 0x55dbeb9f98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.502+0000 7f1501400640 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeb9f74e0 0x55dbeb9f98d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d228c5b59cf86f06 server_cookie=38551395d19e2871 in_seq=0 out_seq=0 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.502+0000 7f14e7e00640 10 client.25634 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f14e7e00640 1 -- 172.21.15.146:0/2604591527 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbeb946bf0 con 0x55dbeb9f74e0 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f14e7e00640 10 client.25634 handle_command_reply: tid=0 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 client.25634 shutdown 2024-04-24T08:22:25.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 msgr2=0x55dbeb9de540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9de540 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f14f400b6f0 tx=0x7f14f4009a80 comp rx=0 tx=0).stop 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 shutdown_connections 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeb9f74e0 0x55dbeb9f98d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9db830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f14e7e00640 0 client.25634 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 --2- 172.21.15.146:0/2604591527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9de540 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 >> 172.21.15.146:0/2604591527 conn(0x55dbeb9cfe60 msgr2=0x55dbeb9d5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 shutdown_connections 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.503+0000 7f150be5a740 1 -- 172.21.15.146:0/2604591527 wait complete. 2024-04-24T08:22:25.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.504+0000 7f150be5a740 20 client.25634 trim_cache size 0 max 16384 2024-04-24T08:22:25.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 Processor -- start 2024-04-24T08:22:25.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 -- start start 2024-04-24T08:22:25.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9f3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9ff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f15041ae0c0 con 0x55dbeb9d4460 2024-04-24T08:22:25.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15041ab990 con 0x55dbeb9d3b40 2024-04-24T08:22:25.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f150be5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15041a9860 con 0x55dbeb9deb30 2024-04-24T08:22:25.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.518+0000 7f1501400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54876/0 (socket says 172.21.15.146:54876) 2024-04-24T08:22:25.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 -- 172.21.15.146:0/1312560805 learned_addr learned my addr 172.21.15.146:0/1312560805 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501e00640 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1502800640 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9ff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 -- 172.21.15.146:0/1312560805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 msgr2=0x55dbeb9ff0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9ff0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 -- 172.21.15.146:0/1312560805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 msgr2=0x55dbeb9f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 -- 172.21.15.146:0/1312560805 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbeb7157d0 con 0x55dbeb9d4460 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501e00640 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.519+0000 7f1501400640 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de070 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f14ec005e00 tx=0x7f14ec02a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9e770d70bdc73e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.520+0000 7f14f2a00640 1 -- 172.21.15.146:0/1312560805 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec037040 con 0x55dbeb9d4460 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.520+0000 7f14f2a00640 1 -- 172.21.15.146:0/1312560805 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec030550 con 0x55dbeb9d4460 2024-04-24T08:22:25.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.520+0000 7f14f2a00640 1 -- 172.21.15.146:0/1312560805 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0053d0 con 0x55dbeb9d4460 2024-04-24T08:22:25.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 -- 172.21.15.146:0/1312560805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 msgr2=0x55dbeb9de070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de070 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f14ec005e00 tx=0x7f14ec02a340 comp rx=0 tx=0).stop 2024-04-24T08:22:25.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 -- 172.21.15.146:0/1312560805 shutdown_connections 2024-04-24T08:22:25.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9deb30 0x55dbeb9ff0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9de070 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 --2- 172.21.15.146:0/1312560805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 -- 172.21.15.146:0/1312560805 >> 172.21.15.146:0/1312560805 conn(0x55dbeb9cfe60 msgr2=0x55dbeb9d53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 -- 172.21.15.146:0/1312560805 shutdown_connections 2024-04-24T08:22:25.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 -- 172.21.15.146:0/1312560805 wait complete. 2024-04-24T08:22:25.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.521+0000 7f150be5a740 1 -- start start 2024-04-24T08:22:25.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 1 Processor -- start 2024-04-24T08:22:25.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9ef730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9d8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9deb30 0x55dbeb9d8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f15041a9eb0 con 0x55dbeb9d4460 2024-04-24T08:22:25.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f15041ab2c0 con 0x55dbeb9deb30 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15041aa520 con 0x55dbeb9d3b40 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.522+0000 7f150be5a740 10 client.0 _ll_register_callbacks cb 0x7f14f4004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9d8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9ef730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56338/0 (socket says 172.21.15.146:56338) 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 -- 172.21.15.146:0/396362280 learned_addr learned my addr 172.21.15.146:0/396362280 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1502800640 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9deb30 0x55dbeb9d8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 -- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9deb30 msgr2=0x55dbeb9d8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9deb30 0x55dbeb9d8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 -- 172.21.15.146:0/396362280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 msgr2=0x55dbeb9d8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9d8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 -- 172.21.15.146:0/396362280 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbeb9d95e0 con 0x55dbeb9d3b40 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501400640 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f1501e00640 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9ef730 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f14f400db50 tx=0x7f14f409dbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35bd2e7e577ae67d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:25.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.523+0000 7f14e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f14e7e00640 1 -- 172.21.15.146:0/396362280 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f4021070 con 0x55dbeb9d3b40 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f14e7e00640 1 -- 172.21.15.146:0/396362280 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f14f40d4220 con 0x55dbeb9d3b40 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14ec030550 con 0x55dbeb9d3b40 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f14e7e00640 1 -- 172.21.15.146:0/396362280 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f14f4015070 con 0x55dbeb9d3b40 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f150be5a740 10 client.25639 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dbeb9e0360 con 0x55dbeb9d3b40 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f14e7e00640 1 -- 172.21.15.146:0/396362280 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec030550 con 0x55dbeb9d3b40 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f14e7e00640 1 -- 172.21.15.146:0/396362280 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f14f40b2070 con 0x55dbeb9d3b40 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.524+0000 7f150be5a740 10 client.25639 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f150be5a740 10 client.25639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f150be5a740 20 client.25639 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f150be5a740 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeba5df10 0x55dbeba60330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f150be5a740 4 client.25639 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dbeb940a60 con 0x55dbeba5df10 2024-04-24T08:22:25.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f1501400640 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeba5df10 0x55dbeba60330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f14e7e00640 10 client.25639 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.525+0000 7f1501400640 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeba5df10 0x55dbeba60330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5171bbe4be085cb server_cookie=b4c8b35ef5d03d17 in_seq=0 out_seq=0 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f14e7e00640 1 -- 172.21.15.146:0/396362280 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dbeb940a60 con 0x55dbeba5df10 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f14e7e00640 10 client.25639 handle_command_reply: tid=0 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 client.25639 shutdown 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9d3b40 msgr2=0x55dbeb9ef730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9ef730 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f14f400db50 tx=0x7f14f409dbf0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 shutdown_connections 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dbeba5df10 0x55dbeba60330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dbeb9deb30 0x55dbeb9d8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dbeb9d4460 0x55dbeb9d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 --2- 172.21.15.146:0/396362280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dbeb9d3b40 0x55dbeb9ef730 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 >> 172.21.15.146:0/396362280 conn(0x55dbeb9cfe60 msgr2=0x55dbeb9d5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f14e7e00640 0 client.25639 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:25.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 shutdown_connections 2024-04-24T08:22:25.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 1 -- 172.21.15.146:0/396362280 wait complete. 2024-04-24T08:22:25.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.526+0000 7f150be5a740 20 client.25639 trim_cache size 0 max 16384 2024-04-24T08:22:25.533 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:22:25.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14e80a3ed0 msgr2=0x7f14e80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14e80a3ed0 0x7f14e80a6390 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14fc005fd0 tx=0x7f14fc011810 comp rx=0 tx=0).stop 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 msgr2=0x7f15041a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 0x7f15041a06c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f14f400db80 tx=0x7f14f400df80 comp rx=0 tx=0).stop 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 shutdown_connections 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14e80a3ed0 0x7f14e80a6390 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15041071e0 0x7f15041a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f150419a830 0x7f150419ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 --2- 172.21.15.146:0/2342630766 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1504102110 0x7f15041a06c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 >> 172.21.15.146:0/2342630766 conn(0x7f15040fd9d0 msgr2=0x7f15040ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 shutdown_connections 2024-04-24T08:22:25.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:25.527+0000 7f1503200640 1 -- 172.21.15.146:0/2342630766 wait complete. 2024-04-24T08:22:25.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:25 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3143734695' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:26.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:26 smithi153 ceph-mon[32746]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:26.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:26 smithi146 ceph-mon[27805]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:26.545 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:22:26.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:22:26.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:22:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:26 smithi177 ceph-mon[33046]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 -- 172.21.15.146:0/657520915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58881093f0 msgr2=0x7f5888110fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 --2- 172.21.15.146:0/657520915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58881093f0 0x7f5888110fc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5880009be0 tx=0x7f588002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 -- 172.21.15.146:0/657520915 shutdown_connections 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 --2- 172.21.15.146:0/657520915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58881093f0 0x7f5888110fc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 --2- 172.21.15.146:0/657520915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5888104cd0 0x7f5888108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 --2- 172.21.15.146:0/657520915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5888104320 0x7f5888104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 -- 172.21.15.146:0/657520915 >> 172.21.15.146:0/657520915 conn(0x7f58880ffb40 msgr2=0x7f5888101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 -- 172.21.15.146:0/657520915 shutdown_connections 2024-04-24T08:22:26.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.747+0000 7f588f000640 1 -- 172.21.15.146:0/657520915 wait complete. 2024-04-24T08:22:26.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.748+0000 7f588f000640 1 Processor -- start 2024-04-24T08:22:26.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.748+0000 7f588f000640 1 -- start start 2024-04-24T08:22:26.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5888104320 0x7f5888076510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588f000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58881093f0 0x7f5888076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588f000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5888076f90 0x7f58881aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5888114440 con 0x7f5888104320 2024-04-24T08:22:26.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588f000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58881142c0 con 0x7f58881093f0 2024-04-24T08:22:26.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588f000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f58881145c0 con 0x7f5888076f90 2024-04-24T08:22:26.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58881093f0 0x7f5888076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58881093f0 0x7f5888076a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43034/0 (socket says 172.21.15.146:43034) 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588d200640 1 -- 172.21.15.146:0/2780811205 learned_addr learned my addr 172.21.15.146:0/2780811205 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.749+0000 7f588e600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5888076f90 0x7f58881aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588dc00640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5888104320 0x7f5888076510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588d200640 1 -- 172.21.15.146:0/2780811205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5888076f90 msgr2=0x7f58881aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588d200640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5888076f90 0x7f58881aaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588d200640 1 -- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5888104320 msgr2=0x7f5888076510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588d200640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5888104320 0x7f5888076510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588d200640 1 -- 172.21.15.146:0/2780811205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58881ab490 con 0x7f58881093f0 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588dc00640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5888104320 0x7f5888076510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.750+0000 7f588d200640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58881093f0 0x7f5888076a50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f587400e9c0 tx=0x7f587400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.751+0000 7f5886000640 1 -- 172.21.15.146:0/2780811205 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587400cd80 con 0x7f58881093f0 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.751+0000 7f5886000640 1 -- 172.21.15.146:0/2780811205 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5874005500 con 0x7f58881093f0 2024-04-24T08:22:26.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.751+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58881ab780 con 0x7f58881093f0 2024-04-24T08:22:26.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.751+0000 7f5886000640 1 -- 172.21.15.146:0/2780811205 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874010690 con 0x7f58881093f0 2024-04-24T08:22:26.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.751+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58881abca0 con 0x7f58881093f0 2024-04-24T08:22:26.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.755+0000 7f589357f740 1 Processor -- start 2024-04-24T08:22:26.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.755+0000 7f589357f740 1 -- start start 2024-04-24T08:22:26.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.755+0000 7f589357f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbdfc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f589357f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe06d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f589357f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe09650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f589357f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558efbd61e70 con 0x558efbdfcb90 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f589357f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558efbd08380 con 0x558efbdfc270 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f589357f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558efbd5de30 con 0x558efbe07260 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbdfc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe06d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbdfc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43036/0 (socket says 172.21.15.146:43036) 2024-04-24T08:22:26.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 -- 172.21.15.146:0/4021894721 learned_addr learned my addr 172.21.15.146:0/4021894721 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f5886000640 1 -- 172.21.15.146:0/2780811205 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 26) ==== 138931+0+0 (secure 0 0 0) 0x7f58740046c0 con 0x7f58881093f0 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 -- 172.21.15.146:0/4021894721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 msgr2=0x558efbe09650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe09650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 -- 172.21.15.146:0/4021894721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 msgr2=0x558efbe06d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe06d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588e600640 1 -- 172.21.15.146:0/4021894721 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558efbb3c7e0 con 0x558efbdfc270 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.756+0000 7f588dc00640 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe06d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.757+0000 7f588e600640 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbdfc650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5880009be0 tx=0x7f5880005c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b156b73aa34e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:26.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.757+0000 7f587aa00640 1 -- 172.21.15.146:0/4021894721 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880042030 con 0x558efbdfc270 2024-04-24T08:22:26.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.757+0000 7f587aa00640 1 -- 172.21.15.146:0/4021894721 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5880037040 con 0x558efbdfc270 2024-04-24T08:22:26.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.757+0000 7f587aa00640 1 -- 172.21.15.146:0/4021894721 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880036070 con 0x558efbdfc270 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f5886000640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f58700a3f50 0x7f58700a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f589357f740 1 -- 172.21.15.146:0/4021894721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 msgr2=0x558efbdfc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f589357f740 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbdfc650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5880009be0 tx=0x7f5880005c60 comp rx=0 tx=0).stop 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f5886000640 1 -- 172.21.15.146:0/2780811205 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f58740d0e50 con 0x7f58881093f0 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f588dc00640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f58700a3f50 0x7f58700a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f589357f740 1 -- 172.21.15.146:0/4021894721 shutdown_connections 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f589357f740 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe09650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f589357f740 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe06d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f589357f740 1 --2- 172.21.15.146:0/4021894721 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbdfc650 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.758+0000 7f589357f740 1 -- 172.21.15.146:0/4021894721 >> 172.21.15.146:0/4021894721 conn(0x558efbdf8590 msgr2=0x558efbdf8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.759+0000 7f589357f740 1 -- 172.21.15.146:0/4021894721 shutdown_connections 2024-04-24T08:22:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.759+0000 7f589357f740 1 -- 172.21.15.146:0/4021894721 wait complete. 2024-04-24T08:22:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.759+0000 7f589357f740 1 -- start start 2024-04-24T08:22:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.759+0000 7f588dc00640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f58700a3f50 0x7f58700a6410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x558efbdfd830 tx=0x7f587c014040 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:26.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.759+0000 7f589357f740 1 Processor -- start 2024-04-24T08:22:26.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f589357f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfc270 0x558efbe04240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f589357f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfcb90 0x558efbe04780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f589357f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe04cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f589357f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558efbe09ff0 con 0x558efbdfc270 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f589357f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558efbd1b660 con 0x558efbdfcb90 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f589357f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558efbe0a170 con 0x558efbe07260 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f589357f740 10 client.0 _ll_register_callbacks cb 0x558efbd09aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f588dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfc270 0x558efbe04240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f588dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfc270 0x558efbe04240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54910/0 (socket says 172.21.15.146:54910) 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.760+0000 7f588dc00640 1 -- 172.21.15.146:0/3115710358 learned_addr learned my addr 172.21.15.146:0/3115710358 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588d200640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfcb90 0x558efbe04780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588e600640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe04cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588dc00640 1 -- 172.21.15.146:0/3115710358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 msgr2=0x558efbe04cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588dc00640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe04cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588dc00640 1 -- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfcb90 msgr2=0x558efbe04780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588dc00640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfcb90 0x558efbe04780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588dc00640 1 -- 172.21.15.146:0/3115710358 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558efbb9cf70 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588e600640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe04cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f588dc00640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfc270 0x558efbe04240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f587c011c30 tx=0x7f587c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199a42ebd6d67822 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.761+0000 7f586f400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.762+0000 7f586f400640 1 -- 172.21.15.146:0/3115710358 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c023070 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.762+0000 7f586f400640 1 -- 172.21.15.146:0/3115710358 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f587c00fc20 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.762+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5880037040 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.762+0000 7f586f400640 1 -- 172.21.15.146:0/3115710358 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f587c00fe20 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.762+0000 7f589357f740 10 client.15513 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.762+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558efba34170 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.762+0000 7f586f400640 1 -- 172.21.15.146:0/3115710358 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5880037040 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f586f400640 1 -- 172.21.15.146:0/3115710358 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f587c01c040 con 0x558efbdfc270 2024-04-24T08:22:26.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f589357f740 10 client.15513 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:26.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f589357f740 10 client.15513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:26.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f589357f740 20 client.15513 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:26.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f589357f740 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe17580 0x558efbe19990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f589357f740 4 client.15513 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:26.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558efbd6f890 con 0x558efbe17580 2024-04-24T08:22:26.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.763+0000 7f588d200640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe17580 0x558efbe19990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.764+0000 7f588d200640 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe17580 0x558efbe19990 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7cc34bd8006f096 server_cookie=bb05848162695ef6 in_seq=0 out_seq=0 2024-04-24T08:22:26.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.764+0000 7f586f400640 10 client.15513 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:26.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.764+0000 7f586f400640 1 -- 172.21.15.146:0/3115710358 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558efbd6f890 con 0x558efbe17580 2024-04-24T08:22:26.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.764+0000 7f586f400640 10 client.15513 handle_command_reply: tid=0 2024-04-24T08:22:26.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.764+0000 7f589357f740 1 client.15513 shutdown 2024-04-24T08:22:26.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfc270 msgr2=0x558efbe04240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfc270 0x558efbe04240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f587c011c30 tx=0x7f587c004730 comp rx=0 tx=0).stop 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 shutdown_connections 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe17580 0x558efbe19990 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbe07260 0x558efbe04cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfcb90 0x558efbe04780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f586f400640 0 client.15513 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 --2- 172.21.15.146:0/3115710358 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfc270 0x558efbe04240 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 >> 172.21.15.146:0/3115710358 conn(0x558efbdf8590 msgr2=0x558efbdfe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 shutdown_connections 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 1 -- 172.21.15.146:0/3115710358 wait complete. 2024-04-24T08:22:26.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.765+0000 7f589357f740 20 client.15513 trim_cache size 0 max 16384 2024-04-24T08:22:26.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.800+0000 7f589357f740 1 Processor -- start 2024-04-24T08:22:26.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.800+0000 7f589357f740 1 -- start start 2024-04-24T08:22:26.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f589357f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbe12eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f589357f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfcb90 0x558efbe1f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f589357f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbe07260 0x558efbe1f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f589357f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f58881b6a40 con 0x558efbe07260 2024-04-24T08:22:26.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f589357f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58881b4310 con 0x558efbdfc270 2024-04-24T08:22:26.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f589357f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f58881b2180 con 0x558efbdfcb90 2024-04-24T08:22:26.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f588dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbe12eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f588e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbe07260 0x558efbe1f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f588dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbe12eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43058/0 (socket says 172.21.15.146:43058) 2024-04-24T08:22:26.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f588d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfcb90 0x558efbe1f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.801+0000 7f588dc00640 1 -- 172.21.15.146:0/1320560516 learned_addr learned my addr 172.21.15.146:0/1320560516 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f588dc00640 1 -- 172.21.15.146:0/1320560516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfcb90 msgr2=0x558efbe1f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f588dc00640 1 --2- 172.21.15.146:0/1320560516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfcb90 0x558efbe1f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f588dc00640 1 -- 172.21.15.146:0/1320560516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbe07260 msgr2=0x558efbe1f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f588dc00640 1 --2- 172.21.15.146:0/1320560516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbe07260 0x558efbe1f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f588dc00640 1 -- 172.21.15.146:0/1320560516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558efbe09650 con 0x558efbdfc270 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f588dc00640 1 --2- 172.21.15.146:0/1320560516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbe12eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f587c012d20 tx=0x7f587c009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3558ac17cbb2c0f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f587a000640 1 -- 172.21.15.146:0/1320560516 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c023040 con 0x558efbdfc270 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f587a000640 1 -- 172.21.15.146:0/1320560516 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f587c028460 con 0x558efbdfc270 2024-04-24T08:22:26.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.802+0000 7f587a000640 1 -- 172.21.15.146:0/1320560516 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c01f950 con 0x558efbdfc270 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 -- 172.21.15.146:0/1320560516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 msgr2=0x558efbe12eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 --2- 172.21.15.146:0/1320560516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbe12eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f587c012d20 tx=0x7f587c009bf0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 -- 172.21.15.146:0/1320560516 shutdown_connections 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 --2- 172.21.15.146:0/1320560516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfcb90 0x558efbe1f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 --2- 172.21.15.146:0/1320560516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbe07260 0x558efbe1f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 --2- 172.21.15.146:0/1320560516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbdfc270 0x558efbe12eb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 -- 172.21.15.146:0/1320560516 >> 172.21.15.146:0/1320560516 conn(0x558efbdf8590 msgr2=0x558efbdf76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 -- 172.21.15.146:0/1320560516 shutdown_connections 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.803+0000 7f589357f740 1 -- 172.21.15.146:0/1320560516 wait complete. 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 -- start start 2024-04-24T08:22:26.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 Processor -- start 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfc270 0x558efbe0f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe0f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbe07260 0x558efbe0c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f58881b27d0 con 0x558efbdfcb90 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f58881b3c40 con 0x558efbe07260 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.804+0000 7f589357f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f58881b2ea0 con 0x558efbdfc270 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f589357f740 10 client.0 _ll_register_callbacks cb 0x7f587c009770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfc270 0x558efbe0f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe0f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbe07260 0x558efbe0c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbe07260 0x558efbe0c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43060/0 (socket says 172.21.15.146:43060) 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 -- 172.21.15.146:0/4039378264 learned_addr learned my addr 172.21.15.146:0/4039378264 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe0f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54932/0 (socket says 172.21.15.146:54932) 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 -- 172.21.15.146:0/4039378264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfc270 msgr2=0x558efbe0f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfc270 0x558efbe0f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 -- 172.21.15.146:0/4039378264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 msgr2=0x558efbe0f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe0f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.805+0000 7f588e600640 1 -- 172.21.15.146:0/4039378264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558efbe0ca00 con 0x558efbe07260 2024-04-24T08:22:26.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.806+0000 7f588e600640 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbe07260 0x558efbe0c390 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5880004af0 tx=0x7f5880005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c88847923407c463 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.806+0000 7f586f400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.806+0000 7f586f400640 1 -- 172.21.15.146:0/4039378264 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588004f060 con 0x558efbe07260 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.806+0000 7f586f400640 1 -- 172.21.15.146:0/4039378264 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5880030db0 con 0x558efbe07260 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.806+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f587c028460 con 0x558efbe07260 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.807+0000 7f586f400640 1 -- 172.21.15.146:0/4039378264 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5880042040 con 0x558efbe07260 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.807+0000 7f589357f740 10 client.25658 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.807+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558efbe7e480 con 0x558efbe07260 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.807+0000 7f586f400640 1 -- 172.21.15.146:0/4039378264 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f587c028460 con 0x558efbe07260 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.807+0000 7f586f400640 1 -- 172.21.15.146:0/4039378264 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f588003c040 con 0x558efbe07260 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.807+0000 7f589357f740 10 client.25658 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f589357f740 10 client.25658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f589357f740 20 client.25658 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f589357f740 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe7e7b0 0x558efbe80ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:26.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f589357f740 4 client.25658 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558efbd69700 con 0x558efbe7e7b0 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f588dc00640 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe7e7b0 0x558efbe80ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f588dc00640 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe7e7b0 0x558efbe80ba0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c017fe7dce2d1e4e server_cookie=157608fe1ab8eb16 in_seq=0 out_seq=0 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.808+0000 7f586f400640 10 client.25658 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.809+0000 7f586f400640 1 -- 172.21.15.146:0/4039378264 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558efbd69700 con 0x558efbe7e7b0 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.809+0000 7f586f400640 10 client.25658 handle_command_reply: tid=0 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.809+0000 7f589357f740 1 client.25658 shutdown 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbe07260 msgr2=0x558efbe0c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbe07260 0x558efbe0c390 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5880004af0 tx=0x7f5880005d30 comp rx=0 tx=0).stop 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 shutdown_connections 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558efbe7e7b0 0x558efbe80ba0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558efbe07260 0x558efbe0c390 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f586f400640 0 client.25658 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:26.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558efbdfcb90 0x558efbe0f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 --2- 172.21.15.146:0/4039378264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558efbdfc270 0x558efbe0f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 >> 172.21.15.146:0/4039378264 conn(0x558efbdf8590 msgr2=0x558efbdfd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 shutdown_connections 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 1 -- 172.21.15.146:0/4039378264 wait complete. 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.810+0000 7f589357f740 20 client.25658 trim_cache size 0 max 16384 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "f6701b15-3453-4931-9dee-c57f03284ead", 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:22:26.816 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:22:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f58700a3f50 msgr2=0x7f58700a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f58700a3f50 0x7f58700a6410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x558efbdfd830 tx=0x7f587c014040 comp rx=0 tx=0).stop 2024-04-24T08:22:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58881093f0 msgr2=0x7f5888076a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58881093f0 0x7f5888076a50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f587400e9c0 tx=0x7f587400ee80 comp rx=0 tx=0).stop 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 shutdown_connections 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f58700a3f50 0x7f58700a6410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5888076f90 0x7f58881aaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f58881093f0 0x7f5888076a50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 --2- 172.21.15.146:0/2780811205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5888104320 0x7f5888076510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 >> 172.21.15.146:0/2780811205 conn(0x7f58880ffb40 msgr2=0x7f5888100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.811+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 shutdown_connections 2024-04-24T08:22:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:26.812+0000 7f588f000640 1 -- 172.21.15.146:0/2780811205 wait complete. 2024-04-24T08:22:26.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:22:26.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 -- 172.21.15.146:0/755251038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba0102ac0 msgr2=0x7f9ba0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/755251038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba0102ac0 0x7f9ba0106b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009860 tx=0x7f9b8c02a340 comp rx=0 tx=0).stop 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 -- 172.21.15.146:0/755251038 shutdown_connections 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/755251038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 0x7f9ba010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/755251038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba0102ac0 0x7f9ba0106b00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/755251038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0102110 0x7f9ba01024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 -- 172.21.15.146:0/755251038 >> 172.21.15.146:0/755251038 conn(0x7f9ba00fd9d0 msgr2=0x7f9ba00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:27.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 -- 172.21.15.146:0/755251038 shutdown_connections 2024-04-24T08:22:27.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.030+0000 7f9ba5c00640 1 -- 172.21.15.146:0/755251038 wait complete. 2024-04-24T08:22:27.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.031+0000 7f9ba5c00640 1 Processor -- start 2024-04-24T08:22:27.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.031+0000 7f9ba5c00640 1 -- start start 2024-04-24T08:22:27.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9ba5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0102110 0x7f9ba01a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9ba5c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 0x7f9ba01a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9ba5c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba019a850 0x7f9ba019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9ba5c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba0112250 con 0x7f9ba01071e0 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9ba5c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01120d0 con 0x7f9ba019a850 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9ba5c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01123d0 con 0x7f9ba0102110 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 0x7f9ba01a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 0x7f9ba01a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54944/0 (socket says 172.21.15.146:54944) 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 -- 172.21.15.146:0/703620395 learned_addr learned my addr 172.21.15.146:0/703620395 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0102110 0x7f9ba01a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9ba5200640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba019a850 0x7f9ba019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 -- 172.21.15.146:0/703620395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0102110 msgr2=0x7f9ba01a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0102110 0x7f9ba01a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 -- 172.21.15.146:0/703620395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba019a850 msgr2=0x7f9ba019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba019a850 0x7f9ba019ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9f400640 1 -- 172.21.15.146:0/703620395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba019b3e0 con 0x7f9ba01071e0 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.032+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0102110 0x7f9ba01a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:27.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.033+0000 7f9b9f400640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 0x7f9ba01a0c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c035040 tx=0x7f9b8c033b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:27.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.033+0000 7f9b9cc00640 1 -- 172.21.15.146:0/703620395 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c037040 con 0x7f9ba01071e0 2024-04-24T08:22:27.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.033+0000 7f9b9cc00640 1 -- 172.21.15.146:0/703620395 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b8c040070 con 0x7f9ba01071e0 2024-04-24T08:22:27.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.033+0000 7f9b9cc00640 1 -- 172.21.15.146:0/703620395 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c003930 con 0x7f9ba01071e0 2024-04-24T08:22:27.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.033+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba019b670 con 0x7f9ba01071e0 2024-04-24T08:22:27.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.033+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba01a77d0 con 0x7f9ba01071e0 2024-04-24T08:22:27.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.037+0000 7f9b9cc00640 1 -- 172.21.15.146:0/703620395 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 26) ==== 138931+0+0 (secure 0 0 0) 0x7f9b8c030670 con 0x7f9ba01071e0 2024-04-24T08:22:27.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.038+0000 7f9baa04f740 1 Processor -- start 2024-04-24T08:22:27.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.038+0000 7f9baa04f740 1 -- start start 2024-04-24T08:22:27.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71efa30 0x5588f71efe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71fce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9baa04f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5588f71551f0 con 0x5588f71faa20 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9baa04f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5588f70fb350 con 0x5588f71efa30 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9baa04f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588f71511b0 con 0x5588f71f0350 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9ba5200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71efa30 0x5588f71efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9ba5200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71efa30 0x5588f71efe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:43090/0 (socket says 172.21.15.146:43090) 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9ba5200640 1 -- 172.21.15.146:0/306103095 learned_addr learned my addr 172.21.15.146:0/306103095 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:27.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9b9f400640 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71fce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9ba5200640 1 -- 172.21.15.146:0/306103095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 msgr2=0x5588f71fa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9ba5200640 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.039+0000 7f9ba5200640 1 -- 172.21.15.146:0/306103095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 msgr2=0x5588f71fce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9ba5200640 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71fce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9ba5200640 1 -- 172.21.15.146:0/306103095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f714ff50 con 0x5588f71efa30 2024-04-24T08:22:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9b9cc00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b840a3f50 0x7f9b840a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9ba5200640 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71efa30 0x5588f71efe10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b980099f0 tx=0x7f9b980095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d0e3c60af390f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9b9cc00640 1 -- 172.21.15.146:0/703620395 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9b8c0f0850 con 0x7f9ba01071e0 2024-04-24T08:22:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9b95600640 1 -- 172.21.15.146:0/306103095 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9801d070 con 0x5588f71efa30 2024-04-24T08:22:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9b95600640 1 -- 172.21.15.146:0/306103095 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98003e40 con 0x5588f71efa30 2024-04-24T08:22:27.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9b95600640 1 -- 172.21.15.146:0/306103095 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9800db30 con 0x5588f71efa30 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.040+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b840a3f50 0x7f9b840a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 -- 172.21.15.146:0/306103095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71efa30 msgr2=0x5588f71efe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71efa30 0x5588f71efe10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9b980099f0 tx=0x7f9b980095c0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b840a3f50 0x7f9b840a6410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5588f71f0ff0 tx=0x7f9b90005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 -- 172.21.15.146:0/306103095 shutdown_connections 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 --2- 172.21.15.146:0/306103095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71efa30 0x5588f71efe10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 -- 172.21.15.146:0/306103095 >> 172.21.15.146:0/306103095 conn(0x5588f71ebd50 msgr2=0x5588f71ec130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 -- 172.21.15.146:0/306103095 shutdown_connections 2024-04-24T08:22:27.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.041+0000 7f9baa04f740 1 -- 172.21.15.146:0/306103095 wait complete. 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.042+0000 7f9baa04f740 1 -- start start 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.042+0000 7f9baa04f740 1 Processor -- start 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71f7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71f7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f71f8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9baa04f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5588f71fd810 con 0x5588f71efa30 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9baa04f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5588f710e690 con 0x5588f71faa20 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9baa04f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588f71fd990 con 0x5588f71f0350 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9baa04f740 10 client.0 _ll_register_callbacks cb 0x5588f714a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71f7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54974/0 (socket says 172.21.15.146:54974) 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.043+0000 7f9b9fe00640 1 -- 172.21.15.146:0/405315921 learned_addr learned my addr 172.21.15.146:0/405315921 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:27.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b9f400640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71f7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9ba5200640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f71f8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b9fe00640 1 -- 172.21.15.146:0/405315921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 msgr2=0x5588f71f7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71f7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b9fe00640 1 -- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 msgr2=0x5588f71f8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f71f8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b9fe00640 1 -- 172.21.15.146:0/405315921 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f710d920 con 0x5588f71efa30 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9ba5200640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f71f8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71f7990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9b90005bc0 tx=0x7f9b90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b47b8e6f65c022f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:27.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.044+0000 7f9b8a000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.045+0000 7f9b8a000640 1 -- 172.21.15.146:0/405315921 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9002e060 con 0x5588f71efa30 2024-04-24T08:22:27.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.045+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5588f7162c10 con 0x5588f71efa30 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.045+0000 7f9b8a000640 1 -- 172.21.15.146:0/405315921 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9b90018030 con 0x5588f71efa30 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.045+0000 7f9b8a000640 1 -- 172.21.15.146:0/405315921 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b9001e6a0 con 0x5588f71efa30 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.045+0000 7f9baa04f740 10 client.15531 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.045+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5588f6f317d0 con 0x5588f71efa30 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.045+0000 7f9b8a000640 1 -- 172.21.15.146:0/405315921 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5588f7162c10 con 0x5588f71efa30 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9b8a000640 1 -- 172.21.15.146:0/405315921 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9b900319f0 con 0x5588f71efa30 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9baa04f740 10 client.15531 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9baa04f740 10 client.15531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9baa04f740 20 client.15531 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9baa04f740 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f720ad80 0x5588f720d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9baa04f740 4 client.15531 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588f715ca80 con 0x5588f720ad80 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.046+0000 7f9b9f400640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f720ad80 0x5588f720d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.047+0000 7f9b9f400640 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f720ad80 0x5588f720d1f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c2020dcd21b57db server_cookie=ad3d7030042dbcc4 in_seq=0 out_seq=0 2024-04-24T08:22:27.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.047+0000 7f9b8a000640 10 client.15531 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:27.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9b8a000640 1 -- 172.21.15.146:0/405315921 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588f715ca80 con 0x5588f720ad80 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9b8a000640 10 client.15531 handle_command_reply: tid=0 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 client.15531 shutdown 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 msgr2=0x5588f71f7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71f7990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9b90005bc0 tx=0x7f9b90004730 comp rx=0 tx=0).stop 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 shutdown_connections 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f720ad80 0x5588f720d1f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f71f8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 --2- 172.21.15.146:0/405315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71f7990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 >> 172.21.15.146:0/405315921 conn(0x5588f71ebd50 msgr2=0x5588f71f1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9b8a000640 0 client.15531 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 shutdown_connections 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.048+0000 7f9baa04f740 1 -- 172.21.15.146:0/405315921 wait complete. 2024-04-24T08:22:27.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.049+0000 7f9baa04f740 20 client.15531 trim_cache size 0 max 16384 2024-04-24T08:22:27.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.067+0000 7f9baa04f740 1 Processor -- start 2024-04-24T08:22:27.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.067+0000 7f9baa04f740 1 -- start start 2024-04-24T08:22:27.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71efa30 0x5588f72066d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71f0350 0x5588f71f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71f8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9baa04f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01b2680 con 0x5588f71faa20 2024-04-24T08:22:27.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9baa04f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01aff50 con 0x5588f71f0350 2024-04-24T08:22:27.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9baa04f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01adda0 con 0x5588f71efa30 2024-04-24T08:22:27.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9ba5200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71efa30 0x5588f72066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71efa30 0x5588f72066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:56454/0 (socket says 172.21.15.146:56454) 2024-04-24T08:22:27.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9b9fe00640 1 -- 172.21.15.146:0/2791632168 learned_addr learned my addr 172.21.15.146:0/2791632168 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.068+0000 7f9b9f400640 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71f0350 0x5588f71f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b9fe00640 1 -- 172.21.15.146:0/2791632168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71f0350 msgr2=0x5588f71f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71f0350 0x5588f71f6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b9fe00640 1 -- 172.21.15.146:0/2791632168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 msgr2=0x5588f71f8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71f8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b9fe00640 1 -- 172.21.15.146:0/2791632168 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588f6e673f0 con 0x5588f71efa30 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71efa30 0x5588f72066d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009f70 tx=0x7f9b900092a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5fba092fdeaff10 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b94c00640 1 -- 172.21.15.146:0/2791632168 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9001c4a0 con 0x5588f71efa30 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b94c00640 1 -- 172.21.15.146:0/2791632168 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b90012030 con 0x5588f71efa30 2024-04-24T08:22:27.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.069+0000 7f9b94c00640 1 -- 172.21.15.146:0/2791632168 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9001c770 con 0x5588f71efa30 2024-04-24T08:22:27.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 -- 172.21.15.146:0/2791632168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71efa30 msgr2=0x5588f72066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71efa30 0x5588f72066d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009f70 tx=0x7f9b900092a0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 -- 172.21.15.146:0/2791632168 shutdown_connections 2024-04-24T08:22:27.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71faa20 0x5588f71f8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71f0350 0x5588f71f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 --2- 172.21.15.146:0/2791632168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71efa30 0x5588f72066d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 -- 172.21.15.146:0/2791632168 >> 172.21.15.146:0/2791632168 conn(0x5588f71ebd50 msgr2=0x5588f71f7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 -- 172.21.15.146:0/2791632168 shutdown_connections 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.070+0000 7f9baa04f740 1 -- 172.21.15.146:0/2791632168 wait complete. 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 -- start start 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 Processor -- start 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fe0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f726f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01ae410 con 0x5588f71efa30 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01af880 con 0x5588f71faa20 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.071+0000 7f9baa04f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01aeae0 con 0x5588f71f0350 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9baa04f740 10 client.0 _ll_register_callbacks cb 0x7f9b90009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71fdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55000/0 (socket says 172.21.15.146:55000) 2024-04-24T08:22:27.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9fe00640 1 -- 172.21.15.146:0/371911136 learned_addr learned my addr 172.21.15.146:0/371911136 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9f400640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fe0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9ba5200640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f726f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9f400640 1 -- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 msgr2=0x5588f726f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9f400640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f726f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9f400640 1 -- 172.21.15.146:0/371911136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 msgr2=0x5588f71fdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9f400640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71fdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9f400640 1 -- 172.21.15.146:0/371911136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588f6e270f0 con 0x5588f71f0350 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71fdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.072+0000 7f9b9f400640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fe0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0ede80 tx=0x7f9b8c033f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2032d5a5523d392 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9b8a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9b8a000640 1 -- 172.21.15.146:0/371911136 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0d2020 con 0x5588f71f0350 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9b8a000640 1 -- 172.21.15.146:0/371911136 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9b8c0c4eb0 con 0x5588f71f0350 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b90012030 con 0x5588f71f0350 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9b8a000640 1 -- 172.21.15.146:0/371911136 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b8c0bcdd0 con 0x5588f71f0350 2024-04-24T08:22:27.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9baa04f740 10 client.25669 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5588f711a810 con 0x5588f71f0350 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.073+0000 7f9b8a000640 1 -- 172.21.15.146:0/371911136 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b90012030 con 0x5588f71f0350 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9b8a000640 1 -- 172.21.15.146:0/371911136 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9b8c0f2ce0 con 0x5588f71f0350 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9baa04f740 10 client.25669 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9baa04f740 10 client.25669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9baa04f740 20 client.25669 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9baa04f740 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f7270df0 0x5588f7273210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9baa04f740 4 client.25669 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:22:27.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5588f70d8360 con 0x5588f7270df0 2024-04-24T08:22:27.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.074+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f7270df0 0x5588f7273210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:27.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9b9fe00640 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f7270df0 0x5588f7273210 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ae025479803f9e server_cookie=9889b7fe10f8916e in_seq=0 out_seq=0 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9b8a000640 10 client.25669 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9b8a000640 1 -- 172.21.15.146:0/371911136 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5588f70d8360 con 0x5588f7270df0 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9b8a000640 10 client.25669 handle_command_reply: tid=0 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 client.25669 shutdown 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 msgr2=0x5588f71fe0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fe0e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0ede80 tx=0x7f9b8c033f50 comp rx=0 tx=0).stop 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 shutdown_connections 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5588f7270df0 0x5588f7273210 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588f71f0350 0x5588f71fe0e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588f71faa20 0x5588f726f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9b8a000640 0 client.25669 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 --2- 172.21.15.146:0/371911136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588f71efa30 0x5588f71fdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.075+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 >> 172.21.15.146:0/371911136 conn(0x5588f71ebd50 msgr2=0x5588f71f1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.076+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 shutdown_connections 2024-04-24T08:22:27.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.076+0000 7f9baa04f740 1 -- 172.21.15.146:0/371911136 wait complete. 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.076+0000 7f9baa04f740 20 client.25669 trim_cache size 0 max 16384 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "f6701b15-3453-4931-9dee-c57f03284ead": { 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "f6701b15-3453-4931-9dee-c57f03284ead", 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:22:27.081 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:22:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b840a3f50 msgr2=0x7f9b840a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b840a3f50 0x7f9b840a6410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5588f71f0ff0 tx=0x7f9b90005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 msgr2=0x7f9ba01a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 0x7f9ba01a0c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c035040 tx=0x7f9b8c033b60 comp rx=0 tx=0).stop 2024-04-24T08:22:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 shutdown_connections 2024-04-24T08:22:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b840a3f50 0x7f9b840a6410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ba019a850 0x7f9ba019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba01071e0 0x7f9ba01a0c20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 --2- 172.21.15.146:0/703620395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba0102110 0x7f9ba01a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:27.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 >> 172.21.15.146:0/703620395 conn(0x7f9ba00fd9d0 msgr2=0x7f9ba00ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:27.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 shutdown_connections 2024-04-24T08:22:27.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:27.077+0000 7f9ba5c00640 1 -- 172.21.15.146:0/703620395 wait complete. 2024-04-24T08:22:27.100 INFO:tasks.cephfs.filesystem:scrub status for tag:f6701b15-3453-4931-9dee-c57f03284ead - {'path': '/', 'tag': 'f6701b15-3453-4931-9dee-c57f03284ead', 'options': 'recursive,force'} 2024-04-24T08:22:28.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:28 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:22:28.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:28 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:22:28.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:28 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:22:28.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:28 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:22:28.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:28 smithi146 ceph-mon[27805]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:28.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:28 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:22:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:28 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:22:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:28 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:22:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:28 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:22:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:28 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:22:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:28 smithi177 ceph-mon[33046]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:28 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:22:28.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:28 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:22:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:28 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:22:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:28 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:22:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:28 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:22:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:28 smithi153 ceph-mon[32746]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:28 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:22:30.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:30 smithi146 ceph-mon[27805]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:30.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:30 smithi177 ceph-mon[33046]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:30.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:30 smithi153 ceph-mon[32746]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:32.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:32 smithi146 ceph-mon[27805]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:32.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:32 smithi153 ceph-mon[32746]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:32.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:32 smithi177 ceph-mon[33046]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:34.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:34 smithi146 ceph-mon[27805]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:34.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:34 smithi177 ceph-mon[33046]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:34.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:34 smithi153 ceph-mon[32746]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:36.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:36 smithi146 ceph-mon[27805]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:36.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:36 smithi177 ceph-mon[33046]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:36.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:36 smithi153 ceph-mon[32746]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:22:38.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:38 smithi146 ceph-mon[27805]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:38.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:38 smithi177 ceph-mon[33046]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:38.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:38 smithi153 ceph-mon[32746]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:22:40.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:40 smithi146 ceph-mon[27805]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:40.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:40 smithi177 ceph-mon[33046]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:40.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:40 smithi153 ceph-mon[32746]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:42.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:42 smithi146 ceph-mon[27805]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:42.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:42 smithi177 ceph-mon[33046]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:42.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:42 smithi153 ceph-mon[32746]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:44 smithi177 ceph-mon[33046]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:44.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:44 smithi177 ceph-mon[33046]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-24T08:22:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:44 smithi153 ceph-mon[32746]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:44 smithi153 ceph-mon[32746]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-24T08:22:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:44 smithi146 ceph-mon[27805]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:44 smithi146 ceph-mon[27805]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-24T08:22:46.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:46 smithi177 ceph-mon[33046]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:46.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:46 smithi153 ceph-mon[32746]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:46 smithi146 ceph-mon[27805]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:48.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:48 smithi177 ceph-mon[33046]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:48 smithi153 ceph-mon[32746]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:48.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:48 smithi146 ceph-mon[27805]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:50.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:50 smithi177 ceph-mon[33046]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:50.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:50 smithi153 ceph-mon[32746]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:50.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:50 smithi146 ceph-mon[27805]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:52.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:52 smithi177 ceph-mon[33046]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:52.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:52 smithi153 ceph-mon[32746]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:52.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:52 smithi146 ceph-mon[27805]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-24T08:22:54.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:54 smithi177 ceph-mon[33046]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:54.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:54 smithi153 ceph-mon[32746]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:54.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:54 smithi146 ceph-mon[27805]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-24T08:22:56.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:56 smithi177 ceph-mon[33046]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:56.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:56 smithi153 ceph-mon[32746]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:56 smithi146 ceph-mon[27805]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:22:57.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:22:57.101 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:22:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.305+0000 7f02fe000640 1 -- 172.21.15.146:0/218085410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 msgr2=0x7f02f81024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.305+0000 7f02fe000640 1 --2- 172.21.15.146:0/218085410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81024f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009900 tx=0x7f02ec02a410 comp rx=0 tx=0).stop 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.306+0000 7f02fe000640 1 -- 172.21.15.146:0/218085410 shutdown_connections 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.306+0000 7f02fe000640 1 --2- 172.21.15.146:0/218085410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f8102ac0 0x7f02f8106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.306+0000 7f02fe000640 1 --2- 172.21.15.146:0/218085410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f81071e0 0x7f02f810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.306+0000 7f02fe000640 1 --2- 172.21.15.146:0/218085410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81024f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.306+0000 7f02fe000640 1 -- 172.21.15.146:0/218085410 >> 172.21.15.146:0/218085410 conn(0x7f02f80fd9d0 msgr2=0x7f02f80ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.306+0000 7f02fe000640 1 -- 172.21.15.146:0/218085410 shutdown_connections 2024-04-24T08:22:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.306+0000 7f02fe000640 1 -- 172.21.15.146:0/218085410 wait complete. 2024-04-24T08:22:57.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.307+0000 7f02fe000640 1 Processor -- start 2024-04-24T08:22:57.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.307+0000 7f02fe000640 1 -- start start 2024-04-24T08:22:57.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fe000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fe000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81071e0 0x7f02f81a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fe000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f819a7d0 0x7f02f819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fe000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8112270 con 0x7f02f8102110 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fe000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81120f0 con 0x7f02f819a7d0 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fe000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81123f0 con 0x7f02f81071e0 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:60998/0 (socket says 172.21.15.146:60998) 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fcc00640 1 -- 172.21.15.146:0/4247454821 learned_addr learned my addr 172.21.15.146:0/4247454821 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02f7e00640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81071e0 0x7f02f81a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.308+0000 7f02fd600640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f819a7d0 0x7f02f819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02f7e00640 1 -- 172.21.15.146:0/4247454821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f819a7d0 msgr2=0x7f02f819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02f7e00640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f819a7d0 0x7f02f819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02f7e00640 1 -- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 msgr2=0x7f02f81a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02f7e00640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02f7e00640 1 -- 172.21.15.146:0/4247454821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f819b360 con 0x7f02f81071e0 2024-04-24T08:22:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02fd600640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f819a7d0 0x7f02f819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02fcc00640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02f7e00640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81071e0 0x7f02f81a0b50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f02e400e9c0 tx=0x7f02e400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.309+0000 7f02f5600640 1 -- 172.21.15.146:0/4247454821 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e400ccc0 con 0x7f02f81071e0 2024-04-24T08:22:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.310+0000 7f02f5600640 1 -- 172.21.15.146:0/4247454821 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02e4010910 con 0x7f02f81071e0 2024-04-24T08:22:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.310+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f819b650 con 0x7f02f81071e0 2024-04-24T08:22:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.310+0000 7f02f5600640 1 -- 172.21.15.146:0/4247454821 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4018880 con 0x7f02f81071e0 2024-04-24T08:22:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.310+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f81a77e0 con 0x7f02f81071e0 2024-04-24T08:22:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 Processor -- start 2024-04-24T08:22:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f02f5600640 1 -- 172.21.15.146:0/4247454821 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f02e4010ac0 con 0x7f02f81071e0 2024-04-24T08:22:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 -- start start 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31320a30 0x55dc31320e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31321350 0x55dc3132b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc3132de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dc312861f0 con 0x55dc31321350 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc3122c350 con 0x55dc3132ba20 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f0302417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dc312821b0 con 0x55dc31320a30 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31321350 0x55dc3132b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f02fd600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31320a30 0x55dc31320e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31321350 0x55dc3132b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32774/0 (socket says 172.21.15.146:32774) 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.313+0000 7f02fcc00640 1 -- 172.21.15.146:0/4198129163 learned_addr learned my addr 172.21.15.146:0/4198129163 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02f7e00640 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc3132de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02fd600640 1 -- 172.21.15.146:0/4198129163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 msgr2=0x55dc3132de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02fd600640 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc3132de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02fd600640 1 -- 172.21.15.146:0/4198129163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31321350 msgr2=0x55dc3132b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02fd600640 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31321350 0x55dc3132b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02fd600640 1 -- 172.21.15.146:0/4198129163 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc31280f50 con 0x55dc31320a30 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02fcc00640 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31321350 0x55dc3132b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02fd600640 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31320a30 0x55dc31320e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f02f0010770 tx=0x7f02f0010c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7624add2ebc7a5c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02e9c00640 1 -- 172.21.15.146:0/4198129163 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0003970 con 0x55dc31320a30 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02e9c00640 1 -- 172.21.15.146:0/4198129163 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f000e930 con 0x55dc31320a30 2024-04-24T08:22:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.314+0000 7f02e9c00640 1 -- 172.21.15.146:0/4198129163 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0004660 con 0x55dc31320a30 2024-04-24T08:22:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 -- 172.21.15.146:0/4198129163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31320a30 msgr2=0x55dc31320e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31320a30 0x55dc31320e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f02f0010770 tx=0x7f02f0010c30 comp rx=0 tx=0).stop 2024-04-24T08:22:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f02f5600640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02e00a3d20 0x7f02e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 -- 172.21.15.146:0/4198129163 shutdown_connections 2024-04-24T08:22:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc3132de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31321350 0x55dc3132b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 --2- 172.21.15.146:0/4198129163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31320a30 0x55dc31320e10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 -- 172.21.15.146:0/4198129163 >> 172.21.15.146:0/4198129163 conn(0x55dc3131cd50 msgr2=0x55dc3131d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f02f5600640 1 -- 172.21.15.146:0/4247454821 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f02e40d0b80 con 0x7f02f81071e0 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f0302417740 1 -- 172.21.15.146:0/4198129163 shutdown_connections 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.315+0000 7f02fcc00640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02e00a3d20 0x7f02e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.316+0000 7f0302417740 1 -- 172.21.15.146:0/4198129163 wait complete. 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.316+0000 7f0302417740 1 -- start start 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.316+0000 7f0302417740 1 Processor -- start 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.316+0000 7f02fcc00640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02e00a3d20 0x7f02e00a61e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009230 tx=0x7f02ec005a20 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f0302417740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31328a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f0302417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31328f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f0302417740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313294a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f0302417740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dc3132e810 con 0x55dc31320a30 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f0302417740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dc3123f690 con 0x55dc3132ba20 2024-04-24T08:22:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f0302417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dc3132e990 con 0x55dc31321350 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f0302417740 10 client.0 _ll_register_callbacks cb 0x55dc3127b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31328a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31328a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32780/0 (socket says 172.21.15.146:32780) 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 -- 172.21.15.146:0/3870965716 learned_addr learned my addr 172.21.15.146:0/3870965716 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02f7e00640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31328f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fd600640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313294a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 -- 172.21.15.146:0/3870965716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 msgr2=0x55dc31328f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31328f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 -- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 msgr2=0x55dc313294a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313294a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fcc00640 1 -- 172.21.15.146:0/3870965716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc3123e920 con 0x55dc31320a30 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02f7e00640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31328f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.317+0000 7f02fd600640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313294a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f02fcc00640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31328a20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009a20 tx=0x7f02ec009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d0cd71f18025ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f02de000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f02de000640 1 -- 172.21.15.146:0/3870965716 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec04d070 con 0x55dc31320a30 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f02de000640 1 -- 172.21.15.146:0/3870965716 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f02ec004250 con 0x55dc31320a30 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f000e930 con 0x55dc31320a30 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f02de000640 1 -- 172.21.15.146:0/3870965716 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f02ec059050 con 0x55dc31320a30 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f0302417740 10 client.15549 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.318+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dc310627d0 con 0x55dc31320a30 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f02de000640 1 -- 172.21.15.146:0/3870965716 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f000e930 con 0x55dc31320a30 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f02de000640 1 -- 172.21.15.146:0/3870965716 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f02ec003830 con 0x55dc31320a30 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f0302417740 10 client.15549 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f0302417740 10 client.15549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f0302417740 20 client.15549 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f0302417740 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc3133c180 0x55dc3133e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f0302417740 4 client.15549 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc31293c10 con 0x55dc3133c180 2024-04-24T08:22:57.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.319+0000 7f02f7e00640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc3133c180 0x55dc3133e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.320+0000 7f02de000640 10 client.15549 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.320+0000 7f02f7e00640 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc3133c180 0x55dc3133e620 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=352d624498682f25 server_cookie=1af6e01320071d73 in_seq=0 out_seq=0 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f02de000640 1 -- 172.21.15.146:0/3870965716 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc31293c10 con 0x55dc3133c180 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f02de000640 10 client.15549 handle_command_reply: tid=0 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 client.15549 shutdown 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 msgr2=0x55dc31328a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31328a20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009a20 tx=0x7f02ec009260 comp rx=0 tx=0).stop 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 shutdown_connections 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc3133c180 0x55dc3133e620 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31328f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313294a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 --2- 172.21.15.146:0/3870965716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31328a20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f02de000640 0 client.15549 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:57.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 >> 172.21.15.146:0/3870965716 conn(0x55dc3131cd50 msgr2=0x55dc313222c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 shutdown_connections 2024-04-24T08:22:57.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 1 -- 172.21.15.146:0/3870965716 wait complete. 2024-04-24T08:22:57.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.321+0000 7f0302417740 20 client.15549 trim_cache size 0 max 16384 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 Processor -- start 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 -- start start 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc31320a30 0x55dc31343cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31344210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc3132ba20 0x55dc3131b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81b25b0 con 0x55dc3132ba20 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81afe80 con 0x55dc31320a30 2024-04-24T08:22:57.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.340+0000 7f0302417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81adcd0 con 0x55dc31321350 2024-04-24T08:22:57.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc3132ba20 0x55dc3131b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc3132ba20 0x55dc3131b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32796/0 (socket says 172.21.15.146:32796) 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 -- 172.21.15.146:0/1621071495 learned_addr learned my addr 172.21.15.146:0/1621071495 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fcc00640 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc31320a30 0x55dc31343cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02f7e00640 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31344210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 -- 172.21.15.146:0/1621071495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 msgr2=0x55dc31344210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31344210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 -- 172.21.15.146:0/1621071495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc31320a30 msgr2=0x55dc31343cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc31320a30 0x55dc31343cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.341+0000 7f02fd600640 1 -- 172.21.15.146:0/1621071495 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc30f983f0 con 0x55dc3132ba20 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.342+0000 7f02fd600640 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc3132ba20 0x55dc3131b740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009920 tx=0x7f02f0004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309305d2c67b5d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.342+0000 7f02e9200640 1 -- 172.21.15.146:0/1621071495 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f002a070 con 0x55dc3132ba20 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.342+0000 7f02e9200640 1 -- 172.21.15.146:0/1621071495 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f000ee10 con 0x55dc3132ba20 2024-04-24T08:22:57.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.342+0000 7f02e9200640 1 -- 172.21.15.146:0/1621071495 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f00055d0 con 0x55dc3132ba20 2024-04-24T08:22:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 -- 172.21.15.146:0/1621071495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc3132ba20 msgr2=0x55dc3131b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc3132ba20 0x55dc3131b740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009920 tx=0x7f02f0004160 comp rx=0 tx=0).stop 2024-04-24T08:22:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 -- 172.21.15.146:0/1621071495 shutdown_connections 2024-04-24T08:22:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31344210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc3132ba20 0x55dc3131b740 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 --2- 172.21.15.146:0/1621071495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc31320a30 0x55dc31343cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 -- 172.21.15.146:0/1621071495 >> 172.21.15.146:0/1621071495 conn(0x55dc3131cd50 msgr2=0x55dc3131d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 -- 172.21.15.146:0/1621071495 shutdown_connections 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 -- 172.21.15.146:0/1621071495 wait complete. 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 -- start start 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.343+0000 7f0302417740 1 Processor -- start 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f0302417740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31343ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f0302417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31341380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f0302417740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f0302417740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81ae340 con 0x55dc31320a30 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f0302417740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81af7b0 con 0x55dc3132ba20 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f0302417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81aea10 con 0x55dc31321350 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f0302417740 10 client.0 _ll_register_callbacks cb 0x7f02ec031850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31343ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31343ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32798/0 (socket says 172.21.15.146:32798) 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 -- 172.21.15.146:0/2032763715 learned_addr learned my addr 172.21.15.146:0/2032763715 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:57.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fd600640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02f7e00640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31341380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 -- 172.21.15.146:0/2032763715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 msgr2=0x55dc31341380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31341380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 -- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 msgr2=0x55dc313418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.344+0000 7f02fcc00640 1 -- 172.21.15.146:0/2032763715 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc30f580f0 con 0x55dc31320a30 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f02fd600640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f02fcc00640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31343ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0059d0 tx=0x7f02ec003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879481462f1686d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f02de000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f02de000640 1 -- 172.21.15.146:0/2032763715 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec061070 con 0x55dc31320a30 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f000ee10 con 0x55dc31320a30 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f02de000640 1 -- 172.21.15.146:0/2032763715 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f02ec004040 con 0x55dc31320a30 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f02de000640 1 -- 172.21.15.146:0/2032763715 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f02ec06d050 con 0x55dc31320a30 2024-04-24T08:22:57.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f0302417740 10 client.15555 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:57.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.345+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dc3124b810 con 0x55dc31320a30 2024-04-24T08:22:57.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f02de000640 1 -- 172.21.15.146:0/2032763715 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f000ee10 con 0x55dc31320a30 2024-04-24T08:22:57.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f02de000640 1 -- 172.21.15.146:0/2032763715 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f02ec03d450 con 0x55dc31320a30 2024-04-24T08:22:57.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f0302417740 10 client.15555 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:57.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f0302417740 10 client.15555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f0302417740 20 client.15555 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f0302417740 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc313a2820 0x55dc313a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f0302417740 4 client.15555 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dc3128da80 con 0x55dc313a2820 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.346+0000 7f02f7e00640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc313a2820 0x55dc313a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f02de000640 10 client.15555 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f02f7e00640 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc313a2820 0x55dc313a4c60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6469bbcd21abdf80 server_cookie=bc7d3a5b11127f7c in_seq=0 out_seq=0 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f02de000640 1 -- 172.21.15.146:0/2032763715 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dc3128da80 con 0x55dc313a2820 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f02de000640 10 client.15555 handle_command_reply: tid=0 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 client.15555 shutdown 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 msgr2=0x55dc31343ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31343ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0059d0 tx=0x7f02ec003820 comp rx=0 tx=0).stop 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 shutdown_connections 2024-04-24T08:22:57.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dc313a2820 0x55dc313a4c60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc31321350 0x55dc31341380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dc3132ba20 0x55dc313418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f02de000640 0 client.15555 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 --2- 172.21.15.146:0/2032763715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dc31320a30 0x55dc31343ff0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.347+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 >> 172.21.15.146:0/2032763715 conn(0x55dc3131cd50 msgr2=0x55dc3131b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 shutdown_connections 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f0302417740 1 -- 172.21.15.146:0/2032763715 wait complete. 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f0302417740 20 client.15555 trim_cache size 0 max 16384 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:22:57.352 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:22:57.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02e00a3d20 msgr2=0x7f02e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02e00a3d20 0x7f02e00a61e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009230 tx=0x7f02ec005a20 comp rx=0 tx=0).stop 2024-04-24T08:22:57.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81071e0 msgr2=0x7f02f81a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81071e0 0x7f02f81a0b50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f02e400e9c0 tx=0x7f02e400ee80 comp rx=0 tx=0).stop 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 shutdown_connections 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02e00a3d20 0x7f02e00a61e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02f81071e0 0x7f02f81a0b50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02f819a7d0 0x7f02f819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 --2- 172.21.15.146:0/4247454821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02f8102110 0x7f02f81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.348+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 >> 172.21.15.146:0/4247454821 conn(0x7f02f80fd9d0 msgr2=0x7f02f80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.349+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 shutdown_connections 2024-04-24T08:22:57.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.349+0000 7f02fe000640 1 -- 172.21.15.146:0/4247454821 wait complete. 2024-04-24T08:22:57.365 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:22:57.366 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:22:57.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 -- 172.21.15.146:0/1809625559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca01071a0 msgr2=0x7f9ca010ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 --2- 172.21.15.146:0/1809625559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca01071a0 0x7f9ca010ed40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009ca0 tx=0x7f9c9802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 -- 172.21.15.146:0/1809625559 shutdown_connections 2024-04-24T08:22:57.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 --2- 172.21.15.146:0/1809625559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca01071a0 0x7f9ca010ed40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 --2- 172.21.15.146:0/1809625559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca0069db0 0x7f9ca0106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 --2- 172.21.15.146:0/1809625559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca0069400 0x7f9ca00697e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 -- 172.21.15.146:0/1809625559 >> 172.21.15.146:0/1809625559 conn(0x7f9ca0076840 msgr2=0x7f9ca0076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 -- 172.21.15.146:0/1809625559 shutdown_connections 2024-04-24T08:22:57.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.549+0000 7f9ca5800640 1 -- 172.21.15.146:0/1809625559 wait complete. 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.550+0000 7f9ca5800640 1 Processor -- start 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.550+0000 7f9ca5800640 1 -- start start 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.550+0000 7f9ca5800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0069400 0x7f9ca01a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.550+0000 7f9ca5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca01071a0 0x7f9ca01a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9ca5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca019a740 0x7f9ca019abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9ca5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0112280 con 0x7f9ca019a740 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9ca5800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0112100 con 0x7f9ca0069400 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9ca5800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca0112400 con 0x7f9ca01071a0 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0069400 0x7f9ca01a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0069400 0x7f9ca01a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57098/0 (socket says 172.21.15.146:57098) 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 -- 172.21.15.146:0/2730149348 learned_addr learned my addr 172.21.15.146:0/2730149348 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9ca4e00640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca019a740 0x7f9ca019abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9f400640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca01071a0 0x7f9ca01a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 -- 172.21.15.146:0/2730149348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca01071a0 msgr2=0x7f9ca01a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca01071a0 0x7f9ca01a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 -- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca019a740 msgr2=0x7f9ca019abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca019a740 0x7f9ca019abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9c9fe00640 1 -- 172.21.15.146:0/2730149348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca019b340 con 0x7f9ca0069400 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.551+0000 7f9ca4e00640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca019a740 0x7f9ca019abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.552+0000 7f9c9fe00640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0069400 0x7f9ca01a0680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000e9f0 tx=0x7f9c9000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.552+0000 7f9c9cc00640 1 -- 172.21.15.146:0/2730149348 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9000cd80 con 0x7f9ca0069400 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.552+0000 7f9c9cc00640 1 -- 172.21.15.146:0/2730149348 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c90005500 con 0x7f9ca0069400 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.552+0000 7f9c9cc00640 1 -- 172.21.15.146:0/2730149348 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90010690 con 0x7f9ca0069400 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.552+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01a7420 con 0x7f9ca0069400 2024-04-24T08:22:57.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.552+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01a77d0 con 0x7f9ca0069400 2024-04-24T08:22:57.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.553+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca0103550 con 0x7f9ca0069400 2024-04-24T08:22:57.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.555+0000 7f9c9cc00640 1 -- 172.21.15.146:0/2730149348 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f9c900046c0 con 0x7f9ca0069400 2024-04-24T08:22:57.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.556+0000 7f9c9cc00640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9c840a3d20 0x7f9c840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:57.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.557+0000 7f9c9cc00640 1 -- 172.21.15.146:0/2730149348 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9c900d0cf0 con 0x7f9ca0069400 2024-04-24T08:22:57.557 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.557+0000 7f9c9f400640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9c840a3d20 0x7f9c840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:57.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.557+0000 7f9c9f400640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9c840a3d20 0x7f9c840a61e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ca019bf90 tx=0x7f9c8c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:57.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.561+0000 7f9c9cc00640 1 -- 172.21.15.146:0/2730149348 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c9009d340 con 0x7f9ca0069400 2024-04-24T08:22:57.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.806+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ca0068d40 con 0x7f9ca0069400 2024-04-24T08:22:57.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.808+0000 7f9c9cc00640 1 -- 172.21.15.146:0/2730149348 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11891 (secure 0 0 0) 0x7f9c9009cc40 con 0x7f9ca0069400 2024-04-24T08:22:57.809 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:22:57.809 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:21:10.741287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24553,24568]},"id":1}]} 2024-04-24T08:22:57.809 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 24 2024-04-24T08:22:57.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9c840a3d20 msgr2=0x7f9c840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9c840a3d20 0x7f9c840a61e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ca019bf90 tx=0x7f9c8c005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:57.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0069400 msgr2=0x7f9ca01a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:57.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0069400 0x7f9ca01a0680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000e9f0 tx=0x7f9c9000edf0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 shutdown_connections 2024-04-24T08:22:57.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9c840a3d20 0x7f9c840a61e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca01071a0 0x7f9ca01a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca019a740 0x7f9ca019abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 --2- 172.21.15.146:0/2730149348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ca0069400 0x7f9ca01a0680 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:57.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.811+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 >> 172.21.15.146:0/2730149348 conn(0x7f9ca0076840 msgr2=0x7f9ca00fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:57.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.812+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 shutdown_connections 2024-04-24T08:22:57.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:57.812+0000 7f9ca5800640 1 -- 172.21.15.146:0/2730149348 wait complete. 2024-04-24T08:22:57.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:22:58.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 -- 172.21.15.146:0/2893836534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0102ac0 msgr2=0x7f03d0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 --2- 172.21.15.146:0/2893836534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0102ac0 0x7f03d0106b00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009900 tx=0x7f03bc02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 -- 172.21.15.146:0/2893836534 shutdown_connections 2024-04-24T08:22:58.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 --2- 172.21.15.146:0/2893836534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d01071e0 0x7f03d010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 --2- 172.21.15.146:0/2893836534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d0102ac0 0x7f03d0106b00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 --2- 172.21.15.146:0/2893836534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03d0102110 0x7f03d01024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 -- 172.21.15.146:0/2893836534 >> 172.21.15.146:0/2893836534 conn(0x7f03d00fd9d0 msgr2=0x7f03d00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 -- 172.21.15.146:0/2893836534 shutdown_connections 2024-04-24T08:22:58.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.036+0000 7f03cfe00640 1 -- 172.21.15.146:0/2893836534 wait complete. 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.037+0000 7f03cfe00640 1 Processor -- start 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cfe00640 1 -- start start 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d0102110 0x7f03d01a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03d01071e0 0x7f03d01a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d019a800 0x7f03d019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03d01122a0 con 0x7f03d019a800 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cfe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03d0112120 con 0x7f03d0102110 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cfe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03d0112420 con 0x7f03d01071e0 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d0102110 0x7f03d01a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03ce000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03d01071e0 0x7f03d01a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d0102110 0x7f03d01a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57116/0 (socket says 172.21.15.146:57116) 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cea00640 1 -- 172.21.15.146:0/684506968 learned_addr learned my addr 172.21.15.146:0/684506968 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.038+0000 7f03cf400640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d019a800 0x7f03d019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cea00640 1 -- 172.21.15.146:0/684506968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03d01071e0 msgr2=0x7f03d01a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cea00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03d01071e0 0x7f03d01a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cea00640 1 -- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d019a800 msgr2=0x7f03d019acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cea00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d019a800 0x7f03d019acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cea00640 1 -- 172.21.15.146:0/684506968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d019b390 con 0x7f03d0102110 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cf400640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d019a800 0x7f03d019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cea00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d0102110 0x7f03d01a0690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f03c000e9f0 tx=0x7f03c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03c7400640 1 -- 172.21.15.146:0/684506968 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c000cd80 con 0x7f03d0102110 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d019b680 con 0x7f03d0102110 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03c7400640 1 -- 172.21.15.146:0/684506968 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03c0010910 con 0x7f03d0102110 2024-04-24T08:22:58.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03c7400640 1 -- 172.21.15.146:0/684506968 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c00188f0 con 0x7f03d0102110 2024-04-24T08:22:58.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.039+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d01a7880 con 0x7f03d0102110 2024-04-24T08:22:58.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.042+0000 7f03d87d4740 1 Processor -- start 2024-04-24T08:22:58.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03d87d4740 1 -- start start 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03c7400640 1 -- 172.21.15.146:0/684506968 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f03c0018af0 con 0x7f03d0102110 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e789df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e78a8b30 0x55f5e78aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03d87d4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f5e78031d0 con 0x55f5e789e460 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03d87d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f5e77a9450 con 0x55f5e78a8b30 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03d87d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f5e77ff190 con 0x55f5e789db40 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03cf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e789df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03cf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e789df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48488/0 (socket says 172.21.15.146:48488) 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03cf400640 1 -- 172.21.15.146:0/514965803 learned_addr learned my addr 172.21.15.146:0/514965803 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03cea00640 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.043+0000 7f03ce000640 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e78a8b30 0x55f5e78aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03cf400640 1 -- 172.21.15.146:0/514965803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e78a8b30 msgr2=0x55f5e78aaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03cf400640 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e78a8b30 0x55f5e78aaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03cf400640 1 -- 172.21.15.146:0/514965803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 msgr2=0x55f5e78a85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03cf400640 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03cf400640 1 -- 172.21.15.146:0/514965803 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5e7866310 con 0x55f5e789db40 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03cea00640 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03cf400640 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e789df20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03c8005f40 tx=0x7f03c8010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dcf427e4cf0e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03bbe00640 1 -- 172.21.15.146:0/514965803 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8011700 con 0x55f5e789db40 2024-04-24T08:22:58.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03bbe00640 1 -- 172.21.15.146:0/514965803 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03c8011d80 con 0x55f5e789db40 2024-04-24T08:22:58.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.044+0000 7f03bbe00640 1 -- 172.21.15.146:0/514965803 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c800e3c0 con 0x55f5e789db40 2024-04-24T08:22:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03c7400640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f03b40a3d20 0x7f03b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03c7400640 1 -- 172.21.15.146:0/684506968 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f03c00d0ca0 con 0x7f03d0102110 2024-04-24T08:22:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 -- 172.21.15.146:0/514965803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 msgr2=0x55f5e789df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e789df20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03c8005f40 tx=0x7f03c8010860 comp rx=0 tx=0).stop 2024-04-24T08:22:58.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03cea00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f03b40a3d20 0x7f03b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 -- 172.21.15.146:0/514965803 shutdown_connections 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e78a8b30 0x55f5e78aaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 --2- 172.21.15.146:0/514965803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e789df20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 -- 172.21.15.146:0/514965803 >> 172.21.15.146:0/514965803 conn(0x55f5e7899e60 msgr2=0x55f5e789a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 -- 172.21.15.146:0/514965803 shutdown_connections 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.045+0000 7f03d87d4740 1 -- 172.21.15.146:0/514965803 wait complete. 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.046+0000 7f03cea00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f03b40a3d20 0x7f03b40a61e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55f5e789f100 tx=0x7f03c0009310 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.046+0000 7f03d87d4740 1 -- start start 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.046+0000 7f03d87d4740 1 Processor -- start 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.046+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789e460 0x55f5e78a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78a6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03d87d4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f5e78ab9c0 con 0x55f5e78a8b30 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03d87d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f5e77bc810 con 0x55f5e789db40 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03d87d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f5e78abb40 con 0x55f5e789e460 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03d87d4740 10 client.0 _ll_register_callbacks cb 0x55f5e778a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789e460 0x55f5e78a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789e460 0x55f5e78a5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48490/0 (socket says 172.21.15.146:48490) 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 -- 172.21.15.146:0/2541719702 learned_addr learned my addr 172.21.15.146:0/2541719702 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03cf400640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03cf400640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 -- 172.21.15.146:0/2541719702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 msgr2=0x55f5e78a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 -- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 msgr2=0x55f5e78a6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78a6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03ce000640 1 -- 172.21.15.146:0/2541719702 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5e77cd3f0 con 0x55f5e789e460 2024-04-24T08:22:58.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.047+0000 7f03cf400640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03ce000640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789e460 0x55f5e78a5e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009900 tx=0x7f03bc0043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be5741faabff53fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03b8c00640 1 -- 172.21.15.146:0/2541719702 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc035040 con 0x55f5e789e460 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03b8c00640 1 -- 172.21.15.146:0/2541719702 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f03bc005450 con 0x55f5e789e460 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03c8011d80 con 0x55f5e789e460 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03b8c00640 1 -- 172.21.15.146:0/2541719702 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f03bc005650 con 0x55f5e789e460 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03d87d4740 10 client.25705 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.048+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f5e75153f0 con 0x55f5e789e460 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03b8c00640 1 -- 172.21.15.146:0/2541719702 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03c8011d80 con 0x55f5e789e460 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03b8c00640 1 -- 172.21.15.146:0/2541719702 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f03bc003a30 con 0x55f5e789e460 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03d87d4740 10 client.25705 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03d87d4740 10 client.25705 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03d87d4740 20 client.25705 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:58.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03d87d4740 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e78b8cd0 0x55f5e78bb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03d87d4740 4 client.25705 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:58.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5e7810bf0 con 0x55f5e78b8cd0 2024-04-24T08:22:58.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.049+0000 7f03cf400640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e78b8cd0 0x55f5e78bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.050+0000 7f03cf400640 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e78b8cd0 0x55f5e78bb130 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c75893ad732fd59f server_cookie=79efa2c3df911dfa in_seq=0 out_seq=0 2024-04-24T08:22:58.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.050+0000 7f03b8c00640 10 client.25705 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:58.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.050+0000 7f03b8c00640 1 -- 172.21.15.146:0/2541719702 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f5e7810bf0 con 0x55f5e78b8cd0 2024-04-24T08:22:58.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.050+0000 7f03b8c00640 10 client.25705 handle_command_reply: tid=0 2024-04-24T08:22:58.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 client.25705 shutdown 2024-04-24T08:22:58.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789e460 msgr2=0x55f5e78a5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789e460 0x55f5e78a5e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009900 tx=0x7f03bc0043b0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 shutdown_connections 2024-04-24T08:22:58.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e78b8cd0 0x55f5e78bb130 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789e460 0x55f5e78a5e30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 --2- 172.21.15.146:0/2541719702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03b8c00640 0 client.25705 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 >> 172.21.15.146:0/2541719702 conn(0x55f5e7899e60 msgr2=0x55f5e789fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 shutdown_connections 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 1 -- 172.21.15.146:0/2541719702 wait complete. 2024-04-24T08:22:58.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.051+0000 7f03d87d4740 20 client.25705 trim_cache size 0 max 16384 2024-04-24T08:22:58.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.065+0000 7f03d87d4740 1 Processor -- start 2024-04-24T08:22:58.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03d87d4740 1 -- start start 2024-04-24T08:22:58.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e78b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789e460 0x55f5e78a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03d87d4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03d01b2650 con 0x55f5e78a8b30 2024-04-24T08:22:58.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03d87d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03d01aff20 con 0x55f5e789e460 2024-04-24T08:22:58.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03d87d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03d01add70 con 0x55f5e789db40 2024-04-24T08:22:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03cf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03ce000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789e460 0x55f5e78a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03cf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e78b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03cf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e78b4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48506/0 (socket says 172.21.15.146:48506) 2024-04-24T08:22:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.066+0000 7f03cf400640 1 -- 172.21.15.146:0/1251874800 learned_addr learned my addr 172.21.15.146:0/1251874800 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.067+0000 7f03ce000640 1 -- 172.21.15.146:0/1251874800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 msgr2=0x55f5e78b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.067+0000 7f03ce000640 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e78b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.067+0000 7f03ce000640 1 -- 172.21.15.146:0/1251874800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 msgr2=0x55f5e78c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.067+0000 7f03ce000640 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78c0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.067+0000 7f03ce000640 1 -- 172.21.15.146:0/1251874800 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5e75df7d0 con 0x55f5e789e460 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.067+0000 7f03cf400640 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e78b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.067+0000 7f03ce000640 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789e460 0x55f5e78a02a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f03bc003e30 tx=0x7f03bc005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c05fd142a3d03d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.068+0000 7f03bb400640 1 -- 172.21.15.146:0/1251874800 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc035030 con 0x55f5e789e460 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.068+0000 7f03bb400640 1 -- 172.21.15.146:0/1251874800 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03bc041070 con 0x55f5e789e460 2024-04-24T08:22:58.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.068+0000 7f03bb400640 1 -- 172.21.15.146:0/1251874800 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc003870 con 0x55f5e789e460 2024-04-24T08:22:58.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.068+0000 7f03d87d4740 1 -- 172.21.15.146:0/1251874800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789e460 msgr2=0x55f5e78a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.068+0000 7f03d87d4740 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789e460 0x55f5e78a02a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f03bc003e30 tx=0x7f03bc005a20 comp rx=0 tx=0).stop 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 -- 172.21.15.146:0/1251874800 shutdown_connections 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e78a8b30 0x55f5e78c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789e460 0x55f5e78a02a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 --2- 172.21.15.146:0/1251874800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e789db40 0x55f5e78b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 -- 172.21.15.146:0/1251874800 >> 172.21.15.146:0/1251874800 conn(0x55f5e7899e60 msgr2=0x55f5e7898f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 -- 172.21.15.146:0/1251874800 shutdown_connections 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 -- 172.21.15.146:0/1251874800 wait complete. 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 -- start start 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.069+0000 7f03d87d4740 1 Processor -- start 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03d87d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e78a8b30 0x55f5e78aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03d87d4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03d01ae3e0 con 0x55f5e789e460 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03d87d4740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03d01af850 con 0x55f5e789db40 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03d87d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03d01aeab0 con 0x55f5e78a8b30 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03d87d4740 10 client.0 _ll_register_callbacks cb 0x7f03bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03cf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03ce000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03cf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57158/0 (socket says 172.21.15.146:57158) 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03cf400640 1 -- 172.21.15.146:0/146481697 learned_addr learned my addr 172.21.15.146:0/146481697 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03cf400640 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e78a8b30 0x55f5e78aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.070+0000 7f03ce000640 1 -- 172.21.15.146:0/146481697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e78a8b30 msgr2=0x55f5e78aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03ce000640 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e78a8b30 0x55f5e78aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03ce000640 1 -- 172.21.15.146:0/146481697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 msgr2=0x55f5e78a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03ce000640 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03ce000640 1 -- 172.21.15.146:0/146481697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5e78af150 con 0x55f5e789e460 2024-04-24T08:22:58.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03ce000640 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a9f80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55f5e78a6da0 tx=0x7f03bc031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed269eed509b66b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:58.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03b8c00640 1 -- 172.21.15.146:0/146481697 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc035070 con 0x55f5e789e460 2024-04-24T08:22:58.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03bc041070 con 0x55f5e789e460 2024-04-24T08:22:58.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03b8c00640 1 -- 172.21.15.146:0/146481697 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f03bc031700 con 0x55f5e789e460 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03b8c00640 1 -- 172.21.15.146:0/146481697 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f03bc068050 con 0x55f5e789e460 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03d87d4740 10 client.15579 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.071+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f5e791f640 con 0x55f5e789e460 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03b8c00640 1 -- 172.21.15.146:0/146481697 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03bc041070 con 0x55f5e789e460 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03b8c00640 1 -- 172.21.15.146:0/146481697 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f03bc058070 con 0x55f5e789e460 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03d87d4740 10 client.15579 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03d87d4740 10 client.15579 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:22:58.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03d87d4740 20 client.15579 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03d87d4740 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e791fc90 0x55f5e7922080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03d87d4740 4 client.15579 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f5e7899cb0 con 0x55f5e791fc90 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.072+0000 7f03cf400640 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e791fc90 0x55f5e7922080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03cf400640 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e791fc90 0x55f5e7922080 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8246d9c669ed9668 server_cookie=85868b96f254f328 in_seq=0 out_seq=0 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03b8c00640 10 client.15579 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03b8c00640 1 -- 172.21.15.146:0/146481697 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f5e7899cb0 con 0x55f5e791fc90 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03b8c00640 10 client.15579 handle_command_reply: tid=0 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 client.15579 shutdown 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 msgr2=0x55f5e78a9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a9f80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55f5e78a6da0 tx=0x7f03bc031480 comp rx=0 tx=0).stop 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 shutdown_connections 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55f5e791fc90 0x55f5e7922080 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f5e78a8b30 0x55f5e78aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f5e789e460 0x55f5e78a9f80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 --2- 172.21.15.146:0/146481697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f5e789db40 0x55f5e78a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.073+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 >> 172.21.15.146:0/146481697 conn(0x55f5e7899e60 msgr2=0x55f5e78986c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03b8c00640 0 client.15579 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:22:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 shutdown_connections 2024-04-24T08:22:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03d87d4740 1 -- 172.21.15.146:0/146481697 wait complete. 2024-04-24T08:22:58.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03d87d4740 20 client.15579 trim_cache size 0 max 16384 2024-04-24T08:22:58.078 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f03b40a3d20 msgr2=0x7f03b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03cfe00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f03b40a3d20 0x7f03b40a61e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55f5e789f100 tx=0x7f03c0009310 comp rx=0 tx=0).stop 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d0102110 msgr2=0x7f03d01a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.074+0000 7f03cfe00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d0102110 0x7f03d01a0690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f03c000e9f0 tx=0x7f03c000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 shutdown_connections 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f03b40a3d20 0x7f03b40a61e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03d01071e0 0x7f03d01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03d019a800 0x7f03d019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 --2- 172.21.15.146:0/684506968 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03d0102110 0x7f03d01a0690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 >> 172.21.15.146:0/684506968 conn(0x7f03d00fd9d0 msgr2=0x7f03d00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 shutdown_connections 2024-04-24T08:22:58.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.075+0000 7f03cfe00640 1 -- 172.21.15.146:0/684506968 wait complete. 2024-04-24T08:22:58.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 -- 172.21.15.146:0/1995933950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8141030e0 msgr2=0x7ff8141034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 --2- 172.21.15.146:0/1995933950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8141030e0 0x7ff8141034c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009ca0 tx=0x7ff80c02a950 comp rx=0 tx=0).stop 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 -- 172.21.15.146:0/1995933950 shutdown_connections 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 --2- 172.21.15.146:0/1995933950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff81410c4b0 0x7ff81410e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 --2- 172.21.15.146:0/1995933950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff814103a00 0x7ff81410bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 --2- 172.21.15.146:0/1995933950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8141030e0 0x7ff8141034c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 -- 172.21.15.146:0/1995933950 >> 172.21.15.146:0/1995933950 conn(0x7ff8140fc880 msgr2=0x7ff8140feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 -- 172.21.15.146:0/1995933950 shutdown_connections 2024-04-24T08:22:58.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.324+0000 7ff819a00640 1 -- 172.21.15.146:0/1995933950 wait complete. 2024-04-24T08:22:58.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.325+0000 7ff819a00640 1 Processor -- start 2024-04-24T08:22:58.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.325+0000 7ff819a00640 1 -- start start 2024-04-24T08:22:58.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.325+0000 7ff819a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8141030e0 0x7ff814198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.325+0000 7ff819a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814103a00 0x7ff814198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff819a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff81410c4b0 0x7ff81419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff819a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141122b0 con 0x7ff8141030e0 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff819a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff814112130 con 0x7ff814103a00 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff819a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff814112430 con 0x7ff81410c4b0 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8141030e0 0x7ff814198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8141030e0 0x7ff814198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32900/0 (socket says 172.21.15.146:32900) 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 -- 172.21.15.146:0/4212383883 learned_addr learned my addr 172.21.15.146:0/4212383883 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff819000640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff81410c4b0 0x7ff81419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813400640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814103a00 0x7ff814198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 -- 172.21.15.146:0/4212383883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff81410c4b0 msgr2=0x7ff81419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff81410c4b0 0x7ff81419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 -- 172.21.15.146:0/4212383883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814103a00 msgr2=0x7ff814198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814103a00 0x7ff814198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813e00640 1 -- 172.21.15.146:0/4212383883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81419d730 con 0x7ff8141030e0 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff819000640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff81410c4b0 0x7ff81419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.326+0000 7ff813400640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814103a00 0x7ff814198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.327+0000 7ff813e00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8141030e0 0x7ff814198790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff80c005e30 tx=0x7ff80c003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.327+0000 7ff810c00640 1 -- 172.21.15.146:0/4212383883 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c034040 con 0x7ff8141030e0 2024-04-24T08:22:58.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.327+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81419d9c0 con 0x7ff8141030e0 2024-04-24T08:22:58.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.327+0000 7ff810c00640 1 -- 172.21.15.146:0/4212383883 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff80c02fe20 con 0x7ff8141030e0 2024-04-24T08:22:58.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.327+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141a52b0 con 0x7ff8141030e0 2024-04-24T08:22:58.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.327+0000 7ff810c00640 1 -- 172.21.15.146:0/4212383883 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c004760 con 0x7ff8141030e0 2024-04-24T08:22:58.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.330+0000 7ff810c00640 1 -- 172.21.15.146:0/4212383883 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7ff80c037020 con 0x7ff8141030e0 2024-04-24T08:22:58.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff810c00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7f80a3d20 0x7ff7f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 Processor -- start 2024-04-24T08:22:58.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff810c00640 1 -- 172.21.15.146:0/4212383883 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ff80c0f1530 con 0x7ff8141030e0 2024-04-24T08:22:58.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 -- start start 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff813400640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7f80a3d20 0x7ff7f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e106e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e115a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e118330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55880e06c1d0 con 0x55880e106a20 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55880e012450 con 0x55880e115f40 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.332+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55880e068190 con 0x55880e107340 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32910/0 (socket says 172.21.15.146:32910) 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 -- 172.21.15.146:0/4105520712 learned_addr learned my addr 172.21.15.146:0/4105520712 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813400640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7f80a3d20 0x7ff7f80a61e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff8140feea0 tx=0x7ff7fc005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff819000640 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e118330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813400640 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e115a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 -- 172.21.15.146:0/4105520712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 msgr2=0x55880e115a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e115a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 -- 172.21.15.146:0/4105520712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 msgr2=0x55880e118330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e118330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 -- 172.21.15.146:0/4105520712 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55880e0cf310 con 0x55880e106a20 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff819000640 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e118330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.333+0000 7ff813e00640 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e106e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff80c000f80 tx=0x7ff80c0bd420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acce6d50159e4792 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.334+0000 7ff809600640 1 -- 172.21.15.146:0/4105520712 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c02c040 con 0x55880e106a20 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.334+0000 7ff809600640 1 -- 172.21.15.146:0/4105520712 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff80c0bdc90 con 0x55880e106a20 2024-04-24T08:22:58.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.334+0000 7ff809600640 1 -- 172.21.15.146:0/4105520712 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c0b9c00 con 0x55880e106a20 2024-04-24T08:22:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.334+0000 7ff81ddf7740 1 -- 172.21.15.146:0/4105520712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 msgr2=0x55880e106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e106e00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff80c000f80 tx=0x7ff80c0bd420 comp rx=0 tx=0).stop 2024-04-24T08:22:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 -- 172.21.15.146:0/4105520712 shutdown_connections 2024-04-24T08:22:58.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e118330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/4105520712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e106e00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 -- 172.21.15.146:0/4105520712 >> 172.21.15.146:0/4105520712 conn(0x55880e102e60 msgr2=0x55880e103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 -- 172.21.15.146:0/4105520712 shutdown_connections 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 -- 172.21.15.146:0/4105520712 wait complete. 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 -- start start 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.335+0000 7ff81ddf7740 1 Processor -- start 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e115930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e112c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e113230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55880e118b40 con 0x55880e115f40 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55880e025810 con 0x55880e107340 2024-04-24T08:22:58.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55880e118cc0 con 0x55880e106a20 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff81ddf7740 10 client.0 _ll_register_callbacks cb 0x55880dff3140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff813e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff813e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e115930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48540/0 (socket says 172.21.15.146:48540) 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff813e00640 1 -- 172.21.15.146:0/2222779559 learned_addr learned my addr 172.21.15.146:0/2222779559 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff813400640 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e112c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff819000640 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e113230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.336+0000 7ff813e00640 1 -- 172.21.15.146:0/2222779559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 msgr2=0x55880e112c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff813e00640 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e112c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff813e00640 1 -- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 msgr2=0x55880e113230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff813e00640 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e113230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff813e00640 1 -- 172.21.15.146:0/2222779559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55880e0363f0 con 0x55880e106a20 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff813e00640 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e115930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0d3040 tx=0x7ff80c0b9eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f02d5a8e22769f8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff802000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff802000640 1 -- 172.21.15.146:0/2222779559 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c02c040 con 0x55880e106a20 2024-04-24T08:22:58.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff80c0bdc90 con 0x55880e106a20 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff802000640 1 -- 172.21.15.146:0/2222779559 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ff80c0c5f80 con 0x55880e106a20 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff802000640 1 -- 172.21.15.146:0/2222779559 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff80c0bd590 con 0x55880e106a20 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff81ddf7740 10 client.25723 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55880dd7e3f0 con 0x55880e106a20 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.337+0000 7ff802000640 1 -- 172.21.15.146:0/2222779559 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff80c0bdc90 con 0x55880e106a20 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff802000640 1 -- 172.21.15.146:0/2222779559 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff80c04d030 con 0x55880e106a20 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff81ddf7740 10 client.25723 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff81ddf7740 10 client.25723 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff81ddf7740 20 client.25723 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e125cf0 0x55880e1280e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff81ddf7740 4 client.25723 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55880e079bf0 con 0x55880e125cf0 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.338+0000 7ff813400640 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e125cf0 0x55880e1280e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.339+0000 7ff813400640 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e125cf0 0x55880e1280e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f1f929edf27eb323 server_cookie=3c40024f2bb01a95 in_seq=0 out_seq=0 2024-04-24T08:22:58.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.339+0000 7ff802000640 10 client.25723 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.339+0000 7ff802000640 1 -- 172.21.15.146:0/2222779559 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55880e079bf0 con 0x55880e125cf0 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.339+0000 7ff802000640 10 client.25723 handle_command_reply: tid=0 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 client.25723 shutdown 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 msgr2=0x55880e115930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e115930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0d3040 tx=0x7ff80c0b9eb0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 shutdown_connections 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e125cf0 0x55880e1280e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e113230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff802000640 0 client.25723 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e112c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2222779559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e115930 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 >> 172.21.15.146:0/2222779559 conn(0x55880e102e60 msgr2=0x55880e108c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 shutdown_connections 2024-04-24T08:22:58.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2222779559 wait complete. 2024-04-24T08:22:58.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.340+0000 7ff81ddf7740 20 client.25723 trim_cache size 0 max 16384 2024-04-24T08:22:58.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 Processor -- start 2024-04-24T08:22:58.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 -- start start 2024-04-24T08:22:58.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e121a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e111c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e115460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ae1a0 con 0x55880e106a20 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141aba70 con 0x55880e115f40 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.368+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141a98c0 con 0x55880e107340 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e111c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e111c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48546/0 (socket says 172.21.15.146:48546) 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e121a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 -- 172.21.15.146:0/2723555506 learned_addr learned my addr 172.21.15.146:0/2723555506 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff819000640 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e115460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 -- 172.21.15.146:0/2723555506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 msgr2=0x55880e115460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e115460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 -- 172.21.15.146:0/2723555506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 msgr2=0x55880e121a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e121a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 -- 172.21.15.146:0/2723555506 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55880de487d0 con 0x55880e107340 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.369+0000 7ff813400640 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e111c20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55880e102710 tx=0x7ff7fc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfc711ebedba57d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.370+0000 7ff808c00640 1 -- 172.21.15.146:0/2723555506 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc029470 con 0x55880e107340 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.370+0000 7ff808c00640 1 -- 172.21.15.146:0/2723555506 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc02f070 con 0x55880e107340 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.370+0000 7ff808c00640 1 -- 172.21.15.146:0/2723555506 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc01d3f0 con 0x55880e107340 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2723555506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 msgr2=0x55880e111c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e111c20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55880e102710 tx=0x7ff7fc014000 comp rx=0 tx=0).stop 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2723555506 shutdown_connections 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e107340 0x55880e111c20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e115f40 0x55880e115460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/2723555506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e106a20 0x55880e121a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2723555506 >> 172.21.15.146:0/2723555506 conn(0x55880e102e60 msgr2=0x55880e111510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2723555506 shutdown_connections 2024-04-24T08:22:58.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 -- 172.21.15.146:0/2723555506 wait complete. 2024-04-24T08:22:58.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.371+0000 7ff81ddf7740 1 -- start start 2024-04-24T08:22:58.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.372+0000 7ff81ddf7740 1 Processor -- start 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.372+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e11e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.372+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e10fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.372+0000 7ff81ddf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e110400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.372+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141a9f30 con 0x55880e115f40 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.372+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141ab3a0 con 0x55880e107340 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.372+0000 7ff81ddf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8141aa600 con 0x55880e106a20 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff81ddf7740 10 client.0 _ll_register_callbacks cb 0x7ff80c0b9830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e110400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e110400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32936/0 (socket says 172.21.15.146:32936) 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff813400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e10fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 -- 172.21.15.146:0/3349908903 learned_addr learned my addr 172.21.15.146:0/3349908903 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff813e00640 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e11e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 -- 172.21.15.146:0/3349908903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 msgr2=0x55880e11e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e11e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 -- 172.21.15.146:0/3349908903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 msgr2=0x55880e10fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e10fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 -- 172.21.15.146:0/3349908903 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55880e110a40 con 0x55880e115f40 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff813400640 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e10fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff819000640 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e110400 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff804012cb0 tx=0x7ff80400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881b1fe3b92af96a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.373+0000 7ff802000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff802000640 1 -- 172.21.15.146:0/3349908903 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804023060 con 0x55880e115f40 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff802000640 1 -- 172.21.15.146:0/3349908903 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ff804004230 con 0x55880e115f40 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7fc02f070 con 0x55880e115f40 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff802000640 1 -- 172.21.15.146:0/3349908903 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff804012e60 con 0x55880e115f40 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff81ddf7740 10 client.15597 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55880e18bfb0 con 0x55880e115f40 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff802000640 1 -- 172.21.15.146:0/3349908903 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7fc02f070 con 0x55880e115f40 2024-04-24T08:22:58.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.374+0000 7ff802000640 1 -- 172.21.15.146:0/3349908903 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff804010430 con 0x55880e115f40 2024-04-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff81ddf7740 10 client.15597 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff81ddf7740 10 client.15597 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff81ddf7740 20 client.15597 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e18c630 0x55880e18ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff81ddf7740 4 client.15597 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55880e073a60 con 0x55880e18c630 2024-04-24T08:22:58.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff813e00640 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e18c630 0x55880e18ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff813e00640 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e18c630 0x55880e18ea20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=99a910ba6e70ae92 server_cookie=8daaa0794372aad5 in_seq=0 out_seq=0 2024-04-24T08:22:58.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.375+0000 7ff802000640 10 client.15597 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff802000640 1 -- 172.21.15.146:0/3349908903 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55880e073a60 con 0x55880e18c630 2024-04-24T08:22:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff802000640 10 client.15597 handle_command_reply: tid=0 2024-04-24T08:22:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 client.15597 shutdown 2024-04-24T08:22:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 msgr2=0x55880e110400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e110400 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff804012cb0 tx=0x7ff80400f790 comp rx=0 tx=0).stop 2024-04-24T08:22:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 shutdown_connections 2024-04-24T08:22:58.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55880e18c630 0x55880e18ea20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55880e115f40 0x55880e110400 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55880e107340 0x55880e10fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 --2- 172.21.15.146:0/3349908903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55880e106a20 0x55880e11e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff802000640 0 client.15597 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 >> 172.21.15.146:0/3349908903 conn(0x55880e102e60 msgr2=0x55880e108310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 shutdown_connections 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 1 -- 172.21.15.146:0/3349908903 wait complete. 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.376+0000 7ff81ddf7740 20 client.15597 trim_cache size 0 max 16384 2024-04-24T08:22:58.382 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:22:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7f80a3d20 msgr2=0x7ff7f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7f80a3d20 0x7ff7f80a61e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff8140feea0 tx=0x7ff7fc005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8141030e0 msgr2=0x7ff814198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8141030e0 0x7ff814198790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff80c005e30 tx=0x7ff80c003c90 comp rx=0 tx=0).stop 2024-04-24T08:22:58.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 shutdown_connections 2024-04-24T08:22:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7f80a3d20 0x7ff7f80a61e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff81410c4b0 0x7ff81419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff814103a00 0x7ff814198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 --2- 172.21.15.146:0/4212383883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8141030e0 0x7ff814198790 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 >> 172.21.15.146:0/4212383883 conn(0x7ff8140fc880 msgr2=0x7ff8140fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 shutdown_connections 2024-04-24T08:22:58.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.377+0000 7ff819a00640 1 -- 172.21.15.146:0/4212383883 wait complete. 2024-04-24T08:22:58.391 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:58 smithi146 ceph-mon[27805]: pgmap v56: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:22:58.391 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:22:58 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2730149348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:58.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:22:58.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:58 smithi177 ceph-mon[33046]: pgmap v56: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:22:58.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:22:58 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2730149348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 -- 172.21.15.146:0/2921375494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d14102110 msgr2=0x7f3d141024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2921375494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d14102110 0x7f3d141024f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009900 tx=0x7f3d0402a410 comp rx=0 tx=0).stop 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 -- 172.21.15.146:0/2921375494 shutdown_connections 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2921375494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d141071e0 0x7f3d1410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2921375494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d14102ac0 0x7f3d14106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2921375494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d14102110 0x7f3d141024f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 -- 172.21.15.146:0/2921375494 >> 172.21.15.146:0/2921375494 conn(0x7f3d140fd9d0 msgr2=0x7f3d140ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 -- 172.21.15.146:0/2921375494 shutdown_connections 2024-04-24T08:22:58.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.581+0000 7f3d18e00640 1 -- 172.21.15.146:0/2921375494 wait complete. 2024-04-24T08:22:58.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.582+0000 7f3d18e00640 1 Processor -- start 2024-04-24T08:22:58.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.582+0000 7f3d18e00640 1 -- start start 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.582+0000 7f3d18e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d14102110 0x7f3d141a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d18e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d141071e0 0x7f3d141a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d18e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d1419a7b0 0x7f3d1419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d18e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3d141122f0 con 0x7f3d14102110 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d18e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d14112170 con 0x7f3d141071e0 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d18e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d14112470 con 0x7f3d1419a7b0 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d13400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d14102110 0x7f3d141a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d141071e0 0x7f3d141a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d13400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d14102110 0x7f3d141a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32958/0 (socket says 172.21.15.146:32958) 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d13400640 1 -- 172.21.15.146:0/2426073253 learned_addr learned my addr 172.21.15.146:0/2426073253 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d13e00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d1419a7b0 0x7f3d1419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d12a00640 1 -- 172.21.15.146:0/2426073253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d1419a7b0 msgr2=0x7f3d1419ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d12a00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d1419a7b0 0x7f3d1419ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d12a00640 1 -- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d14102110 msgr2=0x7f3d141a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d12a00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d14102110 0x7f3d141a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.583+0000 7f3d12a00640 1 -- 172.21.15.146:0/2426073253 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1419b340 con 0x7f3d141071e0 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.584+0000 7f3d12a00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d141071e0 0x7f3d141a0b80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000e9c0 tx=0x7f3d0000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.584+0000 7f3d0be00640 1 -- 172.21.15.146:0/2426073253 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000ccc0 con 0x7f3d141071e0 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.584+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1419b630 con 0x7f3d141071e0 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.584+0000 7f3d0be00640 1 -- 172.21.15.146:0/2426073253 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d00010910 con 0x7f3d141071e0 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.584+0000 7f3d0be00640 1 -- 172.21.15.146:0/2426073253 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d000188c0 con 0x7f3d141071e0 2024-04-24T08:22:58.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.584+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d141a7810 con 0x7f3d141071e0 2024-04-24T08:22:58.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.588+0000 7f3d1d270740 1 Processor -- start 2024-04-24T08:22:58.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.588+0000 7f3d1d270740 1 -- start start 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394eb40 0x557ee394ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee39595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 0x557ee395bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d1d270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557ee38b41d0 con 0x557ee394eb40 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d1d270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557ee385a450 con 0x557ee394f460 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d1d270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ee38b0190 con 0x557ee3959b30 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394eb40 0x557ee394ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394eb40 0x557ee394ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:32970/0 (socket says 172.21.15.146:32970) 2024-04-24T08:22:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13e00640 1 -- 172.21.15.146:0/3722807780 learned_addr learned my addr 172.21.15.146:0/3722807780 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13400640 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee39595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d0be00640 1 -- 172.21.15.146:0/2426073253 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3d00010ac0 con 0x7f3d141071e0 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13e00640 1 -- 172.21.15.146:0/3722807780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 msgr2=0x557ee395bf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13e00640 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 0x557ee395bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13e00640 1 -- 172.21.15.146:0/3722807780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 msgr2=0x557ee39595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.589+0000 7f3d13e00640 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee39595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.590+0000 7f3d13e00640 1 -- 172.21.15.146:0/3722807780 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee3917310 con 0x557ee394eb40 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.590+0000 7f3d13400640 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee39595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.590+0000 7f3d13e00640 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394eb40 0x557ee394ef20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009f50 tx=0x7f3d0c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40e5b9da50b7e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.590+0000 7f3d08c00640 1 -- 172.21.15.146:0/3722807780 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c004d20 con 0x557ee394eb40 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.590+0000 7f3d08c00640 1 -- 172.21.15.146:0/3722807780 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d0c004050 con 0x557ee394eb40 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.590+0000 7f3d08c00640 1 -- 172.21.15.146:0/3722807780 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c005240 con 0x557ee394eb40 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d0be00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3cf80a3d20 0x7f3cf80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d0be00640 1 -- 172.21.15.146:0/2426073253 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3d000d0b40 con 0x7f3d141071e0 2024-04-24T08:22:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d13400640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3cf80a3d20 0x7f3cf80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 -- 172.21.15.146:0/3722807780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394eb40 msgr2=0x557ee394ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394eb40 0x557ee394ef20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009f50 tx=0x7f3d0c0118d0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 -- 172.21.15.146:0/3722807780 shutdown_connections 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 0x557ee395bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee39595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3722807780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394eb40 0x557ee394ef20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 -- 172.21.15.146:0/3722807780 >> 172.21.15.146:0/3722807780 conn(0x557ee394ae60 msgr2=0x557ee394b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 -- 172.21.15.146:0/3722807780 shutdown_connections 2024-04-24T08:22:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.591+0000 7f3d1d270740 1 -- 172.21.15.146:0/3722807780 wait complete. 2024-04-24T08:22:58.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.592+0000 7f3d13400640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3cf80a3d20 0x7f3cf80a61e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x557ee3950100 tx=0x7f3d040046c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.592+0000 7f3d1d270740 1 -- start start 2024-04-24T08:22:58.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.592+0000 7f3d1d270740 1 Processor -- start 2024-04-24T08:22:58.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.592+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee3959570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.592+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee3956860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3956fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d1d270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557ee395c9c0 con 0x557ee3959b30 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d1d270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557ee386d810 con 0x557ee394f460 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d1d270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ee395cb40 con 0x557ee394eb40 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d1d270740 10 client.0 _ll_register_callbacks cb 0x557ee383b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3956fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee3956860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee3956860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57268/0 (socket says 172.21.15.146:57268) 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d12a00640 1 -- 172.21.15.146:0/506938632 learned_addr learned my addr 172.21.15.146:0/506938632 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d13400640 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee3959570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d13e00640 1 -- 172.21.15.146:0/506938632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 msgr2=0x557ee3959570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d13e00640 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee3959570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d13e00640 1 -- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 msgr2=0x557ee3956860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d13e00640 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee3956860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d13e00640 1 -- 172.21.15.146:0/506938632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee387e3f0 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.593+0000 7f3d12a00640 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee3956860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3d13e00640 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3956fa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c011dd0 tx=0x7f3d0c003cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7d771c0e39cec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3cfd600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3cfd600640 1 -- 172.21.15.146:0/506938632 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c028070 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3cfd600640 1 -- 172.21.15.146:0/506938632 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3d0c005040 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d0c004050 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3cfd600640 1 -- 172.21.15.146:0/506938632 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3d0c009be0 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3d1d270740 10 client.15609 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.594+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557ee35c63f0 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3cfd600640 1 -- 172.21.15.146:0/506938632 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d0c004050 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3cfd600640 1 -- 172.21.15.146:0/506938632 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3d0c024810 con 0x557ee3959b30 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3d1d270740 10 client.15609 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3d1d270740 10 client.15609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3d1d270740 20 client.15609 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3d1d270740 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee3969c60 0x557ee396c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3d1d270740 4 client.15609 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ee38c1bf0 con 0x557ee3969c60 2024-04-24T08:22:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.595+0000 7f3d13400640 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee3969c60 0x557ee396c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.596+0000 7f3d13400640 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee3969c60 0x557ee396c050 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40c9bfb2776ac881 server_cookie=c20f8a470c43a512 in_seq=0 out_seq=0 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.596+0000 7f3cfd600640 10 client.15609 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3cfd600640 1 -- 172.21.15.146:0/506938632 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ee38c1bf0 con 0x557ee3969c60 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3cfd600640 10 client.15609 handle_command_reply: tid=0 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 client.15609 shutdown 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 msgr2=0x557ee3956fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3956fa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c011dd0 tx=0x7f3d0c003cc0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 shutdown_connections 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee3969c60 0x557ee396c050 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3956fa0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3cfd600640 0 client.15609 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee3956860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 --2- 172.21.15.146:0/506938632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee3959570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 >> 172.21.15.146:0/506938632 conn(0x557ee394ae60 msgr2=0x557ee3950b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 shutdown_connections 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 1 -- 172.21.15.146:0/506938632 wait complete. 2024-04-24T08:22:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.597+0000 7f3d1d270740 20 client.15609 trim_cache size 0 max 16384 2024-04-24T08:22:58.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.612+0000 7f3d1d270740 1 Processor -- start 2024-04-24T08:22:58.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.612+0000 7f3d1d270740 1 -- start start 2024-04-24T08:22:58.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.612+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394eb40 0x557ee39658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.612+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394f460 0x557ee3949dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.612+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 0x557ee39590a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.612+0000 7f3d1d270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3d141b25f0 con 0x557ee394f460 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.612+0000 7f3d1d270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d141afec0 con 0x557ee394eb40 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d1d270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d141add10 con 0x557ee3959b30 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394f460 0x557ee3949dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394eb40 0x557ee39658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394eb40 0x557ee39658b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57284/0 (socket says 172.21.15.146:57284) 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 -- 172.21.15.146:0/3885691616 learned_addr learned my addr 172.21.15.146:0/3885691616 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13e00640 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 0x557ee39590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 -- 172.21.15.146:0/3885691616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 msgr2=0x557ee39590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 0x557ee39590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 -- 172.21.15.146:0/3885691616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394f460 msgr2=0x557ee3949dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394f460 0x557ee3949dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 -- 172.21.15.146:0/3885691616 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee36907d0 con 0x557ee394eb40 2024-04-24T08:22:58.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d12a00640 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394f460 0x557ee3949dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.613+0000 7f3d13400640 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394eb40 0x557ee39658b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3d04004650 tx=0x7f3d0403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe903633617aa1e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.614+0000 7f3cffe00640 1 -- 172.21.15.146:0/3885691616 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04035040 con 0x557ee394eb40 2024-04-24T08:22:58.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.614+0000 7f3cffe00640 1 -- 172.21.15.146:0/3885691616 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d0405a070 con 0x557ee394eb40 2024-04-24T08:22:58.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.614+0000 7f3cffe00640 1 -- 172.21.15.146:0/3885691616 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04003ce0 con 0x557ee394eb40 2024-04-24T08:22:58.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.614+0000 7f3d1d270740 1 -- 172.21.15.146:0/3885691616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394eb40 msgr2=0x557ee39658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394eb40 0x557ee39658b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3d04004650 tx=0x7f3d0403f000 comp rx=0 tx=0).stop 2024-04-24T08:22:58.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 -- 172.21.15.146:0/3885691616 shutdown_connections 2024-04-24T08:22:58.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3959b30 0x557ee39590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee394f460 0x557ee3949dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 --2- 172.21.15.146:0/3885691616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394eb40 0x557ee39658b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 -- 172.21.15.146:0/3885691616 >> 172.21.15.146:0/3885691616 conn(0x557ee394ae60 msgr2=0x557ee39496c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 -- 172.21.15.146:0/3885691616 shutdown_connections 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 -- 172.21.15.146:0/3885691616 wait complete. 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 -- start start 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.615+0000 7f3d1d270740 1 Processor -- start 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee395a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee395ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d1d270740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3960dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d1d270740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3d141ae380 con 0x557ee3959b30 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d1d270740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d141af7f0 con 0x557ee394f460 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d1d270740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d141aea50 con 0x557ee394eb40 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d1d270740 10 client.0 _ll_register_callbacks cb 0x7f3d0c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:58.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3960dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee395ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee395a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee395ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57286/0 (socket says 172.21.15.146:57286) 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 -- 172.21.15.146:0/2348586402 learned_addr learned my addr 172.21.15.146:0/2348586402 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d13400640 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee395a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48630/0 (socket says 172.21.15.146:48630) 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 -- 172.21.15.146:0/2348586402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 msgr2=0x557ee395a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee395a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 -- 172.21.15.146:0/2348586402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 msgr2=0x557ee3960dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3960dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d12a00640 1 -- 172.21.15.146:0/2348586402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee3961400 con 0x557ee394f460 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.616+0000 7f3d13e00640 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3960dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3d12a00640 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee395ac40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000cf10 tx=0x7f3d000d0e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f1d2b6a2ba7d6eb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3cfd600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3cfd600640 1 -- 172.21.15.146:0/2348586402 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0009c1f0 con 0x557ee394f460 2024-04-24T08:22:58.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d0405a070 con 0x557ee394f460 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3cfd600640 1 -- 172.21.15.146:0/2348586402 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3d00010430 con 0x557ee394f460 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3cfd600640 1 -- 172.21.15.146:0/2348586402 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3d00010630 con 0x557ee394f460 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3d1d270740 10 client.25751 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557ee39d0050 con 0x557ee394f460 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.617+0000 7f3cfd600640 1 -- 172.21.15.146:0/2348586402 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d0405a070 con 0x557ee394f460 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3cfd600640 1 -- 172.21.15.146:0/2348586402 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3d000a20c0 con 0x557ee394f460 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3d1d270740 10 client.25751 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3d1d270740 10 client.25751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3d1d270740 20 client.25751 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3d1d270740 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee39d0660 0x557ee39d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3d1d270740 4 client.25751 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:22:58.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ee38bba60 con 0x557ee39d0660 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.618+0000 7f3d13400640 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee39d0660 0x557ee39d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d13400640 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee39d0660 0x557ee39d2a80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b47a06949cb1b1a server_cookie=df0d6a64d755e57b in_seq=0 out_seq=0 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3cfd600640 10 client.25751 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3cfd600640 1 -- 172.21.15.146:0/2348586402 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ee38bba60 con 0x557ee39d0660 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3cfd600640 10 client.25751 handle_command_reply: tid=0 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 client.25751 shutdown 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 msgr2=0x557ee395ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee395ac40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000cf10 tx=0x7f3d000d0e60 comp rx=0 tx=0).stop 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 shutdown_connections 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ee39d0660 0x557ee39d2a80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ee3959b30 0x557ee3960dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ee394f460 0x557ee395ac40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3cfd600640 0 client.25751 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:58.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 --2- 172.21.15.146:0/2348586402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee394eb40 0x557ee395a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 >> 172.21.15.146:0/2348586402 conn(0x557ee394ae60 msgr2=0x557ee3955130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 shutdown_connections 2024-04-24T08:22:58.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 1 -- 172.21.15.146:0/2348586402 wait complete. 2024-04-24T08:22:58.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.619+0000 7f3d1d270740 20 client.25751 trim_cache size 0 max 16384 2024-04-24T08:22:58.624 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3cf80a3d20 msgr2=0x7f3cf80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3cf80a3d20 0x7f3cf80a61e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x557ee3950100 tx=0x7f3d040046c0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d141071e0 msgr2=0x7f3d141a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d141071e0 0x7f3d141a0b80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000e9c0 tx=0x7f3d0000ee80 comp rx=0 tx=0).stop 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 shutdown_connections 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3cf80a3d20 0x7f3cf80a61e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d1419a7b0 0x7f3d1419ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d141071e0 0x7f3d141a0b80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 --2- 172.21.15.146:0/2426073253 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d14102110 0x7f3d141a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 >> 172.21.15.146:0/2426073253 conn(0x7f3d140fd9d0 msgr2=0x7f3d140fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:58.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 shutdown_connections 2024-04-24T08:22:58.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:58.620+0000 7f3d18e00640 1 -- 172.21.15.146:0/2426073253 wait complete. 2024-04-24T08:22:58.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:58 smithi153 ceph-mon[32746]: pgmap v56: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:22:58.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:22:58 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2730149348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:58.750 INFO:tasks.workunit.client.0.smithi146.stderr:Updating files: 58% (7227/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 94% (11601/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-24T08:22:59.638 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:22:59.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:22:59.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr:Note: switching to '1f81fb397ae98da7563d451a78c61574c8f4e6e0'. 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr: git switch -c 2024-04-24T08:22:59.678 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:22:59.679 INFO:tasks.workunit.client.0.smithi146.stderr:Or undo this operation with: 2024-04-24T08:22:59.679 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:22:59.679 INFO:tasks.workunit.client.0.smithi146.stderr: git switch - 2024-04-24T08:22:59.679 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:22:59.679 INFO:tasks.workunit.client.0.smithi146.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T08:22:59.679 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:22:59.679 INFO:tasks.workunit.client.0.smithi146.stderr:HEAD is now at 1f81fb397ae DNM: trigger builds 2024-04-24T08:22:59.693 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-24T08:22:59.808 INFO:tasks.workunit.client.0.smithi146.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T08:22:59.816 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T08:22:59.816 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 -- 172.21.15.146:0/1292588938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c10c480 msgr2=0x7f3a4c10e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1292588938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c10c480 0x7f3a4c10e920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009ca0 tx=0x7f3a4402a950 comp rx=0 tx=0).stop 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 -- 172.21.15.146:0/1292588938 shutdown_connections 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1292588938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c10c480 0x7f3a4c10e920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1292588938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a4c103980 0x7f3a4c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1292588938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a4c103060 0x7f3a4c103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 -- 172.21.15.146:0/1292588938 >> 172.21.15.146:0/1292588938 conn(0x7f3a4c0fc880 msgr2=0x7f3a4c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:59.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 -- 172.21.15.146:0/1292588938 shutdown_connections 2024-04-24T08:22:59.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.841+0000 7f3a51c00640 1 -- 172.21.15.146:0/1292588938 wait complete. 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.842+0000 7f3a51c00640 1 Processor -- start 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.842+0000 7f3a51c00640 1 -- start start 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a4c103060 0x7f3a4c100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c103980 0x7f3a4c1011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a4c10c480 0x7f3a4c101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c112260 con 0x7f3a4c103980 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1120e0 con 0x7f3a4c103060 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1123e0 con 0x7f3a4c10c480 2024-04-24T08:22:59.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c103980 0x7f3a4c1011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c103980 0x7f3a4c1011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33030/0 (socket says 172.21.15.146:33030) 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 -- 172.21.15.146:0/1164484194 learned_addr learned my addr 172.21.15.146:0/1164484194 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4be00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a4c103060 0x7f3a4c100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51200640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a4c10c480 0x7f3a4c101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 -- 172.21.15.146:0/1164484194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a4c10c480 msgr2=0x7f3a4c101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a4c10c480 0x7f3a4c101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 -- 172.21.15.146:0/1164484194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a4c103060 msgr2=0x7f3a4c100cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a4c103060 0x7f3a4c100cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4b400640 1 -- 172.21.15.146:0/1164484194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c1a6be0 con 0x7f3a4c103980 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a51200640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a4c10c480 0x7f3a4c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.843+0000 7f3a4be00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a4c103060 0x7f3a4c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:59.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.844+0000 7f3a4b400640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c103980 0x7f3a4c1011f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800e9f0 tx=0x7f3a3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.844+0000 7f3a48c00640 1 -- 172.21.15.146:0/1164484194 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3800ccc0 con 0x7f3a4c103980 2024-04-24T08:22:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.844+0000 7f3a48c00640 1 -- 172.21.15.146:0/1164484194 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a38010910 con 0x7f3a4c103980 2024-04-24T08:22:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.844+0000 7f3a48c00640 1 -- 172.21.15.146:0/1164484194 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a38018920 con 0x7f3a4c103980 2024-04-24T08:22:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.844+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c1a6ed0 con 0x7f3a4c103980 2024-04-24T08:22:59.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.844+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c1a72e0 con 0x7f3a4c103980 2024-04-24T08:22:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 Processor -- start 2024-04-24T08:22:59.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 -- start start 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78aa650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78b7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f2a780fe70 con 0x55f2a78b5260 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f2a77b6380 con 0x55f2a78aab90 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a5600f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f2a780be30 con 0x55f2a78aa270 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a48c00640 1 -- 172.21.15.146:0/1164484194 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3a38010430 con 0x7f3a4c103980 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.848+0000 7f3a51200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78aa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a51200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78aa650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48666/0 (socket says 172.21.15.146:48666) 2024-04-24T08:22:59.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a51200640 1 -- 172.21.15.146:0/1714456682 learned_addr learned my addr 172.21.15.146:0/1714456682 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4b400640 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78b7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4be00640 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4be00640 1 -- 172.21.15.146:0/1714456682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 msgr2=0x55f2a78aa650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4be00640 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78aa650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4be00640 1 -- 172.21.15.146:0/1714456682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 msgr2=0x55f2a78b7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4be00640 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78b7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4be00640 1 -- 172.21.15.146:0/1714456682 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a75ea7e0 con 0x55f2a78aab90 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a51200640 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4b400640 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78b7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:59.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.849+0000 7f3a4be00640 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b4d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009fd0 tx=0x7f3a3c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e754c33d3e6c660e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.850+0000 7f3a41600640 1 -- 172.21.15.146:0/1714456682 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c004310 con 0x55f2a78aab90 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.850+0000 7f3a41600640 1 -- 172.21.15.146:0/1714456682 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c0044b0 con 0x55f2a78aab90 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.850+0000 7f3a41600640 1 -- 172.21.15.146:0/1714456682 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0035f0 con 0x55f2a78aab90 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.850+0000 7f3a48c00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a300a3d50 0x7f3a300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.850+0000 7f3a48c00640 1 -- 172.21.15.146:0/1164484194 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3a380d0a70 con 0x7f3a4c103980 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.850+0000 7f3a51200640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a300a3d50 0x7f3a300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 -- 172.21.15.146:0/1714456682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 msgr2=0x55f2a78b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b4d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009fd0 tx=0x7f3a3c0118d0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 -- 172.21.15.146:0/1714456682 shutdown_connections 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78b7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b4d20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 --2- 172.21.15.146:0/1714456682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 -- 172.21.15.146:0/1714456682 >> 172.21.15.146:0/1714456682 conn(0x55f2a78a6590 msgr2=0x55f2a78a6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 -- 172.21.15.146:0/1714456682 shutdown_connections 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 -- 172.21.15.146:0/1714456682 wait complete. 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a51200640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a300a3d50 0x7f3a300a6210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009430 tx=0x7f3a44005d80 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.851+0000 7f3a5600f740 1 -- start start 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 1 Processor -- start 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aa270 0x55f2a78b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f2a78b8050 con 0x55f2a78aa270 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f2a77c9660 con 0x55f2a78aab90 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f2a78b81d0 con 0x55f2a78b5260 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a5600f740 10 client.0 _ll_register_callbacks cb 0x55f2a77b7aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aa270 0x55f2a78b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.852+0000 7f3a51200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aa270 0x55f2a78b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33046/0 (socket says 172.21.15.146:33046) 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a4be00640 1 -- 172.21.15.146:0/2749593141 learned_addr learned my addr 172.21.15.146:0/2749593141 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a4b400640 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a51200640 1 -- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 msgr2=0x55f2a78b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a51200640 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a51200640 1 -- 172.21.15.146:0/2749593141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aa270 msgr2=0x55f2a78b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a51200640 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aa270 0x55f2a78b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a51200640 1 -- 172.21.15.146:0/2749593141 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a764af70 con 0x55f2a78b5260 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a4be00640 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aa270 0x55f2a78b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a51200640 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b2c00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009ca0 tx=0x7f3a4402a950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=716263731f5654ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a36000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.853+0000 7f3a36000640 1 -- 172.21.15.146:0/2749593141 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44034040 con 0x55f2a78b5260 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c0044b0 con 0x55f2a78b5260 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a36000640 1 -- 172.21.15.146:0/2749593141 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3a44030440 con 0x55f2a78b5260 2024-04-24T08:22:59.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a36000640 1 -- 172.21.15.146:0/2749593141 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3a44030640 con 0x55f2a78b5260 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 10 client.25756 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f2a74e2170 con 0x55f2a78b5260 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a36000640 1 -- 172.21.15.146:0/2749593141 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c0044b0 con 0x55f2a78b5260 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a36000640 1 -- 172.21.15.146:0/2749593141 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3a4405bb30 con 0x55f2a78b5260 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 10 client.25756 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 10 client.25756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 20 client.25756 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a78c5500 0x55f2a78c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 4 client.25756 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.854+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2a781d890 con 0x55f2a78c5500 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.855+0000 7f3a4be00640 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a78c5500 0x55f2a78c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.855+0000 7f3a4be00640 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a78c5500 0x55f2a78c78f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e46a6e967d61cb4 server_cookie=9901934308d242db in_seq=0 out_seq=0 2024-04-24T08:22:59.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.855+0000 7f3a36000640 10 client.25756 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:59.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.856+0000 7f3a36000640 1 -- 172.21.15.146:0/2749593141 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2a781d890 con 0x55f2a78c5500 2024-04-24T08:22:59.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.856+0000 7f3a36000640 10 client.25756 handle_command_reply: tid=0 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.856+0000 7f3a5600f740 1 client.25756 shutdown 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 msgr2=0x55f2a78b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b2c00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009ca0 tx=0x7f3a4402a950 comp rx=0 tx=0).stop 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 shutdown_connections 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a78c5500 0x55f2a78c78f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b2c00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 --2- 172.21.15.146:0/2749593141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aa270 0x55f2a78b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a36000640 0 client.25756 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 >> 172.21.15.146:0/2749593141 conn(0x55f2a78a6590 msgr2=0x55f2a78ac390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 shutdown_connections 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 1 -- 172.21.15.146:0/2749593141 wait complete. 2024-04-24T08:22:59.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.857+0000 7f3a5600f740 20 client.25756 trim_cache size 0 max 16384 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 Processor -- start 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 -- start start 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78c0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1b2640 con 0x55f2a78b5260 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1aff10 con 0x55f2a78aab90 2024-04-24T08:22:59.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.876+0000 7f3a5600f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1add60 con 0x55f2a78aa270 2024-04-24T08:22:59.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a51200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a51200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78ac9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33060/0 (socket says 172.21.15.146:33060) 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a51200640 1 -- 172.21.15.146:0/3601172228 learned_addr learned my addr 172.21.15.146:0/3601172228 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78b10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48678/0 (socket says 172.21.15.146:48678) 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4b400640 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78c0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 -- 172.21.15.146:0/3601172228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 msgr2=0x55f2a78c0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78c0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 -- 172.21.15.146:0/3601172228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 msgr2=0x55f2a78ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78ac9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 -- 172.21.15.146:0/3601172228 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2a78c7c60 con 0x55f2a78aa270 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4b400640 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78c0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a51200640 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.877+0000 7f3a4be00640 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78b10c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c004bb0 tx=0x7f3a3c004c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1972be6212978a15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a40c00640 1 -- 172.21.15.146:0/3601172228 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c02ebd0 con 0x55f2a78aa270 2024-04-24T08:22:59.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a40c00640 1 -- 172.21.15.146:0/3601172228 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c02e890 con 0x55f2a78aa270 2024-04-24T08:22:59.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a40c00640 1 -- 172.21.15.146:0/3601172228 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c003ac0 con 0x55f2a78aa270 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a5600f740 1 -- 172.21.15.146:0/3601172228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 msgr2=0x55f2a78b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78b10c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c004bb0 tx=0x7f3a3c004c30 comp rx=0 tx=0).stop 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a5600f740 1 -- 172.21.15.146:0/3601172228 shutdown_connections 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78b5260 0x55f2a78ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aab90 0x55f2a78c0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3601172228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78aa270 0x55f2a78b10c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.878+0000 7f3a5600f740 1 -- 172.21.15.146:0/3601172228 >> 172.21.15.146:0/3601172228 conn(0x55f2a78a6590 msgr2=0x55f2a78b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:59.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 -- 172.21.15.146:0/3601172228 shutdown_connections 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 -- 172.21.15.146:0/3601172228 wait complete. 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 -- start start 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 Processor -- start 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aa270 0x55f2a78b9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aab90 0x55f2a78b9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.879+0000 7f3a5600f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1ae3d0 con 0x55f2a78aab90 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a5600f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1af840 con 0x55f2a78aa270 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a5600f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1aeaa0 con 0x55f2a78b5260 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a5600f740 10 client.0 _ll_register_callbacks cb 0x7f3a3c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aab90 0x55f2a78b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aa270 0x55f2a78b9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aab90 0x55f2a78b9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33072/0 (socket says 172.21.15.146:33072) 2024-04-24T08:22:59.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4b400640 1 -- 172.21.15.146:0/3377783646 learned_addr learned my addr 172.21.15.146:0/3377783646 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a51200640 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4be00640 1 -- 172.21.15.146:0/3377783646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 msgr2=0x55f2a78b9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4be00640 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4be00640 1 -- 172.21.15.146:0/3377783646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aab90 msgr2=0x55f2a78b9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4be00640 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aab90 0x55f2a78b9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4be00640 1 -- 172.21.15.146:0/3377783646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2a792ace0 con 0x55f2a78aa270 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4b400640 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aab90 0x55f2a78b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a4be00640 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aa270 0x55f2a78b9100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55f2a78b3710 tx=0x7f3a3c009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99a61474e67c31f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.880+0000 7f3a36000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.881+0000 7f3a36000640 1 -- 172.21.15.146:0/3377783646 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c03e060 con 0x55f2a78aa270 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.881+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c02e890 con 0x55f2a78aa270 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.881+0000 7f3a36000640 1 -- 172.21.15.146:0/3377783646 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3a3c02ebd0 con 0x55f2a78aa270 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.881+0000 7f3a36000640 1 -- 172.21.15.146:0/3377783646 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3a3c02ee30 con 0x55f2a78aa270 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.881+0000 7f3a5600f740 10 client.25769 fetch_fsmap learned FSMap version 24 2024-04-24T08:22:59.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.881+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f2a792c020 con 0x55f2a78aa270 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.881+0000 7f3a36000640 1 -- 172.21.15.146:0/3377783646 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c02e890 con 0x55f2a78aa270 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a36000640 1 -- 172.21.15.146:0/3377783646 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3a3c004ca0 con 0x55f2a78aa270 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a5600f740 10 client.25769 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a5600f740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a5600f740 20 client.25769 populate_metadata read hostname 'smithi146' 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a792c5b0 0x55f2a792e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a5600f740 4 client.25769 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f2a7817700 con 0x55f2a792c5b0 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.882+0000 7f3a4b400640 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a792c5b0 0x55f2a792e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.883+0000 7f3a4b400640 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a792c5b0 0x55f2a792e9a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=496babef521fe92 server_cookie=4f16f37e9b3d75ca in_seq=0 out_seq=0 2024-04-24T08:22:59.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.883+0000 7f3a36000640 10 client.25769 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:59.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a36000640 1 -- 172.21.15.146:0/3377783646 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f2a7817700 con 0x55f2a792c5b0 2024-04-24T08:22:59.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a36000640 10 client.25769 handle_command_reply: tid=0 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 client.25769 shutdown 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aa270 msgr2=0x55f2a78b9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aa270 0x55f2a78b9100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55f2a78b3710 tx=0x7f3a3c009b70 comp rx=0 tx=0).stop 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 shutdown_connections 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2a792c5b0 0x55f2a792e9a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2a78b5260 0x55f2a78b9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2a78aab90 0x55f2a78b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 --2- 172.21.15.146:0/3377783646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f2a78aa270 0x55f2a78b9100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a36000640 0 client.25769 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 >> 172.21.15.146:0/3377783646 conn(0x55f2a78a6590 msgr2=0x55f2a78b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 shutdown_connections 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 1 -- 172.21.15.146:0/3377783646 wait complete. 2024-04-24T08:22:59.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.884+0000 7f3a5600f740 20 client.25769 trim_cache size 0 max 16384 2024-04-24T08:22:59.889 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:22:59.889 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:22:59.889 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "21f05781-18bb-48ce-b934-33bb97cf0cfa", 2024-04-24T08:22:59.889 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:22:59.889 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a300a3d50 msgr2=0x7f3a300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a300a3d50 0x7f3a300a6210 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009430 tx=0x7f3a44005d80 comp rx=0 tx=0).stop 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c103980 msgr2=0x7f3a4c1011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c103980 0x7f3a4c1011f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800e9f0 tx=0x7f3a3800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 shutdown_connections 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a300a3d50 0x7f3a300a6210 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a4c10c480 0x7f3a4c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a4c103980 0x7f3a4c1011f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 --2- 172.21.15.146:0/1164484194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a4c103060 0x7f3a4c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 >> 172.21.15.146:0/1164484194 conn(0x7f3a4c0fc880 msgr2=0x7f3a4c0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 shutdown_connections 2024-04-24T08:22:59.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:22:59.885+0000 7f3a51c00640 1 -- 172.21.15.146:0/1164484194 wait complete. 2024-04-24T08:22:59.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:22:59.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:23:00.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 -- 172.21.15.146:0/1188382369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 msgr2=0x7fa4401011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 --2- 172.21.15.146:0/1188382369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401011c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa434009900 tx=0x7fa43402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 -- 172.21.15.146:0/1188382369 shutdown_connections 2024-04-24T08:23:00.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 --2- 172.21.15.146:0/1188382369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa44010e990 0x7fa440110d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 --2- 172.21.15.146:0/1188382369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa440101700 0x7fa44010e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 --2- 172.21.15.146:0/1188382369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401011c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 -- 172.21.15.146:0/1188382369 >> 172.21.15.146:0/1188382369 conn(0x7fa440077280 msgr2=0x7fa440077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:00.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.096+0000 7fa448400640 1 -- 172.21.15.146:0/1188382369 shutdown_connections 2024-04-24T08:23:00.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.097+0000 7fa448400640 1 -- 172.21.15.146:0/1188382369 wait complete. 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.097+0000 7fa448400640 1 Processor -- start 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa448400640 1 -- start start 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa448400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa448400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa440101700 0x7fa4401a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa448400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa44010e990 0x7fa44019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa448400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa440114650 con 0x7fa44010e990 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa448400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401144d0 con 0x7fa440100de0 2024-04-24T08:23:00.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa448400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401147d0 con 0x7fa440101700 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa445a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa446400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa44010e990 0x7fa44019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa445a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57358/0 (socket says 172.21.15.146:57358) 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa445a00640 1 -- 172.21.15.146:0/1137961497 learned_addr learned my addr 172.21.15.146:0/1137961497 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.098+0000 7fa445000640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa440101700 0x7fa4401a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa446400640 1 -- 172.21.15.146:0/1137961497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa440101700 msgr2=0x7fa4401a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa446400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa440101700 0x7fa4401a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa446400640 1 -- 172.21.15.146:0/1137961497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 msgr2=0x7fa4401a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa446400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa446400640 1 -- 172.21.15.146:0/1137961497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44019b090 con 0x7fa44010e990 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa445a00640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa446400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa44010e990 0x7fa44019a910 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa440102fc0 tx=0x7fa43800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa43e000640 1 -- 172.21.15.146:0/1137961497 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43800ccc0 con 0x7fa44010e990 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44019b380 con 0x7fa44010e990 2024-04-24T08:23:00.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.099+0000 7fa43e000640 1 -- 172.21.15.146:0/1137961497 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa438005020 con 0x7fa44010e990 2024-04-24T08:23:00.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.100+0000 7fa43e000640 1 -- 172.21.15.146:0/1137961497 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438004a00 con 0x7fa44010e990 2024-04-24T08:23:00.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.100+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4401a7620 con 0x7fa44010e990 2024-04-24T08:23:00.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.103+0000 7fa44b222740 1 Processor -- start 2024-04-24T08:23:00.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.103+0000 7fa43e000640 1 -- 172.21.15.146:0/1137961497 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fa4380051d0 con 0x7fa44010e990 2024-04-24T08:23:00.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.103+0000 7fa44b222740 1 -- start start 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa44b222740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa44b222740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa44b222740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa44b222740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bdb361f1f0 con 0x55bdb36ba180 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa44b222740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bdb35c5350 con 0x55bdb36c4850 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa44b222740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bdb361b1b0 con 0x55bdb36b9860 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa445a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa445000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa445a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33102/0 (socket says 172.21.15.146:33102) 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa445a00640 1 -- 172.21.15.146:0/2953252874 learned_addr learned my addr 172.21.15.146:0/2953252874 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa445000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36b9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48724/0 (socket says 172.21.15.146:48724) 2024-04-24T08:23:00.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.104+0000 7fa445000640 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445a00640 1 -- 172.21.15.146:0/2953252874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 msgr2=0x55bdb36b9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445a00640 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36b9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445a00640 1 -- 172.21.15.146:0/2953252874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 msgr2=0x55bdb36c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445a00640 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445a00640 1 -- 172.21.15.146:0/2953252874 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdb3619f50 con 0x55bdb36ba180 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445000640 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa43e000640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4280a3da0 0x7fa4280a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445a00640 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c4310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa434006010 tx=0x7fa434005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c215ea62d1497f8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa43e000640 1 -- 172.21.15.146:0/1137961497 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa4380d0580 con 0x7fa44010e990 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa445000640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4280a3da0 0x7fa4280a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa42ea00640 1 -- 172.21.15.146:0/2953252874 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434040070 con 0x55bdb36ba180 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa42ea00640 1 -- 172.21.15.146:0/2953252874 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4340047a0 con 0x55bdb36ba180 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.105+0000 7fa42ea00640 1 -- 172.21.15.146:0/2953252874 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434035040 con 0x55bdb36ba180 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 -- 172.21.15.146:0/2953252874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 msgr2=0x55bdb36c4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa445000640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4280a3da0 0x7fa4280a6260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55bdb36b6d10 tx=0x7fa430009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c4310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa434006010 tx=0x7fa434005dd0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 -- 172.21.15.146:0/2953252874 shutdown_connections 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c4310 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 --2- 172.21.15.146:0/2953252874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36b9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 -- 172.21.15.146:0/2953252874 >> 172.21.15.146:0/2953252874 conn(0x55bdb36b5b80 msgr2=0x55bdb36b5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 -- 172.21.15.146:0/2953252874 shutdown_connections 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.106+0000 7fa44b222740 1 -- 172.21.15.146:0/2953252874 wait complete. 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.107+0000 7fa44b222740 1 -- start start 2024-04-24T08:23:00.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.107+0000 7fa44b222740 1 Processor -- start 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa44b222740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36c1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa44b222740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa44b222740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa44b222740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bdb36c75e0 con 0x55bdb36ba180 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa44b222740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bdb35d8690 con 0x55bdb36c4850 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa44b222740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bdb36c7760 con 0x55bdb36b9860 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa44b222740 10 client.0 _ll_register_callbacks cb 0x55bdb3614650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33106/0 (socket says 172.21.15.146:33106) 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 -- 172.21.15.146:0/3580547218 learned_addr learned my addr 172.21.15.146:0/3580547218 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445a00640 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa446400640 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 -- 172.21.15.146:0/3580547218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 msgr2=0x55bdb36c1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36c1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 -- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 msgr2=0x55bdb36c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.108+0000 7fa445000640 1 -- 172.21.15.146:0/3580547218 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdb35d7920 con 0x55bdb36ba180 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa445000640 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c1d50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa430011f90 tx=0x7fa430004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12aa0351e4d84080 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa423400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa423400640 1 -- 172.21.15.146:0/3580547218 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43002f060 con 0x55bdb36ba180 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa423400640 1 -- 172.21.15.146:0/3580547218 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa430022d40 con 0x55bdb36ba180 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4340047a0 con 0x55bdb36ba180 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa423400640 1 -- 172.21.15.146:0/3580547218 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa430020420 con 0x55bdb36ba180 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa44b222740 10 client.15645 fetch_fsmap learned FSMap version 24 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.109+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bdb33fb7d0 con 0x55bdb36ba180 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa423400640 1 -- 172.21.15.146:0/3580547218 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4340047a0 con 0x55bdb36ba180 2024-04-24T08:23:00.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa423400640 1 -- 172.21.15.146:0/3580547218 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa430029030 con 0x55bdb36ba180 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa44b222740 10 client.15645 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa44b222740 10 client.15645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa44b222740 20 client.15645 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa44b222740 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb36d4c40 0x55bdb36d7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa44b222740 4 client.15645 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdb362cc10 con 0x55bdb36d4c40 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.110+0000 7fa445a00640 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb36d4c40 0x55bdb36d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.111+0000 7fa445a00640 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb36d4c40 0x55bdb36d7080 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b47831a3ac27ca11 server_cookie=32acf22925e9b221 in_seq=0 out_seq=0 2024-04-24T08:23:00.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.111+0000 7fa423400640 10 client.15645 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:00.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa423400640 1 -- 172.21.15.146:0/3580547218 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bdb362cc10 con 0x55bdb36d4c40 2024-04-24T08:23:00.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa423400640 10 client.15645 handle_command_reply: tid=0 2024-04-24T08:23:00.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 client.15645 shutdown 2024-04-24T08:23:00.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 msgr2=0x55bdb36c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c1d50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa430011f90 tx=0x7fa430004a40 comp rx=0 tx=0).stop 2024-04-24T08:23:00.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 shutdown_connections 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb36d4c40 0x55bdb36d7080 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36c4850 0x55bdb36c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36ba180 0x55bdb36c1d50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 --2- 172.21.15.146:0/3580547218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36c1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa423400640 0 client.15645 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 >> 172.21.15.146:0/3580547218 conn(0x55bdb36b5b80 msgr2=0x55bdb36bb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 shutdown_connections 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 1 -- 172.21.15.146:0/3580547218 wait complete. 2024-04-24T08:23:00.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.112+0000 7fa44b222740 20 client.15645 trim_cache size 0 max 16384 2024-04-24T08:23:00.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.131+0000 7fa44b222740 1 Processor -- start 2024-04-24T08:23:00.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.131+0000 7fa44b222740 1 -- start start 2024-04-24T08:23:00.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.131+0000 7fa44b222740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36b9860 0x55bdb36bc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.131+0000 7fa44b222740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36ba180 0x55bdb36c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa44b222740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36dc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa44b222740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401b2890 con 0x55bdb36c4850 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa44b222740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401b0160 con 0x55bdb36b9860 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa44b222740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401adfb0 con 0x55bdb36ba180 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36b9860 0x55bdb36bc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36b9860 0x55bdb36bc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57378/0 (socket says 172.21.15.146:57378) 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 -- 172.21.15.146:0/4687154 learned_addr learned my addr 172.21.15.146:0/4687154 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa446400640 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36dc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445000640 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36ba180 0x55bdb36c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 -- 172.21.15.146:0/4687154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36ba180 msgr2=0x55bdb36c10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36ba180 0x55bdb36c10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 -- 172.21.15.146:0/4687154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 msgr2=0x55bdb36dc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36dc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445a00640 1 -- 172.21.15.146:0/4687154 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdb36b9fb0 con 0x55bdb36b9860 2024-04-24T08:23:00.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.132+0000 7fa445000640 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36ba180 0x55bdb36c10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:23:00.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.133+0000 7fa445a00640 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36b9860 0x55bdb36bc0d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa43402a9e0 tx=0x7fa4340045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fba062ca4eee0247 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:00.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.133+0000 7fa42e000640 1 -- 172.21.15.146:0/4687154 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434035040 con 0x55bdb36b9860 2024-04-24T08:23:00.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.133+0000 7fa42e000640 1 -- 172.21.15.146:0/4687154 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa434057070 con 0x55bdb36b9860 2024-04-24T08:23:00.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.133+0000 7fa42e000640 1 -- 172.21.15.146:0/4687154 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434040040 con 0x55bdb36b9860 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.133+0000 7fa44b222740 1 -- 172.21.15.146:0/4687154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36b9860 msgr2=0x55bdb36bc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.133+0000 7fa44b222740 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36b9860 0x55bdb36bc0d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa43402a9e0 tx=0x7fa4340045b0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 -- 172.21.15.146:0/4687154 shutdown_connections 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36ba180 0x55bdb36c10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36dc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 --2- 172.21.15.146:0/4687154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36b9860 0x55bdb36bc0d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 -- 172.21.15.146:0/4687154 >> 172.21.15.146:0/4687154 conn(0x55bdb36b5b80 msgr2=0x55bdb36b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 -- 172.21.15.146:0/4687154 shutdown_connections 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 -- 172.21.15.146:0/4687154 wait complete. 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 -- start start 2024-04-24T08:23:00.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.134+0000 7fa44b222740 1 Processor -- start 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa44b222740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa44b222740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36ba180 0x55bdb36c8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa44b222740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa44b222740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401ae620 con 0x55bdb36c4850 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa44b222740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401afa90 con 0x55bdb36ba180 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa44b222740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4401aecf0 con 0x55bdb36b9860 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa44b222740 10 client.0 _ll_register_callbacks cb 0x7fa430009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa446400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36ba180 0x55bdb36c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36ba180 0x55bdb36c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57384/0 (socket says 172.21.15.146:57384) 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 -- 172.21.15.146:0/1351119572 learned_addr learned my addr 172.21.15.146:0/1351119572 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445a00640 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 -- 172.21.15.146:0/1351119572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 msgr2=0x55bdb36ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 -- 172.21.15.146:0/1351119572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 msgr2=0x55bdb36c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.135+0000 7fa445000640 1 -- 172.21.15.146:0/1351119572 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdb33313f0 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa445000640 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36ba180 0x55bdb36c8150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa430032830 tx=0x7fa4300325d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ac0a0934df884c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa423400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa423400640 1 -- 172.21.15.146:0/1351119572 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430022800 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa423400640 1 -- 172.21.15.146:0/1351119572 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa4300116a0 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa434057070 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa423400640 1 -- 172.21.15.146:0/1351119572 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa430004ab0 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa44b222740 10 client.25787 fetch_fsmap learned FSMap version 24 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bdb32f10f0 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.136+0000 7fa423400640 1 -- 172.21.15.146:0/1351119572 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa434057070 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa423400640 1 -- 172.21.15.146:0/1351119572 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa430029050 con 0x55bdb36ba180 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa44b222740 10 client.25787 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa44b222740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:00.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa44b222740 20 client.25787 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa44b222740 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb373b530 0x55bdb373d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa44b222740 4 client.25787 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bdb3626a80 con 0x55bdb373b530 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.137+0000 7fa445a00640 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb373b530 0x55bdb373d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa445a00640 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb373b530 0x55bdb373d950 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=993a825d3ba1f03f server_cookie=e973ee11ce598728 in_seq=0 out_seq=0 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa423400640 10 client.25787 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa423400640 1 -- 172.21.15.146:0/1351119572 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bdb3626a80 con 0x55bdb373b530 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa423400640 10 client.25787 handle_command_reply: tid=0 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 client.25787 shutdown 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36ba180 msgr2=0x55bdb36c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36ba180 0x55bdb36c8150 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa430032830 tx=0x7fa4300325d0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 shutdown_connections 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bdb373b530 0x55bdb373d950 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bdb36c4850 0x55bdb36c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa423400640 0 client.25787 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bdb36ba180 0x55bdb36c8150 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 --2- 172.21.15.146:0/1351119572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bdb36b9860 0x55bdb36ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 >> 172.21.15.146:0/1351119572 conn(0x55bdb36b5b80 msgr2=0x55bdb36baff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 shutdown_connections 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 1 -- 172.21.15.146:0/1351119572 wait complete. 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.138+0000 7fa44b222740 20 client.25787 trim_cache size 0 max 16384 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:23:00.143 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:23:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "21f05781-18bb-48ce-b934-33bb97cf0cfa": { 2024-04-24T08:23:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:23:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "21f05781-18bb-48ce-b934-33bb97cf0cfa", 2024-04-24T08:23:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:23:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:23:00.144 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:23:00.144 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4280a3da0 msgr2=0x7fa4280a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4280a3da0 0x7fa4280a6260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55bdb36b6d10 tx=0x7fa430009500 comp rx=0 tx=0).stop 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa44010e990 msgr2=0x7fa44019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa44010e990 0x7fa44019a910 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa440102fc0 tx=0x7fa43800edd0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 shutdown_connections 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4280a3da0 0x7fa4280a6260 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa440101700 0x7fa4401a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa44010e990 0x7fa44019a910 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 --2- 172.21.15.146:0/1137961497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa440100de0 0x7fa4401a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 >> 172.21.15.146:0/1137961497 conn(0x7fa440077280 msgr2=0x7fa44010fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:00.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 shutdown_connections 2024-04-24T08:23:00.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:00.139+0000 7fa448400640 1 -- 172.21.15.146:0/1137961497 wait complete. 2024-04-24T08:23:00.156 INFO:tasks.cephfs.filesystem:scrub status for tag:21f05781-18bb-48ce-b934-33bb97cf0cfa - {'path': '/', 'tag': '21f05781-18bb-48ce-b934-33bb97cf0cfa', 'options': 'recursive,force'} 2024-04-24T08:23:00.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:00 smithi177 ceph-mon[33046]: pgmap v57: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:23:00.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:00 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:23:00.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:00 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:00.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:00 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:23:00.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:00 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:00.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:00 smithi153 ceph-mon[32746]: pgmap v57: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:23:00.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:00 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:23:00.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:00 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:00.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:00 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:23:00.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:00 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:00.719 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T08:23:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:00 smithi146 ceph-mon[27805]: pgmap v57: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:23:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:00 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:23:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:00 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:00 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:23:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:00 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:00.834 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T08:23:00.912 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T08:23:00.918 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T08:23:00.918 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T08:23:00.993 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T08:23:01.004 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:23:01.004 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-24T08:23:01.056 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-04-24T08:23:01.057 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-04-24T08:23:01.057 DEBUG:teuthology.orchestra.run.smithi146:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-04-24T08:23:01.125 INFO:tasks.workunit.client.0.smithi146.stderr:+ set -e 2024-04-24T08:23:01.125 INFO:tasks.workunit.client.0.smithi146.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-04-24T08:23:01.131 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into 'xfstests-dev'... 2024-04-24T08:23:02.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:02 smithi177 ceph-mon[33046]: pgmap v58: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:23:02.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:02 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:23:02.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:02 smithi153 ceph-mon[32746]: pgmap v58: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:23:02.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:02 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:23:02.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:02 smithi146 ceph-mon[27805]: pgmap v58: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:23:02.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:02 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:23:04.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:04 smithi177 ceph-mon[33046]: pgmap v59: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 KiB/s wr, 23 op/s 2024-04-24T08:23:04.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:04 smithi153 ceph-mon[32746]: pgmap v59: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 KiB/s wr, 23 op/s 2024-04-24T08:23:04.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:04 smithi146 ceph-mon[27805]: pgmap v59: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 KiB/s wr, 23 op/s 2024-04-24T08:23:06.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:06 smithi177 ceph-mon[33046]: pgmap v60: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 20 op/s 2024-04-24T08:23:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:06 smithi153 ceph-mon[32746]: pgmap v60: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 20 op/s 2024-04-24T08:23:06.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:06 smithi146 ceph-mon[27805]: pgmap v60: 97 pgs: 97 active+clean; 1.5 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 20 op/s 2024-04-24T08:23:08.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:08 smithi177 ceph-mon[33046]: pgmap v61: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.1 MiB/s wr, 25 op/s 2024-04-24T08:23:08.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:08 smithi177 ceph-mon[33046]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:23:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:08 smithi153 ceph-mon[32746]: pgmap v61: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.1 MiB/s wr, 25 op/s 2024-04-24T08:23:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:08 smithi153 ceph-mon[32746]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:23:08.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:08 smithi146 ceph-mon[27805]: pgmap v61: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.1 MiB/s wr, 25 op/s 2024-04-24T08:23:08.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:08 smithi146 ceph-mon[27805]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:23:10.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:10 smithi153 ceph-mon[32746]: pgmap v62: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 23 op/s 2024-04-24T08:23:10.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:10 smithi146 ceph-mon[27805]: pgmap v62: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 23 op/s 2024-04-24T08:23:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:10 smithi177 ceph-mon[33046]: pgmap v62: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 23 op/s 2024-04-24T08:23:12.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:12 smithi153 ceph-mon[32746]: pgmap v63: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 23 op/s 2024-04-24T08:23:12.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:12 smithi146 ceph-mon[27805]: pgmap v63: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 23 op/s 2024-04-24T08:23:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:12 smithi177 ceph-mon[33046]: pgmap v63: 97 pgs: 97 active+clean; 14 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 1.1 MiB/s wr, 23 op/s 2024-04-24T08:23:14.617 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:14 smithi177 ceph-mon[33046]: pgmap v64: 97 pgs: 97 active+clean; 15 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.2 MiB/s wr, 29 op/s 2024-04-24T08:23:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:14 smithi153 ceph-mon[32746]: pgmap v64: 97 pgs: 97 active+clean; 15 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.2 MiB/s wr, 29 op/s 2024-04-24T08:23:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:14 smithi146 ceph-mon[27805]: pgmap v64: 97 pgs: 97 active+clean; 15 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.2 MiB/s wr, 29 op/s 2024-04-24T08:23:16.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:16 smithi153 ceph-mon[32746]: pgmap v65: 97 pgs: 97 active+clean; 15 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-04-24T08:23:16.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:16 smithi146 ceph-mon[27805]: pgmap v65: 97 pgs: 97 active+clean; 15 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-04-24T08:23:16.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:16 smithi177 ceph-mon[33046]: pgmap v65: 97 pgs: 97 active+clean; 15 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-04-24T08:23:18.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:18 smithi153 ceph-mon[32746]: pgmap v66: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-04-24T08:23:18.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:18 smithi146 ceph-mon[27805]: pgmap v66: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-04-24T08:23:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:18 smithi177 ceph-mon[33046]: pgmap v66: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-04-24T08:23:20.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:20 smithi153 ceph-mon[32746]: pgmap v67: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 124 KiB/s wr, 12 op/s 2024-04-24T08:23:20.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:20 smithi146 ceph-mon[27805]: pgmap v67: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 124 KiB/s wr, 12 op/s 2024-04-24T08:23:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:20 smithi177 ceph-mon[33046]: pgmap v67: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 124 KiB/s wr, 12 op/s 2024-04-24T08:23:22.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:22 smithi153 ceph-mon[32746]: pgmap v68: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 124 KiB/s wr, 12 op/s 2024-04-24T08:23:22.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:22 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:23:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:22 smithi146 ceph-mon[27805]: pgmap v68: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 124 KiB/s wr, 12 op/s 2024-04-24T08:23:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:22 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:23:22.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:22 smithi177 ceph-mon[33046]: pgmap v68: 97 pgs: 97 active+clean; 16 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 124 KiB/s wr, 12 op/s 2024-04-24T08:23:22.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:22 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:23:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:23 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:23:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:23 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:23:23.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:23 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:23:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:23 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:23:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:23 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:23:23.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:23 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:23:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:23 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:23:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:23 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:23:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:23 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:23:24.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:24 smithi153 ceph-mon[32746]: pgmap v69: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 580 KiB/s wr, 33 op/s 2024-04-24T08:23:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:24 smithi146 ceph-mon[27805]: pgmap v69: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 580 KiB/s wr, 33 op/s 2024-04-24T08:23:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:24 smithi177 ceph-mon[33046]: pgmap v69: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 580 KiB/s wr, 33 op/s 2024-04-24T08:23:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:26 smithi153 ceph-mon[32746]: pgmap v70: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 504 KiB/s wr, 26 op/s 2024-04-24T08:23:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:26 smithi146 ceph-mon[27805]: pgmap v70: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 504 KiB/s wr, 26 op/s 2024-04-24T08:23:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:26 smithi177 ceph-mon[33046]: pgmap v70: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 504 KiB/s wr, 26 op/s 2024-04-24T08:23:28.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:28 smithi153 ceph-mon[32746]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 155 op/s 2024-04-24T08:23:28.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:28 smithi146 ceph-mon[27805]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 155 op/s 2024-04-24T08:23:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:28 smithi177 ceph-mon[33046]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.2 MiB/s wr, 155 op/s 2024-04-24T08:23:30.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:23:30.158 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:23:30.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2924336209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b010d500 msgr2=0x7ff6b010d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2924336209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b010d500 0x7ff6b010d8e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009930 tx=0x7ff6a402a420 comp rx=0 tx=0).stop 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2924336209 shutdown_connections 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2924336209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6b0108f40 0x7ff6b01153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2924336209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108580 0x7ff6b0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2924336209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b010d500 0x7ff6b010d8e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2924336209 >> 172.21.15.146:0/2924336209 conn(0x7ff6b0077e20 msgr2=0x7ff6b0106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2924336209 shutdown_connections 2024-04-24T08:23:30.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.371+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2924336209 wait complete. 2024-04-24T08:23:30.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 Processor -- start 2024-04-24T08:23:30.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 -- start start 2024-04-24T08:23:30.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b0108580 0x7ff6b019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108f40 0x7ff6b019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6b010d500 0x7ff6b01a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0118910 con 0x7ff6b0108580 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0118790 con 0x7ff6b0108f40 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.372+0000 7ff6b6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0118a90 con 0x7ff6b010d500 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b4c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108f40 0x7ff6b019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b0108580 0x7ff6b019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b4c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108f40 0x7ff6b019f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57452/0 (socket says 172.21.15.146:57452) 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b0108580 0x7ff6b019ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:57652/0 (socket says 172.21.15.146:57652) 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b4c00640 1 -- 172.21.15.146:0/2879309918 learned_addr learned my addr 172.21.15.146:0/2879309918 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b6000640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6b010d500 0x7ff6b01a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 -- 172.21.15.146:0/2879309918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6b010d500 msgr2=0x7ff6b01a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6b010d500 0x7ff6b01a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 -- 172.21.15.146:0/2879309918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108f40 msgr2=0x7ff6b019f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108f40 0x7ff6b019f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 -- 172.21.15.146:0/2879309918 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b01a3d10 con 0x7ff6b0108580 2024-04-24T08:23:30.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108f40 0x7ff6b019f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:30.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b6000640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6b010d500 0x7ff6b01a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:30.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.373+0000 7ff6b5600640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b0108580 0x7ff6b019ed70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009230 tx=0x7ff6a4003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.374+0000 7ff6ae000640 1 -- 172.21.15.146:0/2879309918 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a403f070 con 0x7ff6b0108580 2024-04-24T08:23:30.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.374+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b01a3fa0 con 0x7ff6b0108580 2024-04-24T08:23:30.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.374+0000 7ff6ae000640 1 -- 172.21.15.146:0/2879309918 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a4031c90 con 0x7ff6b0108580 2024-04-24T08:23:30.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.374+0000 7ff6ae000640 1 -- 172.21.15.146:0/2879309918 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4035030 con 0x7ff6b0108580 2024-04-24T08:23:30.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.374+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b01ab890 con 0x7ff6b0108580 2024-04-24T08:23:30.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.377+0000 7ff6bafbe740 1 Processor -- start 2024-04-24T08:23:30.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6bafbe740 1 -- start start 2024-04-24T08:23:30.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abacce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6ae000640 1 -- 172.21.15.146:0/2879309918 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7ff6a4042070 con 0x7ff6b0108580 2024-04-24T08:23:30.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abad9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5618aba321f0 con 0x5618abacca30 2024-04-24T08:23:30.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5618ab9d8350 con 0x5618abacd350 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5618aba2e1b0 con 0x5618abad7a20 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6b5600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6b6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abacce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6b5600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57468/0 (socket says 172.21.15.146:57468) 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6b5600640 1 -- 172.21.15.146:0/1202142212 learned_addr learned my addr 172.21.15.146:0/1202142212 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.378+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abad9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6b5600640 1 -- 172.21.15.146:0/1202142212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 msgr2=0x5618abad9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6b5600640 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abad9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6b5600640 1 -- 172.21.15.146:0/1202142212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 msgr2=0x5618abacce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6b5600640 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abacce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6b5600640 1 -- 172.21.15.146:0/1202142212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618aba2cf50 con 0x5618abacd350 2024-04-24T08:23:30.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abad9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6b5600640 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad74e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40ee750 tx=0x7ff6a40f93b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28ae40c4ed8434f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6a2600640 1 -- 172.21.15.146:0/1202142212 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40f9810 con 0x5618abacd350 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6a2600640 1 -- 172.21.15.146:0/1202142212 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a40fe030 con 0x5618abacd350 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.379+0000 7ff6a2600640 1 -- 172.21.15.146:0/1202142212 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a41023d0 con 0x5618abacd350 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6ae000640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff6980a3da0 0x7ff6980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6ae000640 1 -- 172.21.15.146:0/2879309918 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ff6a40f1110 con 0x7ff6b0108580 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 -- 172.21.15.146:0/1202142212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 msgr2=0x5618abad74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad74e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40ee750 tx=0x7ff6a40f93b0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6b6000640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff6980a3da0 0x7ff6980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 -- 172.21.15.146:0/1202142212 shutdown_connections 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abad9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad74e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/1202142212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abacce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 -- 172.21.15.146:0/1202142212 >> 172.21.15.146:0/1202142212 conn(0x5618abac8d50 msgr2=0x5618abac9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 -- 172.21.15.146:0/1202142212 shutdown_connections 2024-04-24T08:23:30.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.380+0000 7ff6bafbe740 1 -- 172.21.15.146:0/1202142212 wait complete. 2024-04-24T08:23:30.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.381+0000 7ff6b6000640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff6980a3da0 0x7ff6980a6260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8005fd0 tx=0x7ff6a8009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.381+0000 7ff6bafbe740 1 -- start start 2024-04-24T08:23:30.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.381+0000 7ff6bafbe740 1 Processor -- start 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abacca30 0x5618abad4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abad7a20 0x5618abad54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5618abada5d0 con 0x5618abad7a20 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5618ab9eb690 con 0x5618abacd350 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5618abada750 con 0x5618abacca30 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6bafbe740 10 client.0 _ll_register_callbacks cb 0x5618aba27650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57474/0 (socket says 172.21.15.146:57474) 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 -- 172.21.15.146:0/189902212 learned_addr learned my addr 172.21.15.146:0/189902212 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b6000640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abad7a20 0x5618abad54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b5600640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abacca30 0x5618abad4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 -- 172.21.15.146:0/189902212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abacca30 msgr2=0x5618abad4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abacca30 0x5618abad4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 -- 172.21.15.146:0/189902212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abad7a20 msgr2=0x5618abad54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abad7a20 0x5618abad54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b4c00640 1 -- 172.21.15.146:0/189902212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618ab9ea920 con 0x5618abacd350 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b6000640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abad7a20 0x5618abad54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.382+0000 7ff6b5600640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abacca30 0x5618abad4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad4f60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff69c012d20 tx=0x7ff69c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6116b14ead76a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff696a00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:30.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff696a00640 1 -- 172.21.15.146:0/189902212 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00fe20 con 0x5618abacd350 2024-04-24T08:23:30.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff696a00640 1 -- 172.21.15.146:0/189902212 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ff69c004030 con 0x5618abacd350 2024-04-24T08:23:30.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6a40fe030 con 0x5618abacd350 2024-04-24T08:23:30.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff696a00640 1 -- 172.21.15.146:0/189902212 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff69c004230 con 0x5618abacd350 2024-04-24T08:23:30.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff6bafbe740 10 client.25799 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:30.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5618ab80e7d0 con 0x5618abacd350 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.383+0000 7ff696a00640 1 -- 172.21.15.146:0/189902212 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a40fe030 con 0x5618abacd350 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff696a00640 1 -- 172.21.15.146:0/189902212 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7ff69c010550 con 0x5618abacd350 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6bafbe740 10 client.25799 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6bafbe740 10 client.25799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6bafbe740 20 client.25799 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abae7b80 0x5618abae9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6bafbe740 4 client.25799 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618aba3fc10 con 0x5618abae7b80 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6b5600640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abae7b80 0x5618abae9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff6b5600640 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abae7b80 0x5618abae9f70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b810076b688ce54 server_cookie=8a13370b3ebae35a in_seq=0 out_seq=0 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.384+0000 7ff696a00640 10 client.25799 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff696a00640 1 -- 172.21.15.146:0/189902212 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618aba3fc10 con 0x5618abae7b80 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff696a00640 10 client.25799 handle_command_reply: tid=0 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 client.25799 shutdown 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 msgr2=0x5618abad4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad4f60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff69c012d20 tx=0x7ff69c00f790 comp rx=0 tx=0).stop 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 shutdown_connections 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abae7b80 0x5618abae9f70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abad7a20 0x5618abad54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff696a00640 0 client.25799 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abad4f60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/189902212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abacca30 0x5618abad4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 >> 172.21.15.146:0/189902212 conn(0x5618abac8d50 msgr2=0x5618abac7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 shutdown_connections 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 1 -- 172.21.15.146:0/189902212 wait complete. 2024-04-24T08:23:30.391 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.386+0000 7ff6bafbe740 20 client.25799 trim_cache size 0 max 16384 2024-04-24T08:23:30.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.405+0000 7ff6bafbe740 1 Processor -- start 2024-04-24T08:23:30.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.405+0000 7ff6bafbe740 1 -- start start 2024-04-24T08:23:30.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abac8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abac3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abae33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01b6960 con 0x5618abacca30 2024-04-24T08:23:30.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01b4230 con 0x5618abacd350 2024-04-24T08:23:30.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01b2080 con 0x5618abad7a20 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b4c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abac3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b4c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abac3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57476/0 (socket says 172.21.15.146:57476) 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b4c00640 1 -- 172.21.15.146:0/2804147415 learned_addr learned my addr 172.21.15.146:0/2804147415 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b5600640 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abac8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b6000640 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abae33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b4c00640 1 -- 172.21.15.146:0/2804147415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 msgr2=0x5618abae33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abae33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.406+0000 7ff6b4c00640 1 -- 172.21.15.146:0/2804147415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 msgr2=0x5618abac8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.407+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abac8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.407+0000 7ff6b4c00640 1 -- 172.21.15.146:0/2804147415 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618abad39d0 con 0x5618abacd350 2024-04-24T08:23:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.407+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abac3e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff69c012cf0 tx=0x7ff69c00fcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e14095a5a942ab87 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.407+0000 7ff6a1c00640 1 -- 172.21.15.146:0/2804147415 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c005440 con 0x5618abacd350 2024-04-24T08:23:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.407+0000 7ff6a1c00640 1 -- 172.21.15.146:0/2804147415 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0055e0 con 0x5618abacd350 2024-04-24T08:23:30.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.407+0000 7ff6a1c00640 1 -- 172.21.15.146:0/2804147415 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c01c940 con 0x5618abacd350 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2804147415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 msgr2=0x5618abac3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abac3e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff69c012cf0 tx=0x7ff69c00fcb0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2804147415 shutdown_connections 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abae33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacd350 0x5618abac3e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2804147415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacca30 0x5618abac8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2804147415 >> 172.21.15.146:0/2804147415 conn(0x5618abac8d50 msgr2=0x5618abacea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2804147415 shutdown_connections 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2804147415 wait complete. 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.408+0000 7ff6bafbe740 1 -- start start 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 1 Processor -- start 2024-04-24T08:23:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacca30 0x5618abadb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacd350 0x5618abadb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abadbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01b26f0 con 0x5618abacd350 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01b3b60 con 0x5618abacca30 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01b2dc0 con 0x5618abad7a20 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6bafbe740 10 client.0 _ll_register_callbacks cb 0x7ff69c009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6b5600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacca30 0x5618abadb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6b5600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacca30 0x5618abadb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57478/0 (socket says 172.21.15.146:57478) 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6b5600640 1 -- 172.21.15.146:0/2121120928 learned_addr learned my addr 172.21.15.146:0/2121120928 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6b6000640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abadbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.409+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacd350 0x5618abadb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6b5600640 1 -- 172.21.15.146:0/2121120928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 msgr2=0x5618abadbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6b5600640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abadbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6b5600640 1 -- 172.21.15.146:0/2121120928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacd350 msgr2=0x5618abadb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6b5600640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacd350 0x5618abadb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6b5600640 1 -- 172.21.15.146:0/2121120928 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618ab7443f0 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6b6000640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abadbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6b5600640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacca30 0x5618abadb4a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4006010 tx=0x7ff6a4031490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a133f1ae2dd7864 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff696a00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff696a00640 1 -- 172.21.15.146:0/2121120928 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4049040 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff696a00640 1 -- 172.21.15.146:0/2121120928 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7ff6a40bb360 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff69c0055e0 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff696a00640 1 -- 172.21.15.146:0/2121120928 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6a40b9300 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6bafbe740 10 client.25805 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.410+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5618ab7040f0 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff696a00640 1 -- 172.21.15.146:0/2121120928 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c0055e0 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff696a00640 1 -- 172.21.15.146:0/2121120928 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7ff6a40b96d0 con 0x5618abacca30 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff6bafbe740 10 client.25805 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:30.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff6bafbe740 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff6bafbe740 20 client.25805 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abb4e760 0x5618abb50b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff6bafbe740 4 client.25805 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:23:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5618aba39a80 con 0x5618abb4e760 2024-04-24T08:23:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.411+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abb4e760 0x5618abb50b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.412+0000 7ff6b4c00640 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abb4e760 0x5618abb50b50 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=613cbc5baff2f0ef server_cookie=66f9747f6aba9d3a in_seq=0 out_seq=0 2024-04-24T08:23:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.412+0000 7ff696a00640 10 client.25805 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff696a00640 1 -- 172.21.15.146:0/2121120928 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5618aba39a80 con 0x5618abb4e760 2024-04-24T08:23:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff696a00640 10 client.25805 handle_command_reply: tid=0 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 client.25805 shutdown 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacca30 msgr2=0x5618abadb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacca30 0x5618abadb4a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4006010 tx=0x7ff6a4031490 comp rx=0 tx=0).stop 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 shutdown_connections 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5618abb4e760 0x5618abb50b50 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618abad7a20 0x5618abadbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5618abacd350 0x5618abadb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 --2- 172.21.15.146:0/2121120928 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5618abacca30 0x5618abadb4a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 >> 172.21.15.146:0/2121120928 conn(0x5618abac8d50 msgr2=0x5618abad4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff696a00640 0 client.25805 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 shutdown_connections 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 1 -- 172.21.15.146:0/2121120928 wait complete. 2024-04-24T08:23:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.415+0000 7ff6bafbe740 20 client.25805 trim_cache size 0 max 16384 2024-04-24T08:23:30.419 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:23:30.419 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:23:30.419 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:23:30.419 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff6980a3da0 msgr2=0x7ff6980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff6980a3da0 0x7ff6980a6260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8005fd0 tx=0x7ff6a8009500 comp rx=0 tx=0).stop 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b0108580 msgr2=0x7ff6b019ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b0108580 0x7ff6b019ed70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009230 tx=0x7ff6a4003b20 comp rx=0 tx=0).stop 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 shutdown_connections 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff6980a3da0 0x7ff6980a6260 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff6b010d500 0x7ff6b01a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6b0108f40 0x7ff6b019f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 --2- 172.21.15.146:0/2879309918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6b0108580 0x7ff6b019ed70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 >> 172.21.15.146:0/2879309918 conn(0x7ff6b0077e20 msgr2=0x7ff6b0078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 shutdown_connections 2024-04-24T08:23:30.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.416+0000 7ff6b6a00640 1 -- 172.21.15.146:0/2879309918 wait complete. 2024-04-24T08:23:30.436 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:23:30.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:23:30.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:30 smithi146 ceph-mon[27805]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-24T08:23:30.555 INFO:tasks.workunit.client.0.smithi146.stderr:Updating files: 7% (321/4290) Updating files: 8% (344/4290) Updating files: 9% (387/4290) Updating files: 10% (429/4290) Updating files: 11% (472/4290) Updating files: 11% (490/4290) Updating files: 12% (515/4290) Updating files: 13% (558/4290) Updating files: 14% (601/4290) Updating files: 15% (644/4290) Updating files: 16% (687/4290) Updating files: 17% (730/4290) Updating files: 17% (756/4290) Updating files: 18% (773/4290) Updating files: 19% (816/4290) Updating files: 20% (858/4290) Updating files: 21% (901/4290) Updating files: 22% (944/4290) Updating files: 23% (987/4290) Updating files: 24% (1030/4290) Updating files: 25% (1073/4290) Updating files: 26% (1116/4290) Updating files: 27% (1159/4290) Updating files: 28% (1202/4290) Updating files: 29% (1245/4290) Updating files: 30% (1287/4290) Updating files: 30% (1288/4290) Updating files: 31% (1330/4290) Updating files: 31% (1368/4290) Updating files: 32% (1373/4290) Updating files: 33% (1416/4290) Updating files: 34% (1459/4290) Updating files: 35% (1502/4290) Updating files: 36% (1545/4290) Updating files: 37% (1588/4290) Updating files: 38% (1631/4290) Updating files: 39% (1674/4290) Updating files: 40% (1716/4290) Updating files: 41% (1759/4290) Updating files: 42% (1802/4290) Updating files: 43% (1845/4290) Updating files: 44% (1888/4290) Updating files: 44% (1911/4290) Updating files: 45% (1931/4290) Updating files: 46% (1974/4290) Updating files: 47% (2017/4290) Updating files: 48% (2060/4290) Updating files: 49% (2103/4290) Updating files: 50% (2145/4290) Updating files: 51% (2188/4290) Updating files: 52% (2231/4290) Updating files: 53% (2274/4290) Updating files: 54% (2317/4290) Updating files: 54% (2351/4290) Updating files: 55% (2360/4290) Updating files: 56% (2403/4290) Updating files: 57% (2446/4290) Updating files: 58% (2489/4290) Updating files: 59% (2532/4290) Updating files: 60% (2574/4290) Updating files: 61% (2617/4290) Updating files: 62% (2660/4290) Updating files: 63% (2703/4290) Updating files: 64% (2746/4290) Updating files: 64% (2784/4290) Updating files: 65% (2789/4290) Updating files: 66% (2832/4290) Updating files: 67% (2875/4290) Updating files: 68% (2918/4290) Updating files: 69% (2961/4290) Updating files: 70% (3003/4290) Updating files: 70% (3022/4290) Updating files: 71% (3046/4290) Updating files: 72% (3089/4290) Updating files: 73% (3132/4290) Updating files: 74% (3175/4290) Updating files: 75% (3218/4290) Updating files: 76% (3261/4290) Updating files: 77% (3304/4290) Updating files: 78% (3347/4290) Updating files: 79% (3390/4290) Updating files: 80% (3432/4290) Updating files: 81% (3475/4290) Updating files: 82% (3518/4290) Updating files: 83% (3561/4290) Updating files: 83% (3578/4290) Updating files: 84% (3604/4290) Updating files: 85% (3647/4290) Updating files: 86% (3690/4290) Updating files: 87% (3733/4290) Updating files: 88% (3776/4290) Updating files: 89% (3819/4290) Updating files: 90% (3861/4290) Updating files: 91% (3904/4290) Updating files: 91% (3944/4290) Updating files: 92% (3947/4290) Updating files: 93% (3990/4290) Updating files: 94% (4033/4290) Updating files: 95% (4076/4290) Updating files: 96% (4119/4290) Updating files: 97% (4162/4290) Updating files: 98% (4205/4290) Updating files: 99% (4248/4290) Updating files: 100% (4290/4290) Updating files: 100% (4290/4290), done. 2024-04-24T08:23:30.575 INFO:tasks.workunit.client.0.smithi146.stderr:+ cd xfstests-dev 2024-04-24T08:23:30.575 INFO:tasks.workunit.client.0.smithi146.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 -- 172.21.15.146:0/1763351880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 msgr2=0x7f0aec0697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 --2- 172.21.15.146:0/1763351880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec0697e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009900 tx=0x7f0adc02a410 comp rx=0 tx=0).stop 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 -- 172.21.15.146:0/1763351880 shutdown_connections 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 --2- 172.21.15.146:0/1763351880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0aec069db0 0x7f0aec106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 --2- 172.21.15.146:0/1763351880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aec1071a0 0x7f0aec10ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 --2- 172.21.15.146:0/1763351880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec0697e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 -- 172.21.15.146:0/1763351880 >> 172.21.15.146:0/1763351880 conn(0x7f0aec076840 msgr2=0x7f0aec076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 -- 172.21.15.146:0/1763351880 shutdown_connections 2024-04-24T08:23:30.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.626+0000 7f0af0a00640 1 -- 172.21.15.146:0/1763351880 wait complete. 2024-04-24T08:23:30.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.627+0000 7f0af0a00640 1 Processor -- start 2024-04-24T08:23:30.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0af0a00640 1 -- start start 2024-04-24T08:23:30.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0af0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0af0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0aec1071a0 0x7f0aec1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0af0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aec19a7f0 0x7f0aec19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0af0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec112270 con 0x7f0aec19a7f0 2024-04-24T08:23:30.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0af0a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec1120f0 con 0x7f0aec069400 2024-04-24T08:23:30.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0af0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec1123f0 con 0x7f0aec1071a0 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.628+0000 7f0aea800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0ae9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0aec1071a0 0x7f0aec1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0aea800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec1a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57498/0 (socket says 172.21.15.146:57498) 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0aea800640 1 -- 172.21.15.146:0/303611915 learned_addr learned my addr 172.21.15.146:0/303611915 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0aeb200640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aec19a7f0 0x7f0aec19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0ae9e00640 1 -- 172.21.15.146:0/303611915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 msgr2=0x7f0aec1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0ae9e00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0ae9e00640 1 -- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aec19a7f0 msgr2=0x7f0aec19aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0ae9e00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aec19a7f0 0x7f0aec19aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0ae9e00640 1 -- 172.21.15.146:0/303611915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aec19b3d0 con 0x7f0aec1071a0 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0aeb200640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aec19a7f0 0x7f0aec19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:30.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0aea800640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.629+0000 7f0ae9e00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0aec1071a0 0x7f0aec1a0be0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0aec077ec0 tx=0x7f0ad800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.630+0000 7f0ae3400640 1 -- 172.21.15.146:0/303611915 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad800cc40 con 0x7f0aec1071a0 2024-04-24T08:23:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.630+0000 7f0ae3400640 1 -- 172.21.15.146:0/303611915 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ad800cde0 con 0x7f0aec1071a0 2024-04-24T08:23:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.630+0000 7f0ae3400640 1 -- 172.21.15.146:0/303611915 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8004a00 con 0x7f0aec1071a0 2024-04-24T08:23:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.630+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aec1a74b0 con 0x7f0aec1071a0 2024-04-24T08:23:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.630+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aec1a7860 con 0x7f0aec1071a0 2024-04-24T08:23:30.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.631+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aec068d40 con 0x7f0aec1071a0 2024-04-24T08:23:30.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.636+0000 7f0ae3400640 1 -- 172.21.15.146:0/303611915 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f0ad8004ba0 con 0x7f0aec1071a0 2024-04-24T08:23:30.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.637+0000 7f0ae3400640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0ad00a3da0 0x7f0ad00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:30.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.637+0000 7f0ae3400640 1 -- 172.21.15.146:0/303611915 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0ad80d0750 con 0x7f0aec1071a0 2024-04-24T08:23:30.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.638+0000 7f0aea800640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0ad00a3da0 0x7f0ad00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:30.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.639+0000 7f0aea800640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0ad00a3da0 0x7f0ad00a6260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0096c0 tx=0x7f0adc0046c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.641+0000 7f0ae3400640 1 -- 172.21.15.146:0/303611915 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ad809cda0 con 0x7f0aec1071a0 2024-04-24T08:23:30.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:30 smithi153 ceph-mon[32746]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-24T08:23:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:30 smithi177 ceph-mon[33046]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-24T08:23:30.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.920+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0aec1005b0 con 0x7f0aec1071a0 2024-04-24T08:23:30.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.921+0000 7f0ae3400640 1 -- 172.21.15.146:0/303611915 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11892 (secure 0 0 0) 0x7f0ad809c6a0 con 0x7f0aec1071a0 2024-04-24T08:23:30.923 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:23:30.923 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:23:07.211437+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24568,24569,24553]},"id":1}]} 2024-04-24T08:23:30.924 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 25 2024-04-24T08:23:30.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0ad00a3da0 msgr2=0x7f0ad00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0ad00a3da0 0x7f0ad00a6260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0adc0096c0 tx=0x7f0adc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0aec1071a0 msgr2=0x7f0aec1a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:30.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0aec1071a0 0x7f0aec1a0be0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0aec077ec0 tx=0x7f0ad800ee50 comp rx=0 tx=0).stop 2024-04-24T08:23:30.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 shutdown_connections 2024-04-24T08:23:30.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0ad00a3da0 0x7f0ad00a6260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0aec1071a0 0x7f0aec1a0be0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0aec19a7f0 0x7f0aec19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 --2- 172.21.15.146:0/303611915 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0aec069400 0x7f0aec1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:30.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 >> 172.21.15.146:0/303611915 conn(0x7f0aec076840 msgr2=0x7f0aec10d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:30.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 shutdown_connections 2024-04-24T08:23:30.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:30.924+0000 7f0af0a00640 1 -- 172.21.15.146:0/303611915 wait complete. 2024-04-24T08:23:30.945 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:23:31.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.179+0000 7f02ee600640 1 -- 172.21.15.146:0/1860216328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e80742d0 msgr2=0x7f02e80746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.179+0000 7f02ee600640 1 --2- 172.21.15.146:0/1860216328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e80742d0 0x7f02e80746b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009900 tx=0x7f02d802a480 comp rx=0 tx=0).stop 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.179+0000 7f02ee600640 1 -- 172.21.15.146:0/1860216328 shutdown_connections 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.179+0000 7f02ee600640 1 --2- 172.21.15.146:0/1860216328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e810e8e0 0x7f02e8110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.179+0000 7f02ee600640 1 --2- 172.21.15.146:0/1860216328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02e8074bf0 0x7f02e810e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.179+0000 7f02ee600640 1 --2- 172.21.15.146:0/1860216328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e80742d0 0x7f02e80746b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.179+0000 7f02ee600640 1 -- 172.21.15.146:0/1860216328 >> 172.21.15.146:0/1860216328 conn(0x7f02e8079450 msgr2=0x7f02e8079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.180+0000 7f02ee600640 1 -- 172.21.15.146:0/1860216328 shutdown_connections 2024-04-24T08:23:31.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.180+0000 7f02ee600640 1 -- 172.21.15.146:0/1860216328 wait complete. 2024-04-24T08:23:31.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 Processor -- start 2024-04-24T08:23:31.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 -- start start 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e80742d0 0x7f02e81a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02e8074bf0 0x7f02e81a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e810e8e0 0x7f02e819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81145d0 con 0x7f02e8074bf0 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02e8114450 con 0x7f02e810e8e0 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02ee600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02e8114750 con 0x7f02e80742d0 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.181+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e810e8e0 0x7f02e819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02e8074bf0 0x7f02e81a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e810e8e0 0x7f02e819a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60006/0 (socket says 172.21.15.146:60006) 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 -- 172.21.15.146:0/2775466063 learned_addr learned my addr 172.21.15.146:0/2775466063 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7400640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e80742d0 0x7f02e81a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 -- 172.21.15.146:0/2775466063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e80742d0 msgr2=0x7f02e81a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e80742d0 0x7f02e81a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 -- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02e8074bf0 msgr2=0x7f02e81a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02e8074bf0 0x7f02e81a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 -- 172.21.15.146:0/2775466063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e819b0b0 con 0x7f02e810e8e0 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02e7e00640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e810e8e0 0x7f02e819a930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02e000eb40 tx=0x7f02e000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02dfe00640 1 -- 172.21.15.146:0/2775466063 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0004b20 con 0x7f02e810e8e0 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02dfe00640 1 -- 172.21.15.146:0/2775466063 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02e0004cc0 con 0x7f02e810e8e0 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e819b3a0 con 0x7f02e810e8e0 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.182+0000 7f02dfe00640 1 -- 172.21.15.146:0/2775466063 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0010850 con 0x7f02e810e8e0 2024-04-24T08:23:31.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.183+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e81a7640 con 0x7f02e810e8e0 2024-04-24T08:23:31.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.186+0000 7f02f13f6740 1 Processor -- start 2024-04-24T08:23:31.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.186+0000 7f02dfe00640 1 -- 172.21.15.146:0/2775466063 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f02e0005020 con 0x7f02e810e8e0 2024-04-24T08:23:31.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.186+0000 7f02f13f6740 1 -- start start 2024-04-24T08:23:31.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.186+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bda8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.186+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdb5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02f13f6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56322bd0e1d0 con 0x56322bda8950 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02f13f6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56322bcb4450 con 0x56322bdb3940 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02f13f6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56322bd0a190 con 0x56322bda9270 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bda8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bda8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36312/0 (socket says 172.21.15.146:36312) 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 -- 172.21.15.146:0/2489366292 learned_addr learned my addr 172.21.15.146:0/2489366292 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdb5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e7400640 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 -- 172.21.15.146:0/2489366292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 msgr2=0x56322bdb3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 -- 172.21.15.146:0/2489366292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 msgr2=0x56322bdb5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdb5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02e6a00640 1 -- 172.21.15.146:0/2489366292 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56322bd71310 con 0x56322bda8950 2024-04-24T08:23:31.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.187+0000 7f02dfe00640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02cc0a3da0 0x7f02cc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.188+0000 7f02e6a00640 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bda8d30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f02d40058c0 tx=0x7f02d4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136ab3d036dc0157 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.188+0000 7f02dfe00640 1 -- 172.21.15.146:0/2775466063 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f02e00d0f70 con 0x7f02e810e8e0 2024-04-24T08:23:31.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.188+0000 7f02e7400640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02cc0a3da0 0x7f02cc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.188+0000 7f02dcc00640 1 -- 172.21.15.146:0/2489366292 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d4013d30 con 0x56322bda8950 2024-04-24T08:23:31.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.188+0000 7f02dcc00640 1 -- 172.21.15.146:0/2489366292 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02d4013130 con 0x56322bda8950 2024-04-24T08:23:31.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.188+0000 7f02dcc00640 1 -- 172.21.15.146:0/2489366292 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d40047a0 con 0x56322bda8950 2024-04-24T08:23:31.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 -- 172.21.15.146:0/2489366292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 msgr2=0x56322bda8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bda8d30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f02d40058c0 tx=0x7f02d4005850 comp rx=0 tx=0).stop 2024-04-24T08:23:31.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02e7400640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02cc0a3da0 0x7f02cc0a6260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f02e819ba70 tx=0x7f02d80058d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 -- 172.21.15.146:0/2489366292 shutdown_connections 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdb5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2489366292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bda8d30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 -- 172.21.15.146:0/2489366292 >> 172.21.15.146:0/2489366292 conn(0x56322bda4c70 msgr2=0x56322bda5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 -- 172.21.15.146:0/2489366292 shutdown_connections 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.189+0000 7f02f13f6740 1 -- 172.21.15.146:0/2489366292 wait complete. 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 -- start start 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 Processor -- start 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdb3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bdb3940 0x56322bdb0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56322bdb6770 con 0x56322bdb3940 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56322bcc7810 con 0x56322bda8950 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56322bdb68f0 con 0x56322bda9270 2024-04-24T08:23:31.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02f13f6740 10 client.0 _ll_register_callbacks cb 0x56322bc95140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.190+0000 7f02e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdb3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdb3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60026/0 (socket says 172.21.15.146:60026) 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 -- 172.21.15.146:0/2278160832 learned_addr learned my addr 172.21.15.146:0/2278160832 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7e00640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bdb3940 0x56322bdb0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e6a00640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 -- 172.21.15.146:0/2278160832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 msgr2=0x56322bdb0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 -- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bdb3940 msgr2=0x56322bdb0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bdb3940 0x56322bdb0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 -- 172.21.15.146:0/2278160832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56322bcd83f0 con 0x56322bda8950 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7e00640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bdb3940 0x56322bdb0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02e7400640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdb3310 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02d8033000 tx=0x7f02d8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=574e9f6d78507e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.191+0000 7f02d1600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.192+0000 7f02d1600640 1 -- 172.21.15.146:0/2278160832 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d804e070 con 0x56322bda8950 2024-04-24T08:23:31.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.192+0000 7f02d1600640 1 -- 172.21.15.146:0/2278160832 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f02d8003b80 con 0x56322bda8950 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.192+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02d4013130 con 0x56322bda8950 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.192+0000 7f02d1600640 1 -- 172.21.15.146:0/2278160832 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f02d805a050 con 0x56322bda8950 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.192+0000 7f02f13f6740 10 client.25823 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.192+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56322ba203f0 con 0x56322bda8950 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.192+0000 7f02d1600640 1 -- 172.21.15.146:0/2278160832 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02d4013130 con 0x56322bda8950 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02d1600640 1 -- 172.21.15.146:0/2278160832 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f02d803d4e0 con 0x56322bda8950 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02f13f6740 10 client.25823 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02f13f6740 10 client.25823 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02f13f6740 20 client.25823 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322bdc3a50 0x56322bdc5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02f13f6740 4 client.25823 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56322bd1bbf0 con 0x56322bdc3a50 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.193+0000 7f02e6a00640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322bdc3a50 0x56322bdc5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.194+0000 7f02e6a00640 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322bdc3a50 0x56322bdc5eb0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9fa2d6c978fda209 server_cookie=b52e27507d7855ce in_seq=0 out_seq=0 2024-04-24T08:23:31.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.194+0000 7f02d1600640 10 client.25823 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:23:31.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02d1600640 1 -- 172.21.15.146:0/2278160832 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56322bd1bbf0 con 0x56322bdc3a50 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02d1600640 10 client.25823 handle_command_reply: tid=0 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 client.25823 shutdown 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 msgr2=0x56322bdb3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdb3310 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f02d8033000 tx=0x7f02d8009260 comp rx=0 tx=0).stop 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 shutdown_connections 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322bdc3a50 0x56322bdc5eb0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdb0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bdb3940 0x56322bdb0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2278160832 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdb3310 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02d1600640 0 client.25823 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 >> 172.21.15.146:0/2278160832 conn(0x56322bda4c70 msgr2=0x56322bdaa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 shutdown_connections 2024-04-24T08:23:31.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.195+0000 7f02f13f6740 1 -- 172.21.15.146:0/2278160832 wait complete. 2024-04-24T08:23:31.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.196+0000 7f02f13f6740 20 client.25823 trim_cache size 0 max 16384 2024-04-24T08:23:31.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.210+0000 7f02f13f6740 1 Processor -- start 2024-04-24T08:23:31.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.210+0000 7f02f13f6740 1 -- start start 2024-04-24T08:23:31.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdbf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda9270 0x56322bdb2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bdb3940 0x56322bdcb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02f13f6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81b2720 con 0x56322bda9270 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02f13f6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81afff0 con 0x56322bda8950 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02f13f6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81ade40 con 0x56322bdb3940 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdbf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda9270 0x56322bdb2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdbf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60032/0 (socket says 172.21.15.146:60032) 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 -- 172.21.15.146:0/2451105904 learned_addr learned my addr 172.21.15.146:0/2451105904 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7e00640 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bdb3940 0x56322bdcb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 -- 172.21.15.146:0/2451105904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bdb3940 msgr2=0x56322bdcb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bdb3940 0x56322bdcb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 -- 172.21.15.146:0/2451105904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda9270 msgr2=0x56322bdb2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda9270 0x56322bdb2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7400640 1 -- 172.21.15.146:0/2451105904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56322baea7d0 con 0x56322bda8950 2024-04-24T08:23:31.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.211+0000 7f02e7e00640 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bdb3940 0x56322bdcb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:31.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.212+0000 7f02e7400640 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdbf660 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02d8004770 tx=0x7f02d802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaec61e0de6d2a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.212+0000 7f02d3e00640 1 -- 172.21.15.146:0/2451105904 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d804e070 con 0x56322bda8950 2024-04-24T08:23:31.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.212+0000 7f02d3e00640 1 -- 172.21.15.146:0/2451105904 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02d8063070 con 0x56322bda8950 2024-04-24T08:23:31.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.212+0000 7f02d3e00640 1 -- 172.21.15.146:0/2451105904 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8035040 con 0x56322bda8950 2024-04-24T08:23:31.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 -- 172.21.15.146:0/2451105904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 msgr2=0x56322bdbf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdbf660 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02d8004770 tx=0x7f02d802a480 comp rx=0 tx=0).stop 2024-04-24T08:23:31.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 -- 172.21.15.146:0/2451105904 shutdown_connections 2024-04-24T08:23:31.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bdb3940 0x56322bdcb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda9270 0x56322bdb2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 --2- 172.21.15.146:0/2451105904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bda8950 0x56322bdbf660 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 -- 172.21.15.146:0/2451105904 >> 172.21.15.146:0/2451105904 conn(0x56322bda4c70 msgr2=0x56322bdaef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 -- 172.21.15.146:0/2451105904 shutdown_connections 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 -- 172.21.15.146:0/2451105904 wait complete. 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.213+0000 7f02f13f6740 1 -- start start 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 1 Processor -- start 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bdbbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81ae4b0 con 0x56322bda8950 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81af920 con 0x56322bdb3940 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81aeb80 con 0x56322bda9270 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02f13f6740 10 client.0 _ll_register_callbacks cb 0x7f02d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.220 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60046/0 (socket says 172.21.15.146:60046) 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.214+0000 7f02e7e00640 1 -- 172.21.15.146:0/1749893252 learned_addr learned my addr 172.21.15.146:0/1749893252 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e7400640 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bdbbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e6a00640 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e7e00640 1 -- 172.21.15.146:0/1749893252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 msgr2=0x56322bdad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e7e00640 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e7e00640 1 -- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 msgr2=0x56322bdbbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e7e00640 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bdbbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e7e00640 1 -- 172.21.15.146:0/1749893252 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56322bdae340 con 0x56322bdb3940 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02e7e00640 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdade00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f02e0099320 tx=0x7f02e009c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa6a1f74b0c9c9d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02d1600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02d1600640 1 -- 172.21.15.146:0/1749893252 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e00ba070 con 0x56322bdb3940 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02d1600640 1 -- 172.21.15.146:0/1749893252 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f02e00a58a0 con 0x56322bdb3940 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02d8063070 con 0x56322bdb3940 2024-04-24T08:23:31.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.215+0000 7f02d1600640 1 -- 172.21.15.146:0/1749893252 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f02e009cec0 con 0x56322bdb3940 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 10 client.25829 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56322bdb4ee0 con 0x56322bdb3940 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02d1600640 1 -- 172.21.15.146:0/1749893252 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02d8063070 con 0x56322bdb3940 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02d1600640 1 -- 172.21.15.146:0/1749893252 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f02e007bce0 con 0x56322bdb3940 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 10 client.25829 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 10 client.25829 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 20 client.25829 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322be2a4a0 0x56322be2c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 4 client.25829 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56322bd15a60 con 0x56322be2a4a0 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.216+0000 7f02e7400640 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322be2a4a0 0x56322be2c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02e7400640 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322be2a4a0 0x56322be2c8c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d0d459860734d96e server_cookie=2cda36c5211b5540 in_seq=0 out_seq=0 2024-04-24T08:23:31.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02d1600640 10 client.25829 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02d1600640 1 -- 172.21.15.146:0/1749893252 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56322bd15a60 con 0x56322be2a4a0 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02d1600640 10 client.25829 handle_command_reply: tid=0 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 client.25829 shutdown 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 msgr2=0x56322bdade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdade00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f02e0099320 tx=0x7f02e009c8e0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 shutdown_connections 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56322be2a4a0 0x56322be2c8c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56322bda9270 0x56322bdad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56322bdb3940 0x56322bdade00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 --2- 172.21.15.146:0/1749893252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56322bda8950 0x56322bdbbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 >> 172.21.15.146:0/1749893252 conn(0x56322bda4c70 msgr2=0x56322bda3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.217+0000 7f02d1600640 0 client.25829 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.218+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 shutdown_connections 2024-04-24T08:23:31.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.218+0000 7f02f13f6740 1 -- 172.21.15.146:0/1749893252 wait complete. 2024-04-24T08:23:31.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.218+0000 7f02f13f6740 20 client.25829 trim_cache size 0 max 16384 2024-04-24T08:23:31.224 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02cc0a3da0 msgr2=0x7f02cc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02cc0a3da0 0x7f02cc0a6260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f02e819ba70 tx=0x7f02d80058d0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e810e8e0 msgr2=0x7f02e819a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e810e8e0 0x7f02e819a930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f02e000eb40 tx=0x7f02e000c2d0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 shutdown_connections 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f02cc0a3da0 0x7f02cc0a6260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02e810e8e0 0x7f02e819a930 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02e8074bf0 0x7f02e81a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 --2- 172.21.15.146:0/2775466063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e80742d0 0x7f02e81a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 >> 172.21.15.146:0/2775466063 conn(0x7f02e8079450 msgr2=0x7f02e80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 shutdown_connections 2024-04-24T08:23:31.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.219+0000 7f02ee600640 1 -- 172.21.15.146:0/2775466063 wait complete. 2024-04-24T08:23:31.238 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:23:31.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 -- 172.21.15.146:0/3700872127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c1044b0 msgr2=0x7f9b4c104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 --2- 172.21.15.146:0/3700872127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c1044b0 0x7f9b4c104890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 -- 172.21.15.146:0/3700872127 shutdown_connections 2024-04-24T08:23:31.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 --2- 172.21.15.146:0/3700872127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b4c109580 0x7f9b4c111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 --2- 172.21.15.146:0/3700872127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4c104e60 0x7f9b4c108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 --2- 172.21.15.146:0/3700872127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c1044b0 0x7f9b4c104890 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 -- 172.21.15.146:0/3700872127 >> 172.21.15.146:0/3700872127 conn(0x7f9b4c0ffd90 msgr2=0x7f9b4c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 -- 172.21.15.146:0/3700872127 shutdown_connections 2024-04-24T08:23:31.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.437+0000 7f9b53000640 1 -- 172.21.15.146:0/3700872127 wait complete. 2024-04-24T08:23:31.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.438+0000 7f9b53000640 1 Processor -- start 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.438+0000 7f9b53000640 1 -- start start 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b53000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b4c1044b0 0x7f9b4c1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b53000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4c109580 0x7f9b4c1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b53000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c19aa60 0x7f9b4c19af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b53000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c114610 con 0x7f9b4c109580 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b53000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c114490 con 0x7f9b4c19aa60 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b53000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c114790 con 0x7f9b4c1044b0 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c19aa60 0x7f9b4c19af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4c109580 0x7f9b4c1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c19aa60 0x7f9b4c19af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60066/0 (socket says 172.21.15.146:60066) 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 -- 172.21.15.146:0/839281698 learned_addr learned my addr 172.21.15.146:0/839281698 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b4be00640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b4c1044b0 0x7f9b4c1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 -- 172.21.15.146:0/839281698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b4c1044b0 msgr2=0x7f9b4c1a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b4c1044b0 0x7f9b4c1a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 -- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4c109580 msgr2=0x7f9b4c1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4c109580 0x7f9b4c1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b51000640 1 -- 172.21.15.146:0/839281698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c19b580 con 0x7f9b4c19aa60 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.439+0000 7f9b4b400640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4c109580 0x7f9b4c1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.440+0000 7f9b51000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c19aa60 0x7f9b4c19af10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400eaa0 tx=0x7f9b4400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.440+0000 7f9b48c00640 1 -- 172.21.15.146:0/839281698 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4400cdb0 con 0x7f9b4c19aa60 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.440+0000 7f9b48c00640 1 -- 172.21.15.146:0/839281698 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b44005020 con 0x7f9b4c19aa60 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.440+0000 7f9b48c00640 1 -- 172.21.15.146:0/839281698 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b440116b0 con 0x7f9b4c19aa60 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.440+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c19b870 con 0x7f9b4c19aa60 2024-04-24T08:23:31.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.440+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c1a7a20 con 0x7f9b4c19aa60 2024-04-24T08:23:31.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 Processor -- start 2024-04-24T08:23:31.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b48c00640 1 -- 172.21.15.146:0/839281698 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f9b44005560 con 0x7f9b4c19aa60 2024-04-24T08:23:31.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 -- start start 2024-04-24T08:23:31.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc332950 0x5636fc332d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc33d940 0x5636fc33fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5636fc2981d0 con 0x5636fc333270 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5636fc23e450 con 0x5636fc33d940 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.444+0000 7f9b55d78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5636fc294190 con 0x5636fc332950 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc33d940 0x5636fc33fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc33d940 0x5636fc33fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60074/0 (socket says 172.21.15.146:60074) 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 -- 172.21.15.146:0/12014511 learned_addr learned my addr 172.21.15.146:0/12014511 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4be00640 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc332950 0x5636fc332d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 -- 172.21.15.146:0/12014511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc332950 msgr2=0x5636fc332d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc332950 0x5636fc332d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 -- 172.21.15.146:0/12014511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 msgr2=0x5636fc33d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 -- 172.21.15.146:0/12014511 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636fc2fb310 con 0x5636fc33d940 2024-04-24T08:23:31.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.445+0000 7f9b4b400640 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc33d940 0x5636fc33fd30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9b38011b60 tx=0x7f9b38005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a27a7568217cdbea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.446+0000 7f9b48c00640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b300a3d20 0x7f9b300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.446+0000 7f9b41600640 1 -- 172.21.15.146:0/12014511 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800fc50 con 0x5636fc33d940 2024-04-24T08:23:31.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.446+0000 7f9b41600640 1 -- 172.21.15.146:0/12014511 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b3800fdf0 con 0x5636fc33d940 2024-04-24T08:23:31.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.446+0000 7f9b41600640 1 -- 172.21.15.146:0/12014511 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380049a0 con 0x5636fc33d940 2024-04-24T08:23:31.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.446+0000 7f9b48c00640 1 -- 172.21.15.146:0/839281698 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9b440d0920 con 0x7f9b4c19aa60 2024-04-24T08:23:31.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.446+0000 7f9b4be00640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b300a3d20 0x7f9b300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 -- 172.21.15.146:0/12014511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc33d940 msgr2=0x5636fc33fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc33d940 0x5636fc33fd30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9b38011b60 tx=0x7f9b38005850 comp rx=0 tx=0).stop 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b4be00640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b300a3d20 0x7f9b300a61e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c1001e0 tx=0x7f9b3c0058d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 -- 172.21.15.146:0/12014511 shutdown_connections 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc33d940 0x5636fc33fd30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 --2- 172.21.15.146:0/12014511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc332950 0x5636fc332d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 -- 172.21.15.146:0/12014511 >> 172.21.15.146:0/12014511 conn(0x5636fc32ec70 msgr2=0x5636fc32f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 -- 172.21.15.146:0/12014511 shutdown_connections 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.447+0000 7f9b55d78740 1 -- 172.21.15.146:0/12014511 wait complete. 2024-04-24T08:23:31.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.448+0000 7f9b55d78740 1 -- start start 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.448+0000 7f9b55d78740 1 Processor -- start 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc332950 0x5636fc33d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc333270 0x5636fc33a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc33ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b55d78740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5636fc3407d0 con 0x5636fc332950 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b55d78740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5636fc251810 con 0x5636fc333270 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b55d78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5636fc340950 con 0x5636fc33d940 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b55d78740 10 client.0 _ll_register_callbacks cb 0x5636fc21f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc333270 0x5636fc33a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc333270 0x5636fc33a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60076/0 (socket says 172.21.15.146:60076) 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 -- 172.21.15.146:0/1406006666 learned_addr learned my addr 172.21.15.146:0/1406006666 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4be00640 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc332950 0x5636fc33d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b51000640 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc33ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 -- 172.21.15.146:0/1406006666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 msgr2=0x5636fc33ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc33ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 -- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc332950 msgr2=0x5636fc33d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc332950 0x5636fc33d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.449+0000 7f9b4b400640 1 -- 172.21.15.146:0/1406006666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636fc2623f0 con 0x5636fc333270 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b4b400640 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc333270 0x5636fc33a650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9b380053c0 tx=0x7f9b38005440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bc531828f56c76b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b36000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b36000640 1 -- 172.21.15.146:0/1406006666 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38004ad0 con 0x5636fc333270 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b36000640 1 -- 172.21.15.146:0/1406006666 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9b38004c70 con 0x5636fc333270 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3800fdf0 con 0x5636fc333270 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b36000640 1 -- 172.21.15.146:0/1406006666 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9b38004e70 con 0x5636fc333270 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b55d78740 10 client.25841 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5636fbfaa3f0 con 0x5636fc333270 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b36000640 1 -- 172.21.15.146:0/1406006666 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b3800fdf0 con 0x5636fc333270 2024-04-24T08:23:31.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.450+0000 7f9b36000640 1 -- 172.21.15.146:0/1406006666 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f9b38026070 con 0x5636fc333270 2024-04-24T08:23:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.451+0000 7f9b55d78740 10 client.25841 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.451+0000 7f9b55d78740 10 client.25841 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:23:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.451+0000 7f9b55d78740 20 client.25841 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.451+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc34d980 0x5636fc34fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.451+0000 7f9b55d78740 4 client.25841 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.451+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636fc2a5bf0 con 0x5636fc34d980 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.451+0000 7f9b4be00640 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc34d980 0x5636fc34fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.452+0000 7f9b4be00640 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc34d980 0x5636fc34fd70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60cc658f130f33df server_cookie=d8071252106f332 in_seq=0 out_seq=0 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.452+0000 7f9b36000640 10 client.25841 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.452+0000 7f9b36000640 1 -- 172.21.15.146:0/1406006666 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636fc2a5bf0 con 0x5636fc34d980 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.452+0000 7f9b36000640 10 client.25841 handle_command_reply: tid=0 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.452+0000 7f9b55d78740 1 client.25841 shutdown 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc333270 msgr2=0x5636fc33a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc333270 0x5636fc33a650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9b380053c0 tx=0x7f9b38005440 comp rx=0 tx=0).stop 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 shutdown_connections 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc34d980 0x5636fc34fd70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc33ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc333270 0x5636fc33a650 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b36000640 0 client.25841 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1406006666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc332950 0x5636fc33d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 >> 172.21.15.146:0/1406006666 conn(0x5636fc32ec70 msgr2=0x5636fc334a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 shutdown_connections 2024-04-24T08:23:31.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 1 -- 172.21.15.146:0/1406006666 wait complete. 2024-04-24T08:23:31.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.453+0000 7f9b55d78740 20 client.25841 trim_cache size 0 max 16384 2024-04-24T08:23:31.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 Processor -- start 2024-04-24T08:23:31.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 -- start start 2024-04-24T08:23:31.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc349690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc355500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1b2840 con 0x5636fc333270 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1b0110 con 0x5636fc332950 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b55d78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1adf60 con 0x5636fc33d940 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b4be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc349690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36372/0 (socket says 172.21.15.146:36372) 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b4b400640 1 -- 172.21.15.146:0/1249958356 learned_addr learned my addr 172.21.15.146:0/1249958356 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.474+0000 7f9b4be00640 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc349690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60078/0 (socket says 172.21.15.146:60078) 2024-04-24T08:23:31.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b51000640 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc355500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b4b400640 1 -- 172.21.15.146:0/1249958356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 msgr2=0x5636fc355500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b4b400640 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc355500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b4b400640 1 -- 172.21.15.146:0/1249958356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 msgr2=0x5636fc349690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b4b400640 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc349690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b4b400640 1 -- 172.21.15.146:0/1249958356 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636fc3330a0 con 0x5636fc333270 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b4b400640 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33ce90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009b70 tx=0x7f9b38005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e32497f1662737 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b40c00640 1 -- 172.21.15.146:0/1249958356 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38004e70 con 0x5636fc333270 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b40c00640 1 -- 172.21.15.146:0/1249958356 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b38011510 con 0x5636fc333270 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.475+0000 7f9b40c00640 1 -- 172.21.15.146:0/1249958356 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380309a0 con 0x5636fc333270 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 -- 172.21.15.146:0/1249958356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 msgr2=0x5636fc33ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33ce90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009b70 tx=0x7f9b38005720 comp rx=0 tx=0).stop 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 -- 172.21.15.146:0/1249958356 shutdown_connections 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc355500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc33ce90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 --2- 172.21.15.146:0/1249958356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc349690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 -- 172.21.15.146:0/1249958356 >> 172.21.15.146:0/1249958356 conn(0x5636fc32ec70 msgr2=0x5636fc338f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 -- 172.21.15.146:0/1249958356 shutdown_connections 2024-04-24T08:23:31.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.476+0000 7f9b55d78740 1 -- 172.21.15.146:0/1249958356 wait complete. 2024-04-24T08:23:31.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.477+0000 7f9b55d78740 1 -- start start 2024-04-24T08:23:31.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.477+0000 7f9b55d78740 1 Processor -- start 2024-04-24T08:23:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc345e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc337960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b55d78740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc337ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b55d78740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1ae5d0 con 0x5636fc333270 2024-04-24T08:23:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b55d78740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1afa40 con 0x5636fc332950 2024-04-24T08:23:31.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b55d78740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1aeca0 con 0x5636fc33d940 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b55d78740 10 client.0 _ll_register_callbacks cb 0x7f9b3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc345e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc345e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60084/0 (socket says 172.21.15.146:60084) 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 -- 172.21.15.146:0/2204428020 learned_addr learned my addr 172.21.15.146:0/2204428020 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4b400640 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc337960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b51000640 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc337ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 -- 172.21.15.146:0/2204428020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 msgr2=0x5636fc337ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc337ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 -- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 msgr2=0x5636fc337960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc337960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.478+0000 7f9b4be00640 1 -- 172.21.15.146:0/2204428020 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636fc0747d0 con 0x5636fc332950 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b4be00640 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc345e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c03f070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee66cfa485fcf9ba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b36000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:31.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b36000640 1 -- 172.21.15.146:0/2204428020 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c035040 con 0x5636fc332950 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b36000640 1 -- 172.21.15.146:0/2204428020 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9b3c0525b0 con 0x5636fc332950 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b38011510 con 0x5636fc332950 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b36000640 1 -- 172.21.15.146:0/2204428020 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c0527b0 con 0x5636fc332950 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b55d78740 10 client.25847 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.479+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5636fc33f130 con 0x5636fc332950 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b36000640 1 -- 172.21.15.146:0/2204428020 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b38011510 con 0x5636fc332950 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b36000640 1 -- 172.21.15.146:0/2204428020 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f9b3c063590 con 0x5636fc332950 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b55d78740 10 client.25847 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b55d78740 10 client.25847 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b55d78740 20 client.25847 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b55d78740 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc3b41a0 0x5636fc3b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b55d78740 4 client.25847 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636fc29fa60 con 0x5636fc3b41a0 2024-04-24T08:23:31.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.480+0000 7f9b4b400640 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc3b41a0 0x5636fc3b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.481+0000 7f9b4b400640 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc3b41a0 0x5636fc3b65c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=79826bd75ce437ad server_cookie=117a3e2d1b275396 in_seq=0 out_seq=0 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.481+0000 7f9b36000640 10 client.25847 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.481+0000 7f9b36000640 1 -- 172.21.15.146:0/2204428020 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636fc29fa60 con 0x5636fc3b41a0 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.481+0000 7f9b36000640 10 client.25847 handle_command_reply: tid=0 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.481+0000 7f9b55d78740 1 client.25847 shutdown 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 msgr2=0x5636fc345e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc345e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c03f070 comp rx=0 tx=0).stop 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 shutdown_connections 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5636fc3b41a0 0x5636fc3b65c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636fc33d940 0x5636fc337ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5636fc333270 0x5636fc337960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b36000640 0 client.25847 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:23:31.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 --2- 172.21.15.146:0/2204428020 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5636fc332950 0x5636fc345e70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 >> 172.21.15.146:0/2204428020 conn(0x5636fc32ec70 msgr2=0x5636fc32d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 shutdown_connections 2024-04-24T08:23:31.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 1 -- 172.21.15.146:0/2204428020 wait complete. 2024-04-24T08:23:31.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.482+0000 7f9b55d78740 20 client.25847 trim_cache size 0 max 16384 2024-04-24T08:23:31.488 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:23:31.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b300a3d20 msgr2=0x7f9b300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b300a3d20 0x7f9b300a61e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c1001e0 tx=0x7f9b3c0058d0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c19aa60 msgr2=0x7f9b4c19af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c19aa60 0x7f9b4c19af10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b4400eaa0 tx=0x7f9b4400ef60 comp rx=0 tx=0).stop 2024-04-24T08:23:31.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 shutdown_connections 2024-04-24T08:23:31.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9b300a3d20 0x7f9b300a61e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9b4c19aa60 0x7f9b4c19af10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4c109580 0x7f9b4c1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 --2- 172.21.15.146:0/839281698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b4c1044b0 0x7f9b4c1a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 >> 172.21.15.146:0/839281698 conn(0x7f9b4c0ffd90 msgr2=0x7f9b4c101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 shutdown_connections 2024-04-24T08:23:31.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.483+0000 7f9b53000640 1 -- 172.21.15.146:0/839281698 wait complete. 2024-04-24T08:23:31.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:31 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/303611915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:23:31.504 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:23:31.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:31 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/303611915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:23:31.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 -- 172.21.15.146:0/3805090718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54106250 msgr2=0x7f8b541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 --2- 172.21.15.146:0/3805090718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54106250 0x7f8b541066e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009ca0 tx=0x7f8b4c02a950 comp rx=0 tx=0).stop 2024-04-24T08:23:31.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 -- 172.21.15.146:0/3805090718 shutdown_connections 2024-04-24T08:23:31.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 --2- 172.21.15.146:0/3805090718 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b54101770 0x7f8b54105c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.728 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 --2- 172.21.15.146:0/3805090718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54106250 0x7f8b541066e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 --2- 172.21.15.146:0/3805090718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b54100e50 0x7f8b54101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 -- 172.21.15.146:0/3805090718 >> 172.21.15.146:0/3805090718 conn(0x7f8b540fc880 msgr2=0x7f8b540feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.726+0000 7f8b5a200640 1 -- 172.21.15.146:0/3805090718 shutdown_connections 2024-04-24T08:23:31.729 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.727+0000 7f8b5a200640 1 -- 172.21.15.146:0/3805090718 wait complete. 2024-04-24T08:23:31.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.728+0000 7f8b5a200640 1 Processor -- start 2024-04-24T08:23:31.730 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.728+0000 7f8b5a200640 1 -- start start 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.728+0000 7f8b5a200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54100e50 0x7f8b541a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b5a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b54101770 0x7f8b541a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b5a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b54106250 0x7f8b5419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b5a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b54118840 con 0x7f8b54106250 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b5a200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541186c0 con 0x7f8b54100e50 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b5a200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541189c0 con 0x7f8b54101770 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54100e50 0x7f8b541a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54100e50 0x7f8b541a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60100/0 (socket says 172.21.15.146:60100) 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 -- 172.21.15.146:0/748354267 learned_addr learned my addr 172.21.15.146:0/748354267 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b53e00640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b54101770 0x7f8b541a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.731 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b59800640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b54106250 0x7f8b5419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 -- 172.21.15.146:0/748354267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b54101770 msgr2=0x7f8b541a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b54101770 0x7f8b541a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 -- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b54106250 msgr2=0x7f8b5419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b54106250 0x7f8b5419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b58e00640 1 -- 172.21.15.146:0/748354267 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5419f3e0 con 0x7f8b54100e50 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b53e00640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b54101770 0x7f8b541a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.729+0000 7f8b59800640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b54106250 0x7f8b5419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.730+0000 7f8b58e00640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54100e50 0x7f8b541a4a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800e9f0 tx=0x7f8b4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.730+0000 7f8b51600640 1 -- 172.21.15.146:0/748354267 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800cd80 con 0x7f8b54100e50 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.730+0000 7f8b51600640 1 -- 172.21.15.146:0/748354267 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b48010910 con 0x7f8b54100e50 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.730+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b5419f6d0 con 0x7f8b54100e50 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.730+0000 7f8b51600640 1 -- 172.21.15.146:0/748354267 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48018990 con 0x7f8b54100e50 2024-04-24T08:23:31.733 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.730+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b541ab9e0 con 0x7f8b54100e50 2024-04-24T08:23:31.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.734+0000 7f8b51600640 1 -- 172.21.15.146:0/748354267 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f8b48010ac0 con 0x7f8b54100e50 2024-04-24T08:23:31.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.735+0000 7f8b5e67c740 1 Processor -- start 2024-04-24T08:23:31.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.735+0000 7f8b5e67c740 1 -- start start 2024-04-24T08:23:31.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.736+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x563548168f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.736+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x5635481735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.736+0000 7f8b51600640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b3c0a3da0 0x7f8b3c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.736+0000 7f8b51600640 1 -- 172.21.15.146:0/748354267 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8b480d0b80 con 0x7f8b54100e50 2024-04-24T08:23:31.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.736+0000 7f8b53e00640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b3c0a3da0 0x7f8b3c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x5635481735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x5635481735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60114/0 (socket says 172.21.15.146:60114) 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b59800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x563548168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b59800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x563548168f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46452/0 (socket says 172.21.15.146:46452) 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x563548175f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5635480ce1d0 con 0x563548173b30 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b58e00640 1 -- 172.21.15.146:0/138861671 learned_addr learned my addr 172.21.15.146:0/138861671 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b5e67c740 1 -- 172.21.15.146:0/138861671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563548074450 con 0x563548169460 2024-04-24T08:23:31.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b5e67c740 1 -- 172.21.15.146:0/138861671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5635480ca190 con 0x563548168b40 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b53e00640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b3c0a3da0 0x7f8b3c0a6260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8b400095f0 tx=0x7f8b40005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.737+0000 7f8b58e00640 1 -- 172.21.15.146:0/138861671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 msgr2=0x563548168f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.738+0000 7f8b58e00640 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x563548168f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.738+0000 7f8b58e00640 1 -- 172.21.15.146:0/138861671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 msgr2=0x563548175f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.738+0000 7f8b59800640 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x563548175f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.738+0000 7f8b58e00640 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x563548175f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.738+0000 7f8b58e00640 1 -- 172.21.15.146:0/138861671 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563548131310 con 0x563548169460 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.738+0000 7f8b59800640 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x563548175f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:23:31.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.738+0000 7f8b58e00640 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x5635481735f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8b480a5210 tx=0x7f8b480b5640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7018349249e7e513 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.739+0000 7f8b46000640 1 -- 172.21.15.146:0/138861671 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b480a5740 con 0x563548169460 2024-04-24T08:23:31.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.739+0000 7f8b46000640 1 -- 172.21.15.146:0/138861671 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b4809ab70 con 0x563548169460 2024-04-24T08:23:31.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.739+0000 7f8b46000640 1 -- 172.21.15.146:0/138861671 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b480ce500 con 0x563548169460 2024-04-24T08:23:31.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 -- 172.21.15.146:0/138861671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 msgr2=0x5635481735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x5635481735f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8b480a5210 tx=0x7f8b480b5640 comp rx=0 tx=0).stop 2024-04-24T08:23:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 -- 172.21.15.146:0/138861671 shutdown_connections 2024-04-24T08:23:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x563548175f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x5635481735f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/138861671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x563548168f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 -- 172.21.15.146:0/138861671 >> 172.21.15.146:0/138861671 conn(0x563548164e60 msgr2=0x563548165240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 -- 172.21.15.146:0/138861671 shutdown_connections 2024-04-24T08:23:31.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 -- 172.21.15.146:0/138861671 wait complete. 2024-04-24T08:23:31.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.740+0000 7f8b5e67c740 1 -- start start 2024-04-24T08:23:31.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.741+0000 7f8b5e67c740 1 Processor -- start 2024-04-24T08:23:31.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.741+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x5635481708f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.741+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548170e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.741+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x563548171370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5635481769c0 con 0x563548168b40 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563548087810 con 0x563548169460 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563548176b40 con 0x563548173b30 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b5e67c740 10 client.0 _ll_register_callbacks cb 0x563548055140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x5635481708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x5635481708f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36428/0 (socket says 172.21.15.146:36428) 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b58e00640 1 -- 172.21.15.146:0/2411331591 learned_addr learned my addr 172.21.15.146:0/2411331591 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548170e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b59800640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x563548171370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b53e00640 1 -- 172.21.15.146:0/2411331591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 msgr2=0x563548171370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x563548171370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b53e00640 1 -- 172.21.15.146:0/2411331591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 msgr2=0x5635481708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x5635481708f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b53e00640 1 -- 172.21.15.146:0/2411331591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635480983f0 con 0x563548169460 2024-04-24T08:23:31.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.742+0000 7f8b58e00640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x5635481708f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.743+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548170e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000ec80 tx=0x7f8b40004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=551e0ffd7ce0b5af server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.743+0000 7f8b3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.743+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2411331591 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4001ccb0 con 0x563548169460 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.743+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b4809ab70 con 0x563548169460 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.743+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2411331591 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8b4001d410 con 0x563548169460 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.743+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2411331591 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8b4001ce50 con 0x563548169460 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.744+0000 7f8b5e67c740 10 client.25859 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.744+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563547de03f0 con 0x563548169460 2024-04-24T08:23:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.744+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2411331591 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b4809ab70 con 0x563548169460 2024-04-24T08:23:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2411331591 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f8b40028800 con 0x563548169460 2024-04-24T08:23:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b5e67c740 10 client.25859 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b5e67c740 10 client.25859 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b5e67c740 20 client.25859 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563548183cf0 0x5635481860e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b5e67c740 4 client.25859 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:23:31.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635480dbbf0 con 0x563548183cf0 2024-04-24T08:23:31.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.745+0000 7f8b58e00640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563548183cf0 0x5635481860e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.746+0000 7f8b58e00640 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563548183cf0 0x5635481860e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fb9429ed8a6fdee server_cookie=3a77f544c92ff438 in_seq=0 out_seq=0 2024-04-24T08:23:31.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.746+0000 7f8b3aa00640 10 client.25859 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.746+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2411331591 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635480dbbf0 con 0x563548183cf0 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.746+0000 7f8b3aa00640 10 client.25859 handle_command_reply: tid=0 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.746+0000 7f8b5e67c740 1 client.25859 shutdown 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 msgr2=0x563548170e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548170e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000ec80 tx=0x7f8b40004730 comp rx=0 tx=0).stop 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 shutdown_connections 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563548183cf0 0x5635481860e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x563548171370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548170e30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2411331591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x5635481708f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b3aa00640 0 client.25859 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 >> 172.21.15.146:0/2411331591 conn(0x563548164e60 msgr2=0x56354816ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 shutdown_connections 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2411331591 wait complete. 2024-04-24T08:23:31.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.747+0000 7f8b5e67c740 20 client.25859 trim_cache size 0 max 16384 2024-04-24T08:23:31.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.762+0000 7f8b5e67c740 1 Processor -- start 2024-04-24T08:23:31.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.762+0000 7f8b5e67c740 1 -- start start 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.762+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x56354817f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.762+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548171190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.762+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x56354818b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541b6a30 con 0x563548173b30 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541b4330 con 0x563548169460 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541b2180 con 0x563548168b40 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548171190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b59800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x56354818b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x56354817f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548171190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60130/0 (socket says 172.21.15.146:60130) 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 -- 172.21.15.146:0/387757954 learned_addr learned my addr 172.21.15.146:0/387757954 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 -- 172.21.15.146:0/387757954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 msgr2=0x56354817f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 --2- 172.21.15.146:0/387757954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x56354817f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 -- 172.21.15.146:0/387757954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 msgr2=0x56354818b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 --2- 172.21.15.146:0/387757954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x56354818b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.763+0000 7f8b53e00640 1 -- 172.21.15.146:0/387757954 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563547eaa7d0 con 0x563548169460 2024-04-24T08:23:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.764+0000 7f8b53e00640 1 --2- 172.21.15.146:0/387757954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548171190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8b40028dc0 tx=0x7f8b400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dbc34fd2786d2ff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.764+0000 7f8b45600640 1 -- 172.21.15.146:0/387757954 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4001ce50 con 0x563548169460 2024-04-24T08:23:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.764+0000 7f8b45600640 1 -- 172.21.15.146:0/387757954 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b40036070 con 0x563548169460 2024-04-24T08:23:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.764+0000 7f8b45600640 1 -- 172.21.15.146:0/387757954 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40031b60 con 0x563548169460 2024-04-24T08:23:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 -- 172.21.15.146:0/387757954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 msgr2=0x563548171190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/387757954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548171190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8b40028dc0 tx=0x7f8b400046c0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 -- 172.21.15.146:0/387757954 shutdown_connections 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/387757954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548173b30 0x56354818b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/387757954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548171190 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/387757954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548168b40 0x56354817f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 -- 172.21.15.146:0/387757954 >> 172.21.15.146:0/387757954 conn(0x563548164e60 msgr2=0x56354816f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 -- 172.21.15.146:0/387757954 shutdown_connections 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.765+0000 7f8b5e67c740 1 -- 172.21.15.146:0/387757954 wait complete. 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 -- start start 2024-04-24T08:23:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 Processor -- start 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x563548174700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548174c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x56354817ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541b27f0 con 0x563548168b40 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541b3c60 con 0x563548169460 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b541b2ec0 con 0x563548173b30 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b5e67c740 10 client.0 _ll_register_callbacks cb 0x7f8b4000e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.766+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x563548174700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b58e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x563548174700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36444/0 (socket says 172.21.15.146:36444) 2024-04-24T08:23:31.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b58e00640 1 -- 172.21.15.146:0/2025050646 learned_addr learned my addr 172.21.15.146:0/2025050646 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548174c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b59800640 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x56354817ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b53e00640 1 -- 172.21.15.146:0/2025050646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 msgr2=0x56354817ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x56354817ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b53e00640 1 -- 172.21.15.146:0/2025050646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 msgr2=0x563548174700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x563548174700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b53e00640 1 -- 172.21.15.146:0/2025050646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56354817b350 con 0x563548169460 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b53e00640 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548174c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x563548171a80 tx=0x7f8b40014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b34f52af6cc302 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.767+0000 7f8b3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.768+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2025050646 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40031470 con 0x563548169460 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.768+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2025050646 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8b40005040 con 0x563548169460 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.768+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b40036070 con 0x563548169460 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.768+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2025050646 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8b40005240 con 0x563548169460 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.768+0000 7f8b5e67c740 10 client.25865 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:31.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.768+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56354817c610 con 0x563548169460 2024-04-24T08:23:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2025050646 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b40036070 con 0x563548169460 2024-04-24T08:23:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2025050646 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f8b40004a50 con 0x563548169460 2024-04-24T08:23:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b5e67c740 10 client.25865 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b5e67c740 10 client.25865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b5e67c740 20 client.25865 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5635481ea6d0 0x5635481ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b5e67c740 4 client.25865 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5635480d5a60 con 0x5635481ea6d0 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.769+0000 7f8b58e00640 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5635481ea6d0 0x5635481ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b3aa00640 10 client.25865 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b58e00640 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5635481ea6d0 0x5635481ecaf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95764a763c4e8e9c server_cookie=e9b1ceb88a31ecae in_seq=0 out_seq=0 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b3aa00640 1 -- 172.21.15.146:0/2025050646 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5635480d5a60 con 0x5635481ea6d0 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b3aa00640 10 client.25865 handle_command_reply: tid=0 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 client.25865 shutdown 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 msgr2=0x563548174c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548174c40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x563548171a80 tx=0x7f8b40014000 comp rx=0 tx=0).stop 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 shutdown_connections 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5635481ea6d0 0x5635481ecaf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563548173b30 0x56354817ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563548169460 0x563548174c40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 --2- 172.21.15.146:0/2025050646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563548168b40 0x563548174700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 >> 172.21.15.146:0/2025050646 conn(0x563548164e60 msgr2=0x5635481636c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b3aa00640 0 client.25865 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.770+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 shutdown_connections 2024-04-24T08:23:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.771+0000 7f8b5e67c740 1 -- 172.21.15.146:0/2025050646 wait complete. 2024-04-24T08:23:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.771+0000 7f8b5e67c740 20 client.25865 trim_cache size 0 max 16384 2024-04-24T08:23:31.775 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.771+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b3c0a3da0 msgr2=0x7f8b3c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.771+0000 7f8b5a200640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b3c0a3da0 0x7f8b3c0a6260 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8b400095f0 tx=0x7f8b40005f90 comp rx=0 tx=0).stop 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.771+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54100e50 msgr2=0x7f8b541a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.771+0000 7f8b5a200640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54100e50 0x7f8b541a4a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800e9f0 tx=0x7f8b4800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 shutdown_connections 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b3c0a3da0 0x7f8b3c0a6260 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b54101770 0x7f8b541a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b54106250 0x7f8b5419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 --2- 172.21.15.146:0/748354267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b54100e50 0x7f8b541a4a60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 >> 172.21.15.146:0/748354267 conn(0x7f8b540fc880 msgr2=0x7f8b540fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 shutdown_connections 2024-04-24T08:23:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:31.772+0000 7f8b5a200640 1 -- 172.21.15.146:0/748354267 wait complete. 2024-04-24T08:23:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:31 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/303611915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:23:32.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:32 smithi153 ceph-mon[32746]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-24T08:23:32.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:32 smithi146 ceph-mon[27805]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-24T08:23:32.793 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:23:32.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:23:32.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:23:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:32 smithi177 ceph-mon[33046]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 -- 172.21.15.146:0/1105247084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 msgr2=0x7f5748104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 --2- 172.21.15.146:0/1105247084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f5748104830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5730009900 tx=0x7f573002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 -- 172.21.15.146:0/1105247084 shutdown_connections 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 --2- 172.21.15.146:0/1105247084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5748104e00 0x7f5748108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 --2- 172.21.15.146:0/1105247084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5748109520 0x7f57481110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 --2- 172.21.15.146:0/1105247084 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f5748104830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 -- 172.21.15.146:0/1105247084 >> 172.21.15.146:0/1105247084 conn(0x7f57480ffc70 msgr2=0x7f5748102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 -- 172.21.15.146:0/1105247084 shutdown_connections 2024-04-24T08:23:33.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.015+0000 7f574e800640 1 -- 172.21.15.146:0/1105247084 wait complete. 2024-04-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 Processor -- start 2024-04-24T08:23:33.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 -- start start 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f57481a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5748109520 0x7f57481a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574819a940 0x7f574819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5748114600 con 0x7f5748109520 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5748114480 con 0x7f5748104450 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.016+0000 7f574e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5748114780 con 0x7f574819a940 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f57481a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5746a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5748109520 0x7f57481a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f57481a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60140/0 (socket says 172.21.15.146:60140) 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 -- 172.21.15.146:0/3124347618 learned_addr learned my addr 172.21.15.146:0/3124347618 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747e00640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574819a940 0x7f574819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 -- 172.21.15.146:0/3124347618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574819a940 msgr2=0x7f574819adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574819a940 0x7f574819adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 -- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5748109520 msgr2=0x7f57481a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5748109520 0x7f57481a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 -- 172.21.15.146:0/3124347618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574819b460 con 0x7f5748104450 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747e00640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574819a940 0x7f574819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.017+0000 7f5747400640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f57481a0790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5730004770 tx=0x7f5730003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.018+0000 7f573fe00640 1 -- 172.21.15.146:0/3124347618 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573003f070 con 0x7f5748104450 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.018+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574819b6f0 con 0x7f5748104450 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.018+0000 7f573fe00640 1 -- 172.21.15.146:0/3124347618 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5730031ca0 con 0x7f5748104450 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.018+0000 7f573fe00640 1 -- 172.21.15.146:0/3124347618 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730035030 con 0x7f5748104450 2024-04-24T08:23:33.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.018+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57481a78e0 con 0x7f5748104450 2024-04-24T08:23:33.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.021+0000 7f573fe00640 1 -- 172.21.15.146:0/3124347618 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f5730042070 con 0x7f5748104450 2024-04-24T08:23:33.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 Processor -- start 2024-04-24T08:23:33.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 -- start start 2024-04-24T08:23:33.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa025470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa02fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa032470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5603f9f8ae70 con 0x5603fa025090 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5603f9f31380 con 0x5603fa030080 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.022+0000 7f57517ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5603f9f86e30 con 0x5603fa0259b0 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5747400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa02fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa032470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60146/0 (socket says 172.21.15.146:60146) 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 -- 172.21.15.146:0/1621207168 learned_addr learned my addr 172.21.15.146:0/1621207168 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5747e00640 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa025470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 -- 172.21.15.146:0/1621207168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 msgr2=0x5603fa02fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa02fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 -- 172.21.15.146:0/1621207168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 msgr2=0x5603fa025470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa025470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 -- 172.21.15.146:0/1621207168 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603f9d657e0 con 0x5603fa030080 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5747e00640 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa025470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f5746a00640 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa032470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f573800bc30 tx=0x7f57380107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5aab5ec3e7417ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.023+0000 7f573fe00640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f572c0a3d20 0x7f572c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.024+0000 7f573cc00640 1 -- 172.21.15.146:0/1621207168 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573801c480 con 0x5603fa030080 2024-04-24T08:23:33.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.024+0000 7f573cc00640 1 -- 172.21.15.146:0/1621207168 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f573801cb00 con 0x5603fa030080 2024-04-24T08:23:33.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.024+0000 7f573cc00640 1 -- 172.21.15.146:0/1621207168 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738011c00 con 0x5603fa030080 2024-04-24T08:23:33.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.024+0000 7f573fe00640 1 -- 172.21.15.146:0/3124347618 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f57300f1060 con 0x7f5748104450 2024-04-24T08:23:33.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.024+0000 7f5747e00640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f572c0a3d20 0x7f572c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.024+0000 7f57517ae740 1 -- 172.21.15.146:0/1621207168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 msgr2=0x5603fa032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.024+0000 7f57517ae740 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa032470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f573800bc30 tx=0x7f57380107a0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f5747e00640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f572c0a3d20 0x7f572c0a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5740005fd0 tx=0x7f5740009990 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 -- 172.21.15.146:0/1621207168 shutdown_connections 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa02fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa032470 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 --2- 172.21.15.146:0/1621207168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa025470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 -- 172.21.15.146:0/1621207168 >> 172.21.15.146:0/1621207168 conn(0x5603fa0213b0 msgr2=0x5603fa021790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 -- 172.21.15.146:0/1621207168 shutdown_connections 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 -- 172.21.15.146:0/1621207168 wait complete. 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.025+0000 7f57517ae740 1 -- start start 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 1 Processor -- start 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa02d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa0259b0 0x5603fa02d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa030080 0x5603fa02daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5603fa032e70 con 0x5603fa025090 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5603f9f44660 con 0x5603fa0259b0 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5603fa032ff0 con 0x5603fa030080 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f57517ae740 10 client.0 _ll_register_callbacks cb 0x5603f9f32aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f5746a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa0259b0 0x5603fa02d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f5747e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa030080 0x5603fa02daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f5746a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa0259b0 0x5603fa02d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60160/0 (socket says 172.21.15.146:60160) 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f5746a00640 1 -- 172.21.15.146:0/2121510768 learned_addr learned my addr 172.21.15.146:0/2121510768 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.026+0000 7f5747400640 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa02d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5746a00640 1 -- 172.21.15.146:0/2121510768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa030080 msgr2=0x5603fa02daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5746a00640 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa030080 0x5603fa02daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5746a00640 1 -- 172.21.15.146:0/2121510768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 msgr2=0x5603fa02d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5746a00640 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa02d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5746a00640 1 -- 172.21.15.146:0/2121510768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603f9dc5f70 con 0x5603fa0259b0 2024-04-24T08:23:33.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5747400640 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa02d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5746a00640 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa0259b0 0x5603fa02d560 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5738005890 tx=0x7f573800b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac40700243f42474 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5735600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5735600640 1 -- 172.21.15.146:0/2121510768 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573802b060 con 0x5603fa0259b0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f573801cb00 con 0x5603fa0259b0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.027+0000 7f5735600640 1 -- 172.21.15.146:0/2121510768 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f573801c680 con 0x5603fa0259b0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f5735600640 1 -- 172.21.15.146:0/2121510768 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f573801cdf0 con 0x5603fa0259b0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 10 client.25877 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5603f9c5d170 con 0x5603fa0259b0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f5735600640 1 -- 172.21.15.146:0/2121510768 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f573801cb00 con 0x5603fa0259b0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f5735600640 1 -- 172.21.15.146:0/2121510768 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f5738025070 con 0x5603fa0259b0 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 10 client.25877 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 10 client.25877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:33.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 20 client.25877 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:33.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa040310 0x5603fa042700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 4 client.25877 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:23:33.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.028+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603f9f98890 con 0x5603fa040310 2024-04-24T08:23:33.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.029+0000 7f5747400640 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa040310 0x5603fa042700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.029+0000 7f5747400640 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa040310 0x5603fa042700 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c1fe8dd75651437 server_cookie=890699e1fca9796d in_seq=0 out_seq=0 2024-04-24T08:23:33.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.029+0000 7f5735600640 10 client.25877 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.030+0000 7f5735600640 1 -- 172.21.15.146:0/2121510768 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603f9f98890 con 0x5603fa040310 2024-04-24T08:23:33.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.030+0000 7f5735600640 10 client.25877 handle_command_reply: tid=0 2024-04-24T08:23:33.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 client.25877 shutdown 2024-04-24T08:23:33.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa0259b0 msgr2=0x5603fa02d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa0259b0 0x5603fa02d560 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5738005890 tx=0x7f573800b860 comp rx=0 tx=0).stop 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 shutdown_connections 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa040310 0x5603fa042700 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa030080 0x5603fa02daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa0259b0 0x5603fa02d560 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 --2- 172.21.15.146:0/2121510768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa02d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f5735600640 0 client.25877 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 >> 172.21.15.146:0/2121510768 conn(0x5603fa0213b0 msgr2=0x5603fa0271b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 shutdown_connections 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 1 -- 172.21.15.146:0/2121510768 wait complete. 2024-04-24T08:23:33.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.031+0000 7f57517ae740 20 client.25877 trim_cache size 0 max 16384 2024-04-24T08:23:33.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 Processor -- start 2024-04-24T08:23:33.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 -- start start 2024-04-24T08:23:33.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa025090 0x5603fa03bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa020c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa030080 0x5603fa047d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f57481b2810 con 0x5603fa030080 2024-04-24T08:23:33.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57481b00e0 con 0x5603fa025090 2024-04-24T08:23:33.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.051+0000 7f57517ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f57481adf30 con 0x5603fa0259b0 2024-04-24T08:23:33.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa025090 0x5603fa03bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa025090 0x5603fa03bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60172/0 (socket says 172.21.15.146:60172) 2024-04-24T08:23:33.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 -- 172.21.15.146:0/130551031 learned_addr learned my addr 172.21.15.146:0/130551031 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747e00640 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa030080 0x5603fa047d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5746a00640 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa020c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 -- 172.21.15.146:0/130551031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 msgr2=0x5603fa020c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa020c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 -- 172.21.15.146:0/130551031 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa030080 msgr2=0x5603fa047d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa030080 0x5603fa047d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 -- 172.21.15.146:0/130551031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603fa02bfd0 con 0x5603fa025090 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5746a00640 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa020c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.052+0000 7f5747400640 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa025090 0x5603fa03bd30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5730000c00 tx=0x7f573003b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18b2e111ace1a031 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f5737e00640 1 -- 172.21.15.146:0/130551031 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730049040 con 0x5603fa025090 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f5737e00640 1 -- 172.21.15.146:0/130551031 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57300bd6b0 con 0x5603fa025090 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f5737e00640 1 -- 172.21.15.146:0/130551031 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57300dd040 con 0x5603fa025090 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f57517ae740 1 -- 172.21.15.146:0/130551031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa025090 msgr2=0x5603fa03bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f57517ae740 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa025090 0x5603fa03bd30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5730000c00 tx=0x7f573003b990 comp rx=0 tx=0).stop 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f57517ae740 1 -- 172.21.15.146:0/130551031 shutdown_connections 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f57517ae740 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa020c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f57517ae740 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa030080 0x5603fa047d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f57517ae740 1 --2- 172.21.15.146:0/130551031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa025090 0x5603fa03bd30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.053+0000 7f57517ae740 1 -- 172.21.15.146:0/130551031 >> 172.21.15.146:0/130551031 conn(0x5603fa0213b0 msgr2=0x5603fa02d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.054+0000 7f57517ae740 1 -- 172.21.15.146:0/130551031 shutdown_connections 2024-04-24T08:23:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.054+0000 7f57517ae740 1 -- 172.21.15.146:0/130551031 wait complete. 2024-04-24T08:23:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.054+0000 7f57517ae740 1 -- start start 2024-04-24T08:23:33.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.054+0000 7f57517ae740 1 Processor -- start 2024-04-24T08:23:33.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f57517ae740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa0383f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f57517ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa034770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f57517ae740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa034cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f57517ae740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f57481ae5a0 con 0x5603fa025090 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f57517ae740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57481afa10 con 0x5603fa030080 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f57517ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f57481aec70 con 0x5603fa0259b0 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f57517ae740 10 client.0 _ll_register_callbacks cb 0x7f57300f49e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa0383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa034cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa0383f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36498/0 (socket says 172.21.15.146:36498) 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 -- 172.21.15.146:0/565162421 learned_addr learned my addr 172.21.15.146:0/565162421 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5746a00640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa034770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 -- 172.21.15.146:0/565162421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 msgr2=0x5603fa034770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa034770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 -- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 msgr2=0x5603fa034cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa034cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747400640 1 -- 172.21.15.146:0/565162421 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603fa0327e0 con 0x5603fa025090 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5746a00640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa034770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:33.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.055+0000 7f5747e00640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa034cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f5747400640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa0383f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f57300bcf40 tx=0x7f573002a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b5d802d21c7277 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f5735600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f5735600640 1 -- 172.21.15.146:0/565162421 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730049070 con 0x5603fa025090 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f5735600640 1 -- 172.21.15.146:0/565162421 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f573003b5f0 con 0x5603fa025090 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57300bd6b0 con 0x5603fa025090 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f5735600640 1 -- 172.21.15.146:0/565162421 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f57300dd070 con 0x5603fa025090 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f57517ae740 10 client.15741 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5603fa0a71e0 con 0x5603fa025090 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.056+0000 7f5735600640 1 -- 172.21.15.146:0/565162421 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57300bd6b0 con 0x5603fa025090 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f5735600640 1 -- 172.21.15.146:0/565162421 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f57300eea70 con 0x5603fa025090 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f57517ae740 10 client.15741 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f57517ae740 10 client.15741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f57517ae740 20 client.15741 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f57517ae740 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa0a7850 0x5603fa0a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f57517ae740 4 client.15741 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:23:33.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5603f9f92700 con 0x5603fa0a7850 2024-04-24T08:23:33.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f5746a00640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa0a7850 0x5603fa0a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f5746a00640 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa0a7850 0x5603fa0a9c70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4547dd13cd1072a server_cookie=5096b96cec75b9c5 in_seq=0 out_seq=0 2024-04-24T08:23:33.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.057+0000 7f5735600640 10 client.15741 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.348+0000 7f5735600640 1 -- 172.21.15.146:0/565162421 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5603f9f92700 con 0x5603fa0a7850 2024-04-24T08:23:33.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.348+0000 7f5735600640 10 client.15741 handle_command_reply: tid=0 2024-04-24T08:23:33.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.348+0000 7f57517ae740 1 client.15741 shutdown 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 msgr2=0x5603fa0383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa0383f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f57300bcf40 tx=0x7f573002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 shutdown_connections 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603fa0a7850 0x5603fa0a9c70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603fa0259b0 0x5603fa034770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603fa030080 0x5603fa034cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 --2- 172.21.15.146:0/565162421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603fa025090 0x5603fa0383f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f5735600640 0 client.15741 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.349+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 >> 172.21.15.146:0/565162421 conn(0x5603fa0213b0 msgr2=0x5603fa026800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.350+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 shutdown_connections 2024-04-24T08:23:33.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.350+0000 7f57517ae740 1 -- 172.21.15.146:0/565162421 wait complete. 2024-04-24T08:23:33.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.350+0000 7f57517ae740 20 client.15741 trim_cache size 0 max 16384 2024-04-24T08:23:33.353 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:23:33.353 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:23:33.353 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "bf469489-e50d-44c3-b271-2c783de6a32b", 2024-04-24T08:23:33.353 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:23:33.353 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:23:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f572c0a3d20 msgr2=0x7f572c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f572c0a3d20 0x7f572c0a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5740005fd0 tx=0x7f5740009990 comp rx=0 tx=0).stop 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 msgr2=0x7f57481a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f57481a0790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5730004770 tx=0x7f5730003c10 comp rx=0 tx=0).stop 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 shutdown_connections 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f572c0a3d20 0x7f572c0a61e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f574819a940 0x7f574819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5748109520 0x7f57481a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 --2- 172.21.15.146:0/3124347618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5748104450 0x7f57481a0790 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.351+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 >> 172.21.15.146:0/3124347618 conn(0x7f57480ffc70 msgr2=0x7f5748101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.352+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 shutdown_connections 2024-04-24T08:23:33.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.352+0000 7f574e800640 1 -- 172.21.15.146:0/3124347618 wait complete. 2024-04-24T08:23:33.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:23:33.379 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:23:33.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 -- 172.21.15.146:0/3334781101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5098103980 msgr2=0x7f509810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 --2- 172.21.15.146:0/3334781101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5098103980 0x7f509810be60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f508000dd10 tx=0x7f5080032a30 comp rx=0 tx=0).stop 2024-04-24T08:23:33.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 -- 172.21.15.146:0/3334781101 shutdown_connections 2024-04-24T08:23:33.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 --2- 172.21.15.146:0/3334781101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f509810c480 0x7f509810e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 --2- 172.21.15.146:0/3334781101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5098103980 0x7f509810be60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 --2- 172.21.15.146:0/3334781101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5098103060 0x7f5098103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 -- 172.21.15.146:0/3334781101 >> 172.21.15.146:0/3334781101 conn(0x7f50980fc880 msgr2=0x7f50980feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 -- 172.21.15.146:0/3334781101 shutdown_connections 2024-04-24T08:23:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.606+0000 7f5097200640 1 -- 172.21.15.146:0/3334781101 wait complete. 2024-04-24T08:23:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.607+0000 7f5097200640 1 Processor -- start 2024-04-24T08:23:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.608+0000 7f5097200640 1 -- start start 2024-04-24T08:23:33.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.608+0000 7f5097200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5098103060 0x7f50981a7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.608+0000 7f5097200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098103980 0x7f50981a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.608+0000 7f5097200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509810c480 0x7f50981abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.608+0000 7f5097200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5098121130 con 0x7f5098103060 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.608+0000 7f5097200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5098120fb0 con 0x7f509810c480 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.608+0000 7f5097200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f50981212b0 con 0x7f5098103980 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5095e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5098103060 0x7f50981a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5095400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098103980 0x7f50981a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5095400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098103980 0x7f50981a7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46566/0 (socket says 172.21.15.146:46566) 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5095400640 1 -- 172.21.15.146:0/1771054264 learned_addr learned my addr 172.21.15.146:0/1771054264 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5096800640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509810c480 0x7f50981abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5096800640 1 -- 172.21.15.146:0/1771054264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098103980 msgr2=0x7f50981a7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5096800640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098103980 0x7f50981a7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5096800640 1 -- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5098103060 msgr2=0x7f50981a7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5096800640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5098103060 0x7f50981a7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.609+0000 7f5096800640 1 -- 172.21.15.146:0/1771054264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50981ac620 con 0x7f509810c480 2024-04-24T08:23:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.610+0000 7f5096800640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509810c480 0x7f50981abef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f509000be20 tx=0x7f509000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.610+0000 7f508ea00640 1 -- 172.21.15.146:0/1771054264 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090019070 con 0x7f509810c480 2024-04-24T08:23:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.610+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50981ac8b0 con 0x7f509810c480 2024-04-24T08:23:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.610+0000 7f508ea00640 1 -- 172.21.15.146:0/1771054264 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5090014920 con 0x7f509810c480 2024-04-24T08:23:33.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.610+0000 7f508ea00640 1 -- 172.21.15.146:0/1771054264 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090012920 con 0x7f509810c480 2024-04-24T08:23:33.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.610+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50981b41a0 con 0x7f509810c480 2024-04-24T08:23:33.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.614+0000 7f508ea00640 1 -- 172.21.15.146:0/1771054264 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f509001c070 con 0x7f509810c480 2024-04-24T08:23:33.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.615+0000 7f509fe29740 1 Processor -- start 2024-04-24T08:23:33.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.616+0000 7f508ea00640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f507c0a3da0 0x7f507c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.616+0000 7f508ea00640 1 -- 172.21.15.146:0/1771054264 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f50900cffc0 con 0x7f509810c480 2024-04-24T08:23:33.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.616+0000 7f5095400640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f507c0a3da0 0x7f507c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.617+0000 7f509fe29740 1 -- start start 2024-04-24T08:23:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.617+0000 7f509fe29740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f02a30 0x5596c6f02e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.617+0000 7f509fe29740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f0d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.617+0000 7f509fe29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f0da20 0x5596c6f0fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.617+0000 7f509fe29740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5596c6e681f0 con 0x5596c6f02a30 2024-04-24T08:23:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.617+0000 7f509fe29740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5596c6e0e350 con 0x5596c6f03350 2024-04-24T08:23:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.617+0000 7f509fe29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5596c6e641b0 con 0x5596c6f0da20 2024-04-24T08:23:33.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f507c0a3da0 0x7f507c0a6260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5080005ab0 tx=0x7f5080005a40 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f0d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f0d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60210/0 (socket says 172.21.15.146:60210) 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 -- 172.21.15.146:0/4082409831 learned_addr learned my addr 172.21.15.146:0/4082409831 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5096800640 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f0da20 0x5596c6f0fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095e00640 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f02a30 0x5596c6f02e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 -- 172.21.15.146:0/4082409831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f0da20 msgr2=0x5596c6f0fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f0da20 0x5596c6f0fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 -- 172.21.15.146:0/4082409831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f02a30 msgr2=0x5596c6f02e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f02a30 0x5596c6f02e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095400640 1 -- 172.21.15.146:0/4082409831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596c6e62f50 con 0x5596c6f03350 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.618+0000 7f5095e00640 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f02a30 0x5596c6f02e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.619+0000 7f5095400640 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f0d4e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f508000de30 tx=0x7f5080014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8254ecc421b70de3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.619+0000 7f5087400640 1 -- 172.21.15.146:0/4082409831 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080014070 con 0x5596c6f03350 2024-04-24T08:23:33.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.619+0000 7f5087400640 1 -- 172.21.15.146:0/4082409831 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f508005f070 con 0x5596c6f03350 2024-04-24T08:23:33.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.620+0000 7f5087400640 1 -- 172.21.15.146:0/4082409831 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508000b9b0 con 0x5596c6f03350 2024-04-24T08:23:33.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 -- 172.21.15.146:0/4082409831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 msgr2=0x5596c6f0d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f0d4e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f508000de30 tx=0x7f5080014000 comp rx=0 tx=0).stop 2024-04-24T08:23:33.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 -- 172.21.15.146:0/4082409831 shutdown_connections 2024-04-24T08:23:33.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f0da20 0x5596c6f0fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f0d4e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 --2- 172.21.15.146:0/4082409831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f02a30 0x5596c6f02e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 -- 172.21.15.146:0/4082409831 >> 172.21.15.146:0/4082409831 conn(0x5596c6efed50 msgr2=0x5596c6eff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 -- 172.21.15.146:0/4082409831 shutdown_connections 2024-04-24T08:23:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.621+0000 7f509fe29740 1 -- 172.21.15.146:0/4082409831 wait complete. 2024-04-24T08:23:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.622+0000 7f509fe29740 1 -- start start 2024-04-24T08:23:33.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.622+0000 7f509fe29740 1 Processor -- start 2024-04-24T08:23:33.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f509fe29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f0a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f509fe29740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f03350 0x5596c6f0ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f509fe29740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f0da20 0x5596c6f0b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f509fe29740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5596c6f10850 con 0x5596c6f03350 2024-04-24T08:23:33.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f509fe29740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5596c6e21690 con 0x5596c6f0da20 2024-04-24T08:23:33.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f509fe29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5596c6f109d0 con 0x5596c6f02a30 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f509fe29740 10 client.0 _ll_register_callbacks cb 0x5596c6e5d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.624+0000 7f5095400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f03350 0x5596c6f0ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.625+0000 7f5095400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f03350 0x5596c6f0ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36526/0 (socket says 172.21.15.146:36526) 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.625+0000 7f5095400640 1 -- 172.21.15.146:0/3425753956 learned_addr learned my addr 172.21.15.146:0/3425753956 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.625+0000 7f5095400640 1 -- 172.21.15.146:0/3425753956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 msgr2=0x5596c6f0a930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.625+0000 7f5095400640 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f0a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.625+0000 7f5095400640 1 -- 172.21.15.146:0/3425753956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f0da20 msgr2=0x5596c6f0b3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.625+0000 7f5095400640 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f0da20 0x5596c6f0b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.625+0000 7f5095400640 1 -- 172.21.15.146:0/3425753956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596c6e20920 con 0x5596c6f03350 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.626+0000 7f5095400640 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f03350 0x5596c6f0ae70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5080004410 tx=0x7f5080003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b885b69e431cafc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.626+0000 7f507be00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.626+0000 7f507be00640 1 -- 172.21.15.146:0/3425753956 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508000b950 con 0x5596c6f03350 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.626+0000 7f507be00640 1 -- 172.21.15.146:0/3425753956 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f508000baf0 con 0x5596c6f03350 2024-04-24T08:23:33.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.626+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f508005f070 con 0x5596c6f03350 2024-04-24T08:23:33.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.626+0000 7f507be00640 1 -- 172.21.15.146:0/3425753956 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f508000bcf0 con 0x5596c6f03350 2024-04-24T08:23:33.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 10 client.15753 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:33.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596c6c447d0 con 0x5596c6f03350 2024-04-24T08:23:33.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f507be00640 1 -- 172.21.15.146:0/3425753956 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f508005f070 con 0x5596c6f03350 2024-04-24T08:23:33.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f507be00640 1 -- 172.21.15.146:0/3425753956 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f5080076020 con 0x5596c6f03350 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 10 client.15753 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 20 client.15753 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f1dc20 0x5596c6f20010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 4 client.15753 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.627+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596c6e75c10 con 0x5596c6f1dc20 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.628+0000 7f5095e00640 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f1dc20 0x5596c6f20010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.628+0000 7f5095e00640 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f1dc20 0x5596c6f20010 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ee39b26a63198ad server_cookie=3f977279615c0826 in_seq=0 out_seq=0 2024-04-24T08:23:33.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.628+0000 7f507be00640 10 client.15753 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f507be00640 1 -- 172.21.15.146:0/3425753956 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596c6e75c10 con 0x5596c6f1dc20 2024-04-24T08:23:33.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f507be00640 10 client.15753 handle_command_reply: tid=0 2024-04-24T08:23:33.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 client.15753 shutdown 2024-04-24T08:23:33.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f03350 msgr2=0x5596c6f0ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f03350 0x5596c6f0ae70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5080004410 tx=0x7f5080003f90 comp rx=0 tx=0).stop 2024-04-24T08:23:33.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 shutdown_connections 2024-04-24T08:23:33.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f1dc20 0x5596c6f20010 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f0da20 0x5596c6f0b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f03350 0x5596c6f0ae70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 --2- 172.21.15.146:0/3425753956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f0a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 >> 172.21.15.146:0/3425753956 conn(0x5596c6efed50 msgr2=0x5596c6efdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.633+0000 7f507be00640 0 client.15753 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.634+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 shutdown_connections 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.634+0000 7f509fe29740 1 -- 172.21.15.146:0/3425753956 wait complete. 2024-04-24T08:23:33.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.634+0000 7f509fe29740 20 client.15753 trim_cache size 0 max 16384 2024-04-24T08:23:33.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 Processor -- start 2024-04-24T08:23:33.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 -- start start 2024-04-24T08:23:33.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f19740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f09200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f25730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f50981b69e0 con 0x5596c6f0da20 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50980740b0 con 0x5596c6f03350 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.659+0000 7f509fe29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5098071f00 con 0x5596c6f02a30 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5095400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f09200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5096800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f25730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5095400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f09200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:60214/0 (socket says 172.21.15.146:60214) 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5095400640 1 -- 172.21.15.146:0/2899553573 learned_addr learned my addr 172.21.15.146:0/2899553573 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5095e00640 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f19740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5096800640 1 -- 172.21.15.146:0/2899553573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 msgr2=0x5596c6f19740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5096800640 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f19740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5096800640 1 -- 172.21.15.146:0/2899553573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 msgr2=0x5596c6f09200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5096800640 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f09200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5096800640 1 -- 172.21.15.146:0/2899553573 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596c6f03180 con 0x5596c6f0da20 2024-04-24T08:23:33.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5095e00640 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f19740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:33.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.660+0000 7f5095400640 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f09200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:23:33.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.661+0000 7f5096800640 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f25730 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f50900cd7d0 tx=0x7f509000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765076c8e00cf2b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.661+0000 7f5086a00640 1 -- 172.21.15.146:0/2899553573 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900b4070 con 0x5596c6f0da20 2024-04-24T08:23:33.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.661+0000 7f5086a00640 1 -- 172.21.15.146:0/2899553573 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50900a4c90 con 0x5596c6f0da20 2024-04-24T08:23:33.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.661+0000 7f5086a00640 1 -- 172.21.15.146:0/2899553573 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900a4a80 con 0x5596c6f0da20 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.662+0000 7f509fe29740 1 -- 172.21.15.146:0/2899553573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 msgr2=0x5596c6f25730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.662+0000 7f509fe29740 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f25730 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f50900cd7d0 tx=0x7f509000cf10 comp rx=0 tx=0).stop 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.662+0000 7f509fe29740 1 -- 172.21.15.146:0/2899553573 shutdown_connections 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.662+0000 7f509fe29740 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f25730 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.662+0000 7f509fe29740 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f09200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.662+0000 7f509fe29740 1 --2- 172.21.15.146:0/2899553573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f19740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.662+0000 7f509fe29740 1 -- 172.21.15.146:0/2899553573 >> 172.21.15.146:0/2899553573 conn(0x5596c6efed50 msgr2=0x5596c6efdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.663+0000 7f509fe29740 1 -- 172.21.15.146:0/2899553573 shutdown_connections 2024-04-24T08:23:33.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.663+0000 7f509fe29740 1 -- 172.21.15.146:0/2899553573 wait complete. 2024-04-24T08:23:33.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.663+0000 7f509fe29740 1 -- start start 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 1 Processor -- start 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f16130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f07370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f07910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5098072570 con 0x5596c6f0da20 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50980739e0 con 0x5596c6f03350 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5098072c40 con 0x5596c6f02a30 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f509fe29740 10 client.0 _ll_register_callbacks cb 0x7f509009bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f5095e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f16130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f5095400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f07370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.664+0000 7f5095e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f16130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46602/0 (socket says 172.21.15.146:46602) 2024-04-24T08:23:33.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5096800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f07910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5095e00640 1 -- 172.21.15.146:0/3181730741 learned_addr learned my addr 172.21.15.146:0/3181730741 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5095e00640 1 -- 172.21.15.146:0/3181730741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 msgr2=0x5596c6f07370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5095e00640 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f07370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5095e00640 1 -- 172.21.15.146:0/3181730741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 msgr2=0x5596c6f07910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5095e00640 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f07910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5095e00640 1 -- 172.21.15.146:0/3181730741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596c6b7a3f0 con 0x5596c6f02a30 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f5095e00640 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f16130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5088005d90 tx=0x7f5088010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ff4c6861e85481 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:23:33.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f507be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.665+0000 7f507be00640 1 -- 172.21.15.146:0/3181730741 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088020070 con 0x5596c6f02a30 2024-04-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50900a4c90 con 0x5596c6f02a30 2024-04-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f507be00640 1 -- 172.21.15.146:0/3181730741 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5088004250 con 0x5596c6f02a30 2024-04-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f507be00640 1 -- 172.21.15.146:0/3181730741 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f508802c050 con 0x5596c6f02a30 2024-04-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 10 client.25882 fetch_fsmap learned FSMap version 25 2024-04-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596c6b3a0f0 con 0x5596c6f02a30 2024-04-24T08:23:33.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f507be00640 1 -- 172.21.15.146:0/3181730741 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50900a4c90 con 0x5596c6f02a30 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f507be00640 1 -- 172.21.15.146:0/3181730741 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f5088003990 con 0x5596c6f02a30 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 10 client.25882 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 10 client.25882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 20 client.25882 populate_metadata read hostname 'smithi146' 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f844e0 0x5596c6f86900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 4 client.25882 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.666+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5596c6e6fa80 con 0x5596c6f844e0 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.667+0000 7f5095400640 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f844e0 0x5596c6f86900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.667+0000 7f507be00640 10 client.25882 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.667+0000 7f5095400640 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f844e0 0x5596c6f86900 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=470faf599144cbdc server_cookie=5de8ae14b65877ff in_seq=0 out_seq=0 2024-04-24T08:23:33.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f507be00640 1 -- 172.21.15.146:0/3181730741 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5596c6e6fa80 con 0x5596c6f844e0 2024-04-24T08:23:33.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f507be00640 10 client.25882 handle_command_reply: tid=0 2024-04-24T08:23:33.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 client.25882 shutdown 2024-04-24T08:23:33.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 msgr2=0x5596c6f16130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f16130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5088005d90 tx=0x7f5088010800 comp rx=0 tx=0).stop 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 shutdown_connections 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5596c6f844e0 0x5596c6f86900 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5596c6f0da20 0x5596c6f07910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5596c6f03350 0x5596c6f07370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 --2- 172.21.15.146:0/3181730741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5596c6f02a30 0x5596c6f16130 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 >> 172.21.15.146:0/3181730741 conn(0x5596c6efed50 msgr2=0x5596c6f0b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.669+0000 7f507be00640 0 client.25882 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.670+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 shutdown_connections 2024-04-24T08:23:33.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.670+0000 7f509fe29740 1 -- 172.21.15.146:0/3181730741 wait complete. 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "scrub active (172 inodes in the stack)", 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: "bf469489-e50d-44c3-b271-2c783de6a32b": { 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "bf469489-e50d-44c3-b271-2c783de6a32b", 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:23:33.673 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.670+0000 7f509fe29740 20 client.25882 trim_cache size 0 max 16384 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f507c0a3da0 msgr2=0x7f507c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f507c0a3da0 0x7f507c0a6260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5080005ab0 tx=0x7f5080005a40 comp rx=0 tx=0).stop 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509810c480 msgr2=0x7f50981abef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509810c480 0x7f50981abef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f509000be20 tx=0x7f509000bf10 comp rx=0 tx=0).stop 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 shutdown_connections 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f507c0a3da0 0x7f507c0a6260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098103980 0x7f50981a7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f509810c480 0x7f50981abef0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 --2- 172.21.15.146:0/1771054264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5098103060 0x7f50981a7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 >> 172.21.15.146:0/1771054264 conn(0x7f50980fc880 msgr2=0x7f50980fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 shutdown_connections 2024-04-24T08:23:33.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:23:33.671+0000 7f5097200640 1 -- 172.21.15.146:0/1771054264 wait complete. 2024-04-24T08:23:33.691 INFO:tasks.cephfs.filesystem:scrub status for tag:bf469489-e50d-44c3-b271-2c783de6a32b - {'path': '/', 'tag': 'bf469489-e50d-44c3-b271-2c783de6a32b', 'options': 'recursive,force'} 2024-04-24T08:23:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 322 op/s 2024-04-24T08:23:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:23:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:23:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:23:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:23:34.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:34 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 322 op/s 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:23:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:34 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 322 op/s 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:23:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:34 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:35.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:35 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:35.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:35 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:35 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:23:36.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:36 smithi153 ceph-mon[32746]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.6 MiB/s wr, 301 op/s 2024-04-24T08:23:36.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:36 smithi146 ceph-mon[27805]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.6 MiB/s wr, 301 op/s 2024-04-24T08:23:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:36 smithi177 ceph-mon[33046]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 1.6 MiB/s wr, 301 op/s 2024-04-24T08:23:38.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:38 smithi153 ceph-mon[32746]: pgmap v76: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 360 op/s 2024-04-24T08:23:38.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:38 smithi146 ceph-mon[27805]: pgmap v76: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 360 op/s 2024-04-24T08:23:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:38 smithi177 ceph-mon[33046]: pgmap v76: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 360 op/s 2024-04-24T08:23:40.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:40 smithi153 ceph-mon[32746]: pgmap v77: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-04-24T08:23:40.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:40 smithi146 ceph-mon[27805]: pgmap v77: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-04-24T08:23:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:40 smithi177 ceph-mon[33046]: pgmap v77: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-04-24T08:23:42.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:42 smithi153 ceph-mon[32746]: pgmap v78: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-04-24T08:23:42.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:42 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:23:42.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:42 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:23:42.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:42 smithi146 ceph-mon[27805]: pgmap v78: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-04-24T08:23:42.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:42 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:23:42.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:42 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:23:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:42 smithi177 ceph-mon[33046]: pgmap v78: 97 pgs: 97 active+clean; 43 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 232 op/s 2024-04-24T08:23:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:42 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:23:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:42 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:23:44.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:44 smithi153 ceph-mon[32746]: pgmap v79: 97 pgs: 97 active+clean; 46 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 304 op/s 2024-04-24T08:23:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:44 smithi146 ceph-mon[27805]: pgmap v79: 97 pgs: 97 active+clean; 46 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 304 op/s 2024-04-24T08:23:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:44 smithi177 ceph-mon[33046]: pgmap v79: 97 pgs: 97 active+clean; 46 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 304 op/s 2024-04-24T08:23:46.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:46 smithi153 ceph-mon[32746]: pgmap v80: 97 pgs: 97 active+clean; 46 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-24T08:23:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:46 smithi146 ceph-mon[27805]: pgmap v80: 97 pgs: 97 active+clean; 46 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-24T08:23:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:46 smithi177 ceph-mon[33046]: pgmap v80: 97 pgs: 97 active+clean; 46 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-24T08:23:48.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:48 smithi153 ceph-mon[32746]: pgmap v81: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 169 op/s 2024-04-24T08:23:48.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:48 smithi146 ceph-mon[27805]: pgmap v81: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 169 op/s 2024-04-24T08:23:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:48 smithi177 ceph-mon[33046]: pgmap v81: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 169 op/s 2024-04-24T08:23:49.359 INFO:tasks.workunit.client.0.smithi146.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: git switch -c 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr:Or undo this operation with: 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: git switch - 2024-04-24T08:23:49.360 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:23:49.361 INFO:tasks.workunit.client.0.smithi146.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T08:23:49.361 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-04-24T08:23:49.361 INFO:tasks.workunit.client.0.smithi146.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-04-24T08:23:49.371 INFO:tasks.workunit.client.0.smithi146.stderr:+ make -j4 2024-04-24T08:23:49.395 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize -cfi 2024-04-24T08:23:50.218 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: putting auxiliary files in '.'. 2024-04-24T08:23:50.218 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file './config.guess' 2024-04-24T08:23:50.234 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file './config.sub' 2024-04-24T08:23:50.250 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file './install-sh' 2024-04-24T08:23:50.268 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file './ltmain.sh' 2024-04-24T08:23:50.348 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-04-24T08:23:50.348 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-04-24T08:23:50.368 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-04-24T08:23:50.388 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-04-24T08:23:50.408 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-04-24T08:23:50.426 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-04-24T08:23:50.449 INFO:tasks.workunit.client.0.smithi146.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-04-24T08:23:50.452 INFO:tasks.workunit.client.0.smithi146.stdout:cp include/install-sh . 2024-04-24T08:23:50.463 INFO:tasks.workunit.client.0.smithi146.stdout:aclocal -I m4 2024-04-24T08:23:50.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:50 smithi153 ceph-mon[32746]: pgmap v82: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 894 KiB/s wr, 110 op/s 2024-04-24T08:23:50.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:50 smithi146 ceph-mon[27805]: pgmap v82: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 894 KiB/s wr, 110 op/s 2024-04-24T08:23:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:50 smithi177 ceph-mon[33046]: pgmap v82: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 894 KiB/s wr, 110 op/s 2024-04-24T08:23:51.985 INFO:tasks.workunit.client.0.smithi146.stdout:autoheader 2024-04-24T08:23:52.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:52 smithi153 ceph-mon[32746]: pgmap v83: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 895 KiB/s wr, 110 op/s 2024-04-24T08:23:52.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:52 smithi146 ceph-mon[27805]: pgmap v83: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 895 KiB/s wr, 110 op/s 2024-04-24T08:23:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:52 smithi177 ceph-mon[33046]: pgmap v83: 97 pgs: 97 active+clean; 50 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 895 KiB/s wr, 110 op/s 2024-04-24T08:23:52.826 INFO:tasks.workunit.client.0.smithi146.stdout:autoconf 2024-04-24T08:23:53.088 INFO:tasks.workunit.client.0.smithi146.stdout:./configure \ 2024-04-24T08:23:53.089 INFO:tasks.workunit.client.0.smithi146.stdout: --libexecdir=/usr/lib \ 2024-04-24T08:23:53.089 INFO:tasks.workunit.client.0.smithi146.stdout: --exec_prefix=/var/lib 2024-04-24T08:23:53.431 INFO:tasks.workunit.client.0.smithi146.stdout:checking build system type... x86_64-pc-linux-gnu 2024-04-24T08:23:53.431 INFO:tasks.workunit.client.0.smithi146.stdout:checking host system type... x86_64-pc-linux-gnu 2024-04-24T08:23:53.436 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to print strings... printf 2024-04-24T08:23:53.436 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gcc... gcc 2024-04-24T08:23:53.619 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether the C compiler works... yes 2024-04-24T08:23:53.620 INFO:tasks.workunit.client.0.smithi146.stdout:checking for C compiler default output file name... a.out 2024-04-24T08:23:53.700 INFO:tasks.workunit.client.0.smithi146.stdout:checking for suffix of executables... 2024-04-24T08:23:53.789 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether we are cross compiling... no 2024-04-24T08:23:53.864 INFO:tasks.workunit.client.0.smithi146.stdout:checking for suffix of object files... o 2024-04-24T08:23:53.927 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether we are using the GNU C compiler... yes 2024-04-24T08:23:53.984 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether gcc accepts -g... yes 2024-04-24T08:23:54.123 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-24T08:23:54.137 INFO:tasks.workunit.client.0.smithi146.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-04-24T08:23:54.148 INFO:tasks.workunit.client.0.smithi146.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-24T08:23:54.154 INFO:tasks.workunit.client.0.smithi146.stdout:checking for egrep... /usr/bin/grep -E 2024-04-24T08:23:54.159 INFO:tasks.workunit.client.0.smithi146.stdout:checking for fgrep... /usr/bin/grep -F 2024-04-24T08:23:54.174 INFO:tasks.workunit.client.0.smithi146.stdout:checking for ld used by gcc... /usr/bin/ld 2024-04-24T08:23:54.180 INFO:tasks.workunit.client.0.smithi146.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-04-24T08:23:54.195 INFO:tasks.workunit.client.0.smithi146.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-04-24T08:23:54.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:54 smithi153 ceph-mon[32746]: pgmap v84: 97 pgs: 97 active+clean; 58 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 352 op/s 2024-04-24T08:23:54.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:54 smithi146 ceph-mon[27805]: pgmap v84: 97 pgs: 97 active+clean; 58 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 352 op/s 2024-04-24T08:23:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:54 smithi177 ceph-mon[33046]: pgmap v84: 97 pgs: 97 active+clean; 58 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 352 op/s 2024-04-24T08:23:55.281 INFO:tasks.workunit.client.0.smithi146.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-04-24T08:23:55.281 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether ln -s works... yes 2024-04-24T08:23:55.311 INFO:tasks.workunit.client.0.smithi146.stdout:checking the maximum length of command line arguments... 1572864 2024-04-24T08:23:55.323 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-04-24T08:23:55.323 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-04-24T08:23:55.324 INFO:tasks.workunit.client.0.smithi146.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-04-24T08:23:55.326 INFO:tasks.workunit.client.0.smithi146.stdout:checking for objdump... objdump 2024-04-24T08:23:55.327 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to recognize dependent libraries... pass_all 2024-04-24T08:23:55.329 INFO:tasks.workunit.client.0.smithi146.stdout:checking for dlltool... no 2024-04-24T08:23:55.330 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-04-24T08:23:55.331 INFO:tasks.workunit.client.0.smithi146.stdout:checking for ar... ar 2024-04-24T08:23:55.572 INFO:tasks.workunit.client.0.smithi146.stdout:checking for archiver @FILE support... @ 2024-04-24T08:23:55.573 INFO:tasks.workunit.client.0.smithi146.stdout:checking for strip... strip 2024-04-24T08:23:55.575 INFO:tasks.workunit.client.0.smithi146.stdout:checking for ranlib... ranlib 2024-04-24T08:23:55.576 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gawk... gawk 2024-04-24T08:23:55.799 INFO:tasks.workunit.client.0.smithi146.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-04-24T08:23:55.806 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sysroot... no 2024-04-24T08:23:55.830 INFO:tasks.workunit.client.0.smithi146.stdout:checking for a working dd... /usr/bin/dd 2024-04-24T08:23:55.853 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-04-24T08:23:55.924 INFO:tasks.workunit.client.0.smithi146.stdout:checking for mt... no 2024-04-24T08:23:55.948 INFO:tasks.workunit.client.0.smithi146.stdout:checking if : is a manifest tool... no 2024-04-24T08:23:56.115 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to run the C preprocessor... gcc -E 2024-04-24T08:23:56.531 INFO:tasks.workunit.client.0.smithi146.stdout:checking for ANSI C header files... yes 2024-04-24T08:23:56.609 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/types.h... yes 2024-04-24T08:23:56.705 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/stat.h... yes 2024-04-24T08:23:56.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:56 smithi153 ceph-mon[32746]: pgmap v85: 97 pgs: 97 active+clean; 58 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.2 MiB/s wr, 280 op/s 2024-04-24T08:23:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:56 smithi146 ceph-mon[27805]: pgmap v85: 97 pgs: 97 active+clean; 58 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.2 MiB/s wr, 280 op/s 2024-04-24T08:23:56.795 INFO:tasks.workunit.client.0.smithi146.stdout:checking for stdlib.h... yes 2024-04-24T08:23:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:56 smithi177 ceph-mon[33046]: pgmap v85: 97 pgs: 97 active+clean; 58 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.2 MiB/s wr, 280 op/s 2024-04-24T08:23:56.892 INFO:tasks.workunit.client.0.smithi146.stdout:checking for string.h... yes 2024-04-24T08:23:57.000 INFO:tasks.workunit.client.0.smithi146.stdout:checking for memory.h... yes 2024-04-24T08:23:57.095 INFO:tasks.workunit.client.0.smithi146.stdout:checking for strings.h... yes 2024-04-24T08:23:57.188 INFO:tasks.workunit.client.0.smithi146.stdout:checking for inttypes.h... yes 2024-04-24T08:23:57.287 INFO:tasks.workunit.client.0.smithi146.stdout:checking for stdint.h... yes 2024-04-24T08:23:57.384 INFO:tasks.workunit.client.0.smithi146.stdout:checking for unistd.h... yes 2024-04-24T08:23:57.474 INFO:tasks.workunit.client.0.smithi146.stdout:checking for dlfcn.h... yes 2024-04-24T08:23:57.489 INFO:tasks.workunit.client.0.smithi146.stdout:checking for objdir... .libs 2024-04-24T08:23:57.725 INFO:tasks.workunit.client.0.smithi146.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-04-24T08:23:57.725 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-04-24T08:23:57.812 INFO:tasks.workunit.client.0.smithi146.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-04-24T08:23:57.891 INFO:tasks.workunit.client.0.smithi146.stdout:checking if gcc static flag -static works... no 2024-04-24T08:23:58.022 INFO:tasks.workunit.client.0.smithi146.stdout:checking if gcc supports -c -o file.o... yes 2024-04-24T08:23:58.022 INFO:tasks.workunit.client.0.smithi146.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-04-24T08:23:58.048 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-04-24T08:23:58.146 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether -lc should be explicitly linked in... no 2024-04-24T08:23:58.305 INFO:tasks.workunit.client.0.smithi146.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-04-24T08:23:58.305 INFO:tasks.workunit.client.0.smithi146.stdout:checking how to hardcode library paths into programs... immediate 2024-04-24T08:23:58.309 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether stripping libraries is possible... yes 2024-04-24T08:23:58.310 INFO:tasks.workunit.client.0.smithi146.stdout:checking if libtool supports shared libraries... yes 2024-04-24T08:23:58.310 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether to build shared libraries... yes 2024-04-24T08:23:58.310 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether to build static libraries... yes 2024-04-24T08:23:58.330 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gcc... (cached) gcc 2024-04-24T08:23:58.412 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-24T08:23:58.412 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-24T08:23:58.413 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-24T08:23:58.414 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gmake... /usr/bin/gmake 2024-04-24T08:23:58.415 INFO:tasks.workunit.client.0.smithi146.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-04-24T08:23:58.415 INFO:tasks.workunit.client.0.smithi146.stdout:checking for tar... /bin/tar 2024-04-24T08:23:58.416 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gzip... /bin/gzip 2024-04-24T08:23:58.470 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether gcc -MM is supported... yes 2024-04-24T08:23:58.471 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sort... /bin/sort 2024-04-24T08:23:58.471 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether ln -s works... yes 2024-04-24T08:23:58.472 INFO:tasks.workunit.client.0.smithi146.stdout:checking for rpm... /bin/rpm 2024-04-24T08:23:58.483 INFO:tasks.workunit.client.0.smithi146.stdout:checking for rpmbuild... no 2024-04-24T08:23:58.483 INFO:tasks.workunit.client.0.smithi146.stdout:checking for ANSI C header files... (cached) yes 2024-04-24T08:23:58.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:23:58 smithi153 ceph-mon[32746]: pgmap v86: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 393 op/s 2024-04-24T08:23:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:23:58 smithi146 ceph-mon[27805]: pgmap v86: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 393 op/s 2024-04-24T08:23:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:23:58 smithi177 ceph-mon[33046]: pgmap v86: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 393 op/s 2024-04-24T08:23:59.675 INFO:tasks.workunit.client.0.smithi146.stdout:checking assert.h usability... yes 2024-04-24T08:23:59.716 INFO:tasks.workunit.client.0.smithi146.stdout:checking assert.h presence... yes 2024-04-24T08:23:59.716 INFO:tasks.workunit.client.0.smithi146.stdout:checking for assert.h... yes 2024-04-24T08:23:59.834 INFO:tasks.workunit.client.0.smithi146.stdout:checking bstring.h usability... no 2024-04-24T08:23:59.899 INFO:tasks.workunit.client.0.smithi146.stdout:checking bstring.h presence... no 2024-04-24T08:23:59.899 INFO:tasks.workunit.client.0.smithi146.stdout:checking for bstring.h... no 2024-04-24T08:23:59.989 INFO:tasks.workunit.client.0.smithi146.stdout:checking libgen.h usability... yes 2024-04-24T08:24:00.029 INFO:tasks.workunit.client.0.smithi146.stdout:checking libgen.h presence... yes 2024-04-24T08:24:00.029 INFO:tasks.workunit.client.0.smithi146.stdout:checking for libgen.h... yes 2024-04-24T08:24:00.134 INFO:tasks.workunit.client.0.smithi146.stdout:checking dirent.h usability... yes 2024-04-24T08:24:00.174 INFO:tasks.workunit.client.0.smithi146.stdout:checking dirent.h presence... yes 2024-04-24T08:24:00.174 INFO:tasks.workunit.client.0.smithi146.stdout:checking for dirent.h... yes 2024-04-24T08:24:00.282 INFO:tasks.workunit.client.0.smithi146.stdout:checking errno.h usability... yes 2024-04-24T08:24:00.323 INFO:tasks.workunit.client.0.smithi146.stdout:checking errno.h presence... yes 2024-04-24T08:24:00.323 INFO:tasks.workunit.client.0.smithi146.stdout:checking for errno.h... yes 2024-04-24T08:24:00.427 INFO:tasks.workunit.client.0.smithi146.stdout:checking malloc.h usability... yes 2024-04-24T08:24:00.475 INFO:tasks.workunit.client.0.smithi146.stdout:checking malloc.h presence... yes 2024-04-24T08:24:00.475 INFO:tasks.workunit.client.0.smithi146.stdout:checking for malloc.h... yes 2024-04-24T08:24:00.606 INFO:tasks.workunit.client.0.smithi146.stdout:checking uuid.h usability... no 2024-04-24T08:24:00.672 INFO:tasks.workunit.client.0.smithi146.stdout:checking uuid.h presence... no 2024-04-24T08:24:00.672 INFO:tasks.workunit.client.0.smithi146.stdout:checking for uuid.h... no 2024-04-24T08:24:00.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:00 smithi153 ceph-mon[32746]: pgmap v87: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-04-24T08:24:00.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:00 smithi146 ceph-mon[27805]: pgmap v87: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-04-24T08:24:00.772 INFO:tasks.workunit.client.0.smithi146.stdout:checking uuid/uuid.h usability... yes 2024-04-24T08:24:00.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:00 smithi177 ceph-mon[33046]: pgmap v87: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-04-24T08:24:00.837 INFO:tasks.workunit.client.0.smithi146.stdout:checking uuid/uuid.h presence... yes 2024-04-24T08:24:00.837 INFO:tasks.workunit.client.0.smithi146.stdout:checking for uuid/uuid.h... yes 2024-04-24T08:24:00.937 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/uuid.h usability... no 2024-04-24T08:24:00.991 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/uuid.h presence... no 2024-04-24T08:24:00.991 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/uuid.h... no 2024-04-24T08:24:01.084 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/file.h usability... yes 2024-04-24T08:24:01.124 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/file.h presence... yes 2024-04-24T08:24:01.124 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/file.h... yes 2024-04-24T08:24:01.237 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/fcntl.h usability... yes 2024-04-24T08:24:01.276 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/fcntl.h presence... yes 2024-04-24T08:24:01.276 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/fcntl.h... yes 2024-04-24T08:24:01.381 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/syssgi.h usability... no 2024-04-24T08:24:01.439 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/syssgi.h presence... no 2024-04-24T08:24:01.439 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/syssgi.h... no 2024-04-24T08:24:01.576 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/param.h usability... yes 2024-04-24T08:24:01.622 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/param.h presence... yes 2024-04-24T08:24:01.623 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/param.h... yes 2024-04-24T08:24:01.654 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/stat.h... (cached) yes 2024-04-24T08:24:01.740 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/statvfs.h usability... yes 2024-04-24T08:24:01.777 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/statvfs.h presence... yes 2024-04-24T08:24:01.777 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/statvfs.h... yes 2024-04-24T08:24:01.880 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/time.h usability... yes 2024-04-24T08:24:01.920 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/time.h presence... yes 2024-04-24T08:24:01.920 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/time.h... yes 2024-04-24T08:24:02.023 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/ioctl.h usability... yes 2024-04-24T08:24:02.060 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/ioctl.h presence... yes 2024-04-24T08:24:02.060 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/ioctl.h... yes 2024-04-24T08:24:02.171 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/wait.h usability... yes 2024-04-24T08:24:02.214 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/wait.h presence... yes 2024-04-24T08:24:02.215 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/wait.h... yes 2024-04-24T08:24:02.241 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/types.h... (cached) yes 2024-04-24T08:24:02.258 INFO:tasks.workunit.client.0.smithi146.stdout:checking for strings.h... (cached) yes 2024-04-24T08:24:02.342 INFO:tasks.workunit.client.0.smithi146.stdout:checking err.h usability... yes 2024-04-24T08:24:02.379 INFO:tasks.workunit.client.0.smithi146.stdout:checking err.h presence... yes 2024-04-24T08:24:02.379 INFO:tasks.workunit.client.0.smithi146.stdout:checking for err.h... yes 2024-04-24T08:24:02.483 INFO:tasks.workunit.client.0.smithi146.stdout:checking linux/falloc.h usability... yes 2024-04-24T08:24:02.521 INFO:tasks.workunit.client.0.smithi146.stdout:checking linux/falloc.h presence... yes 2024-04-24T08:24:02.521 INFO:tasks.workunit.client.0.smithi146.stdout:checking for linux/falloc.h... yes 2024-04-24T08:24:02.637 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-04-24T08:24:02.699 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-04-24T08:24:02.700 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/fs/xfs_fsops.h... no 2024-04-24T08:24:02.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:02 smithi146 ceph-mon[27805]: pgmap v88: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-04-24T08:24:02.796 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/fs/xfs_itable.h usability... no 2024-04-24T08:24:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:02 smithi177 ceph-mon[33046]: pgmap v88: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-04-24T08:24:02.856 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/fs/xfs_itable.h presence... no 2024-04-24T08:24:02.856 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/fs/xfs_itable.h... no 2024-04-24T08:24:02.952 INFO:tasks.workunit.client.0.smithi146.stdout:checking xfs/platform_defs.h usability... no 2024-04-24T08:24:02.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:02 smithi153 ceph-mon[32746]: pgmap v88: 97 pgs: 97 active+clean; 63 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.4 MiB/s wr, 355 op/s 2024-04-24T08:24:03.013 INFO:tasks.workunit.client.0.smithi146.stdout:checking xfs/platform_defs.h presence... no 2024-04-24T08:24:03.013 INFO:tasks.workunit.client.0.smithi146.stdout:checking for xfs/platform_defs.h... no 2024-04-24T08:24:03.107 INFO:tasks.workunit.client.0.smithi146.stdout:checking btrfs/ioctl.h usability... no 2024-04-24T08:24:03.173 INFO:tasks.workunit.client.0.smithi146.stdout:checking btrfs/ioctl.h presence... no 2024-04-24T08:24:03.173 INFO:tasks.workunit.client.0.smithi146.stdout:checking for btrfs/ioctl.h... no 2024-04-24T08:24:03.268 INFO:tasks.workunit.client.0.smithi146.stdout:checking cifs/ioctl.h usability... no 2024-04-24T08:24:03.331 INFO:tasks.workunit.client.0.smithi146.stdout:checking cifs/ioctl.h presence... no 2024-04-24T08:24:03.332 INFO:tasks.workunit.client.0.smithi146.stdout:checking for cifs/ioctl.h... no 2024-04-24T08:24:03.431 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/mman.h usability... yes 2024-04-24T08:24:03.470 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/mman.h presence... yes 2024-04-24T08:24:03.470 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/mman.h... yes 2024-04-24T08:24:03.550 INFO:tasks.workunit.client.0.smithi146.stdout:checking for xfs/xfs_log_format.h... no 2024-04-24T08:24:03.669 INFO:tasks.workunit.client.0.smithi146.stdout:checking for uuid_compare... no 2024-04-24T08:24:03.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:24:03.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:24:03.753 INFO:tasks.workunit.client.0.smithi146.stdout:checking for uuid_compare in -luuid... yes 2024-04-24T08:24:03.853 INFO:tasks.workunit.client.0.smithi146.stdout:checking for xfs/xfs.h... yes 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 -- 172.21.15.146:0/1967712862 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62641044b0 msgr2=0x7f6264104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 --2- 172.21.15.146:0/1967712862 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62641044b0 0x7f6264104890 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6254009900 tx=0x7f625402a400 comp rx=0 tx=0).stop 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 -- 172.21.15.146:0/1967712862 shutdown_connections 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 --2- 172.21.15.146:0/1967712862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6264104e60 0x7f6264108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 --2- 172.21.15.146:0/1967712862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264109580 0x7f6264111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 --2- 172.21.15.146:0/1967712862 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62641044b0 0x7f6264104890 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 -- 172.21.15.146:0/1967712862 >> 172.21.15.146:0/1967712862 conn(0x7f62640ffd90 msgr2=0x7f62641021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 -- 172.21.15.146:0/1967712862 shutdown_connections 2024-04-24T08:24:03.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.879+0000 7f6269400640 1 -- 172.21.15.146:0/1967712862 wait complete. 2024-04-24T08:24:03.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.880+0000 7f6269400640 1 Processor -- start 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.880+0000 7f6269400640 1 -- start start 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6269400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62641044b0 0x7f62641a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6269400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264109580 0x7f62641a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6269400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626419a9c0 0x7f626419ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6269400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6264114680 con 0x7f626419a9c0 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6269400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6264114500 con 0x7f6264109580 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6269400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6264114800 con 0x7f62641044b0 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626419a9c0 0x7f626419ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6261e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264109580 0x7f62641a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626419a9c0 0x7f626419ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33338/0 (socket says 172.21.15.146:33338) 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 -- 172.21.15.146:0/786093883 learned_addr learned my addr 172.21.15.146:0/786093883 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:03.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6262800640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62641044b0 0x7f62641a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 -- 172.21.15.146:0/786093883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62641044b0 msgr2=0x7f62641a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62641044b0 0x7f62641a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 -- 172.21.15.146:0/786093883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264109580 msgr2=0x7f62641a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264109580 0x7f62641a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6263200640 1 -- 172.21.15.146:0/786093883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626419b4e0 con 0x7f626419a9c0 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.881+0000 7f6261e00640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264109580 0x7f62641a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.882+0000 7f6263200640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626419a9c0 0x7f626419ae70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f625c00eb10 tx=0x7f625c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.882+0000 7f625b400640 1 -- 172.21.15.146:0/786093883 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c004b20 con 0x7f626419a9c0 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.882+0000 7f625b400640 1 -- 172.21.15.146:0/786093883 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f625c004cc0 con 0x7f626419a9c0 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.882+0000 7f625b400640 1 -- 172.21.15.146:0/786093883 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c010760 con 0x7f626419a9c0 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.882+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626419b7d0 con 0x7f626419a9c0 2024-04-24T08:24:03.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.882+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62641a79d0 con 0x7f626419a9c0 2024-04-24T08:24:03.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.886+0000 7f626c1b1740 1 Processor -- start 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f625b400640 1 -- 172.21.15.146:0/786093883 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f625c005020 con 0x7f626419a9c0 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f626c1b1740 1 -- start start 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349d860 0x55a59349dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349e180 0x55a5934a8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f626c1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55a5934031f0 con 0x55a59349e180 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f626c1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a5933a9350 con 0x55a59349d860 2024-04-24T08:24:03.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.887+0000 7f626c1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5933ff1b0 con 0x55a5934a8850 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349d860 0x55a59349dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6262800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349e180 0x55a5934a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349d860 0x55a59349dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39294/0 (socket says 172.21.15.146:39294) 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 -- 172.21.15.146:0/3354065194 learned_addr learned my addr 172.21.15.146:0/3354065194 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 -- 172.21.15.146:0/3354065194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 msgr2=0x55a5934aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 -- 172.21.15.146:0/3354065194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349e180 msgr2=0x55a5934a8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349e180 0x55a5934a8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f6261e00640 1 -- 172.21.15.146:0/3354065194 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5933fdf50 con 0x55a59349d860 2024-04-24T08:24:03.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.888+0000 7f625b400640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f62480a3d20 0x7f62480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.889+0000 7f6261e00640 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349d860 0x55a59349dc40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6250011b60 tx=0x7f6250005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=272ff71cdf51af3b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:03.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.889+0000 7f625b400640 1 -- 172.21.15.146:0/786093883 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f625c0d0d50 con 0x7f626419a9c0 2024-04-24T08:24:03.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.889+0000 7f6262800640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f62480a3d20 0x7f62480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.889+0000 7f624f400640 1 -- 172.21.15.146:0/3354065194 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625000fc50 con 0x55a59349d860 2024-04-24T08:24:03.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.889+0000 7f624f400640 1 -- 172.21.15.146:0/3354065194 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f625000fdf0 con 0x55a59349d860 2024-04-24T08:24:03.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.889+0000 7f624f400640 1 -- 172.21.15.146:0/3354065194 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6250004a00 con 0x55a59349d860 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 -- 172.21.15.146:0/3354065194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349d860 msgr2=0x55a59349dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f6262800640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f62480a3d20 0x7f62480a61e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6254006010 tx=0x7f6254000bc0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349d860 0x55a59349dc40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6250011b60 tx=0x7f6250005850 comp rx=0 tx=0).stop 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 -- 172.21.15.146:0/3354065194 shutdown_connections 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349e180 0x55a5934a8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 --2- 172.21.15.146:0/3354065194 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349d860 0x55a59349dc40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 -- 172.21.15.146:0/3354065194 >> 172.21.15.146:0/3354065194 conn(0x55a593499b80 msgr2=0x55a593499f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 -- 172.21.15.146:0/3354065194 shutdown_connections 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 -- 172.21.15.146:0/3354065194 wait complete. 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.890+0000 7f626c1b1740 1 -- start start 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 1 Processor -- start 2024-04-24T08:24:03.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349d860 0x55a5934a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55a5934ab640 con 0x55a59349d860 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a5933bc690 con 0x55a59349e180 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5934ab7c0 con 0x55a5934a8850 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.891+0000 7f626c1b1740 10 client.0 _ll_register_callbacks cb 0x55a5933f8650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6262800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349d860 0x55a5934a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39298/0 (socket says 172.21.15.146:39298) 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 -- 172.21.15.146:0/1091464499 learned_addr learned my addr 172.21.15.146:0/1091464499 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6263200640 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 -- 172.21.15.146:0/1091464499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 msgr2=0x55a5934a62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934a62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 -- 172.21.15.146:0/1091464499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349d860 msgr2=0x55a5934a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349d860 0x55a5934a5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 -- 172.21.15.146:0/1091464499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5933bb920 con 0x55a59349e180 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6261e00640 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a5d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6250005bc0 tx=0x7f62500096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c32dc54ed7358ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.892+0000 7f6243e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f6243e00640 1 -- 172.21.15.146:0/1091464499 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625000fc20 con 0x55a59349e180 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f6243e00640 1 -- 172.21.15.146:0/1091464499 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6250004240 con 0x55a59349e180 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f625000fdf0 con 0x55a59349e180 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f6243e00640 1 -- 172.21.15.146:0/1091464499 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6250004440 con 0x55a59349e180 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f626c1b1740 10 client.25910 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a5931df7d0 con 0x55a59349e180 2024-04-24T08:24:03.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f6243e00640 1 -- 172.21.15.146:0/1091464499 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f625000fdf0 con 0x55a59349e180 2024-04-24T08:24:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.893+0000 7f6243e00640 1 -- 172.21.15.146:0/1091464499 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6250026020 con 0x55a59349e180 2024-04-24T08:24:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f626c1b1740 10 client.25910 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f626c1b1740 10 client.25910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f626c1b1740 20 client.25910 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:03.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f626c1b1740 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934b8ac0 0x55a5934baeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f626c1b1740 4 client.25910 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a593410c10 con 0x55a5934b8ac0 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f6262800640 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934b8ac0 0x55a5934baeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f6262800640 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934b8ac0 0x55a5934baeb0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=151381f513244717 server_cookie=88994984ecf0a71b in_seq=0 out_seq=0 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.894+0000 7f6243e00640 10 client.25910 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.895+0000 7f6243e00640 1 -- 172.21.15.146:0/1091464499 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a593410c10 con 0x55a5934b8ac0 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.895+0000 7f6243e00640 10 client.25910 handle_command_reply: tid=0 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.895+0000 7f626c1b1740 1 client.25910 shutdown 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.895+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 msgr2=0x55a5934a5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.895+0000 7f626c1b1740 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a5d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6250005bc0 tx=0x7f62500096a0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 shutdown_connections 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934b8ac0 0x55a5934baeb0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5934a8850 0x55a5934a62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a5d80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f6243e00640 0 client.25910 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:03.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 --2- 172.21.15.146:0/1091464499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a59349d860 0x55a5934a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 >> 172.21.15.146:0/1091464499 conn(0x55a593499b80 msgr2=0x55a59349f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:03.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 shutdown_connections 2024-04-24T08:24:03.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 1 -- 172.21.15.146:0/1091464499 wait complete. 2024-04-24T08:24:03.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.896+0000 7f626c1b1740 20 client.25910 trim_cache size 0 max 16384 2024-04-24T08:24:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.915+0000 7f626c1b1740 1 Processor -- start 2024-04-24T08:24:03.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.915+0000 7f626c1b1740 1 -- start start 2024-04-24T08:24:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.915+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.915+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934c0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.915+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934c0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.915+0000 7f626c1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f62641b2860 con 0x55a593499b80 2024-04-24T08:24:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.915+0000 7f626c1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62641b0130 con 0x55a59349e180 2024-04-24T08:24:03.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f626c1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62641adf80 con 0x55a59349d860 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6263200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934c0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6262800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6261e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934c0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6263200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934c0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39304/0 (socket says 172.21.15.146:39304) 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6262800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33374/0 (socket says 172.21.15.146:33374) 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6263200640 1 -- 172.21.15.146:0/2452637353 learned_addr learned my addr 172.21.15.146:0/2452637353 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6261e00640 1 -- 172.21.15.146:0/2452637353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 msgr2=0x55a5934c0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6261e00640 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934c0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6261e00640 1 -- 172.21.15.146:0/2452637353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 msgr2=0x55a5934b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6261e00640 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6261e00640 1 -- 172.21.15.146:0/2452637353 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5931153f0 con 0x55a59349d860 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.916+0000 7f6263200640 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934c0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:03.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.917+0000 7f6261e00640 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934c0660 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55a5934a6e10 tx=0x7f62500043d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e27cbb1fcbd2f00f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.917+0000 7f624ea00640 1 -- 172.21.15.146:0/2452637353 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62500056a0 con 0x55a59349d860 2024-04-24T08:24:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.917+0000 7f624ea00640 1 -- 172.21.15.146:0/2452637353 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62500035f0 con 0x55a59349d860 2024-04-24T08:24:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.917+0000 7f624ea00640 1 -- 172.21.15.146:0/2452637353 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62500045e0 con 0x55a59349d860 2024-04-24T08:24:03.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 -- 172.21.15.146:0/2452637353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 msgr2=0x55a5934c0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934c0660 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55a5934a6e10 tx=0x7f62500043d0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 -- 172.21.15.146:0/2452637353 shutdown_connections 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934c0660 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934c0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 --2- 172.21.15.146:0/2452637353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 -- 172.21.15.146:0/2452637353 >> 172.21.15.146:0/2452637353 conn(0x55a5934bb220 msgr2=0x55a5934a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 -- 172.21.15.146:0/2452637353 shutdown_connections 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 -- 172.21.15.146:0/2452637353 wait complete. 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 -- start start 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.918+0000 7f626c1b1740 1 Processor -- start 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f626c1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f626c1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f62641ae5f0 con 0x55a593499b80 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f626c1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f62641afa60 con 0x55a59349e180 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f626c1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62641aecc0 con 0x55a59349d860 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f626c1b1740 10 client.0 _ll_register_callbacks cb 0x7f6250005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f6261e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f6261e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934b0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54218/0 (socket says 172.21.15.146:54218) 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f6261e00640 1 -- 172.21.15.146:0/159606314 learned_addr learned my addr 172.21.15.146:0/159606314 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f6262800640 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f6263200640 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f6261e00640 1 -- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 msgr2=0x55a5934a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.919+0000 7f6261e00640 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6261e00640 1 -- 172.21.15.146:0/159606314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 msgr2=0x55a5934b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6261e00640 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6261e00640 1 -- 172.21.15.146:0/159606314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5930d50f0 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6261e00640 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934b0f50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62500116d0 tx=0x7f6250011a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94520cf0dc5b3ff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6243e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6243e00640 1 -- 172.21.15.146:0/159606314 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62500049f0 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62500035f0 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6243e00640 1 -- 172.21.15.146:0/159606314 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6250004b90 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f6243e00640 1 -- 172.21.15.146:0/159606314 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6250049050 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f626c1b1740 10 client.25900 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.920+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a5933c8810 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f6243e00640 1 -- 172.21.15.146:0/159606314 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62500035f0 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f6243e00640 1 -- 172.21.15.146:0/159606314 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6250026030 con 0x55a59349d860 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f626c1b1740 10 client.25900 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f626c1b1740 10 client.25900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f626c1b1740 20 client.25900 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f626c1b1740 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934a8850 0x55a593520d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:03.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f626c1b1740 4 client.25900 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:24:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a59340aa80 con 0x55a5934a8850 2024-04-24T08:24:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.921+0000 7f6262800640 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934a8850 0x55a593520d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f6262800640 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934a8850 0x55a593520d90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc50bbdcbf7b333d server_cookie=af218572c6fd2d13 in_seq=0 out_seq=0 2024-04-24T08:24:03.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f6243e00640 10 client.25900 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:03.926 INFO:tasks.workunit.client.0.smithi146.stdout:checking for xfs/libxfs.h... no 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f6243e00640 1 -- 172.21.15.146:0/159606314 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a59340aa80 con 0x55a5934a8850 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f6243e00640 10 client.25900 handle_command_reply: tid=0 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 client.25900 shutdown 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 msgr2=0x55a5934b0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934b0f50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62500116d0 tx=0x7f6250011a60 comp rx=0 tx=0).stop 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 shutdown_connections 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55a5934a8850 0x55a593520d90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a59349d860 0x55a5934b0f50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a59349e180 0x55a5934a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f6243e00640 0 client.25900 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 --2- 172.21.15.146:0/159606314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a593499b80 0x55a5934b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 >> 172.21.15.146:0/159606314 conn(0x55a5934bb220 msgr2=0x55a5934a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:03.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.922+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 shutdown_connections 2024-04-24T08:24:03.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f626c1b1740 1 -- 172.21.15.146:0/159606314 wait complete. 2024-04-24T08:24:03.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f626c1b1740 20 client.25900 trim_cache size 0 max 16384 2024-04-24T08:24:03.928 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:24:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:24:03.928 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:24:03.928 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f62480a3d20 msgr2=0x7f62480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f62480a3d20 0x7f62480a61e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6254006010 tx=0x7f6254000bc0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626419a9c0 msgr2=0x7f626419ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626419a9c0 0x7f626419ae70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f625c00eb10 tx=0x7f625c00c2d0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 shutdown_connections 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f62480a3d20 0x7f62480a61e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626419a9c0 0x7f626419ae70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6264109580 0x7f62641a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 --2- 172.21.15.146:0/786093883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62641044b0 0x7f62641a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 >> 172.21.15.146:0/786093883 conn(0x7f62640ffd90 msgr2=0x7f62641010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:03.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.923+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 shutdown_connections 2024-04-24T08:24:03.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:03.924+0000 7f6269400640 1 -- 172.21.15.146:0/786093883 wait complete. 2024-04-24T08:24:03.942 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:24:03.942 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:24:03.983 INFO:tasks.workunit.client.0.smithi146.stdout:checking whether xlog_assign_lsn is declared... no 2024-04-24T08:24:04.060 INFO:tasks.workunit.client.0.smithi146.stdout:checking for xfs/xqm.h... yes 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.122+0000 7f1672400640 1 -- 172.21.15.146:0/818018780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c104440 msgr2=0x7f166c104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.122+0000 7f1672400640 1 --2- 172.21.15.146:0/818018780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c104440 0x7f166c104820 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f165c009ca0 tx=0x7f165c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.123+0000 7f1672400640 1 -- 172.21.15.146:0/818018780 shutdown_connections 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.123+0000 7f1672400640 1 --2- 172.21.15.146:0/818018780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c104df0 0x7f166c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.123+0000 7f1672400640 1 --2- 172.21.15.146:0/818018780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f166c109510 0x7f166c1110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.123+0000 7f1672400640 1 --2- 172.21.15.146:0/818018780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c104440 0x7f166c104820 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.123+0000 7f1672400640 1 -- 172.21.15.146:0/818018780 >> 172.21.15.146:0/818018780 conn(0x7f166c0ffd00 msgr2=0x7f166c102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.123+0000 7f1672400640 1 -- 172.21.15.146:0/818018780 shutdown_connections 2024-04-24T08:24:04.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.123+0000 7f1672400640 1 -- 172.21.15.146:0/818018780 wait complete. 2024-04-24T08:24:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 Processor -- start 2024-04-24T08:24:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 -- start start 2024-04-24T08:24:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c104440 0x7f166c1a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c109510 0x7f166c1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f166c19a940 0x7f166c19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f166c1145a0 con 0x7f166c109510 2024-04-24T08:24:04.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f166c114420 con 0x7f166c19a940 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.124+0000 7f1672400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f166c114720 con 0x7f166c104440 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c109510 0x7f166c1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c109510 0x7f166c1a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33390/0 (socket says 172.21.15.146:33390) 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 -- 172.21.15.146:0/2818094171 learned_addr learned my addr 172.21.15.146:0/2818094171 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166be00640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f166c19a940 0x7f166c19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166b400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c104440 0x7f166c1a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 -- 172.21.15.146:0/2818094171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c104440 msgr2=0x7f166c1a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c104440 0x7f166c1a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 -- 172.21.15.146:0/2818094171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f166c19a940 msgr2=0x7f166c19adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f166c19a940 0x7f166c19adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 -- 172.21.15.146:0/2818094171 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c19b460 con 0x7f166c109510 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166be00640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f166c19a940 0x7f166c19adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.125+0000 7f166aa00640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c109510 0x7f166c1a0c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f165400e9c0 tx=0x7f165400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.126+0000 7f1663e00640 1 -- 172.21.15.146:0/2818094171 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165400cd80 con 0x7f166c109510 2024-04-24T08:24:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.126+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c19b750 con 0x7f166c109510 2024-04-24T08:24:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.126+0000 7f1663e00640 1 -- 172.21.15.146:0/2818094171 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1654005500 con 0x7f166c109510 2024-04-24T08:24:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.126+0000 7f1663e00640 1 -- 172.21.15.146:0/2818094171 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654010690 con 0x7f166c109510 2024-04-24T08:24:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.126+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c1a7950 con 0x7f166c109510 2024-04-24T08:24:04.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.127+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c19be20 con 0x7f166c109510 2024-04-24T08:24:04.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.129+0000 7f1663e00640 1 -- 172.21.15.146:0/2818094171 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f16540046c0 con 0x7f166c109510 2024-04-24T08:24:04.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.131+0000 7f1663e00640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f16500a3d20 0x7f16500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.131+0000 7f1663e00640 1 -- 172.21.15.146:0/2818094171 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f16540d0bc0 con 0x7f166c109510 2024-04-24T08:24:04.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.131+0000 7f166b400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f16500a3d20 0x7f16500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.132+0000 7f166b400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f16500a3d20 0x7f16500a61e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f166c1001a0 tx=0x7f165c009360 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.136+0000 7f1663e00640 1 -- 172.21.15.146:0/2818094171 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f165409d210 con 0x7f166c109510 2024-04-24T08:24:04.170 INFO:tasks.workunit.client.0.smithi146.stdout:checking xfsctl from xfs/xfs.h... ok 2024-04-24T08:24:04.241 INFO:tasks.workunit.client.0.smithi146.stdout:checking xfs/handle.h usability... yes 2024-04-24T08:24:04.276 INFO:tasks.workunit.client.0.smithi146.stdout:checking xfs/handle.h presence... yes 2024-04-24T08:24:04.276 INFO:tasks.workunit.client.0.smithi146.stdout:checking for xfs/handle.h... yes 2024-04-24T08:24:04.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.381+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f166c104df0 con 0x7f166c109510 2024-04-24T08:24:04.382 INFO:tasks.workunit.client.0.smithi146.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-04-24T08:24:04.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.382+0000 7f1663e00640 1 -- 172.21.15.146:0/2818094171 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11892 (secure 0 0 0) 0x7f165409cb10 con 0x7f166c109510 2024-04-24T08:24:04.383 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:24:04.383 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:23:07.211437+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24568,24569,24553]},"id":1}]} 2024-04-24T08:24:04.383 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 25 2024-04-24T08:24:04.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f16500a3d20 msgr2=0x7f16500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f16500a3d20 0x7f16500a61e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f166c1001a0 tx=0x7f165c009360 comp rx=0 tx=0).stop 2024-04-24T08:24:04.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c109510 msgr2=0x7f166c1a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c109510 0x7f166c1a0c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f165400e9c0 tx=0x7f165400ee80 comp rx=0 tx=0).stop 2024-04-24T08:24:04.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 shutdown_connections 2024-04-24T08:24:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f16500a3d20 0x7f16500a61e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f166c19a940 0x7f166c19adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f166c109510 0x7f166c1a0c80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 --2- 172.21.15.146:0/2818094171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f166c104440 0x7f166c1a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.384+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 >> 172.21.15.146:0/2818094171 conn(0x7f166c0ffd00 msgr2=0x7f166c1010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.385+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 shutdown_connections 2024-04-24T08:24:04.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.385+0000 7f1672400640 1 -- 172.21.15.146:0/2818094171 wait complete. 2024-04-24T08:24:04.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:24:04.527 INFO:tasks.workunit.client.0.smithi146.stdout:checking attr/attributes.h usability... yes 2024-04-24T08:24:04.562 INFO:tasks.workunit.client.0.smithi146.stdout:checking attr/attributes.h presence... yes 2024-04-24T08:24:04.562 INFO:tasks.workunit.client.0.smithi146.stdout:checking for attr/attributes.h... yes 2024-04-24T08:24:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 -- 172.21.15.146:0/621098421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 msgr2=0x7f86841011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 --2- 172.21.15.146:0/621098421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841011e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8674009900 tx=0x7f867402a410 comp rx=0 tx=0).stop 2024-04-24T08:24:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 -- 172.21.15.146:0/621098421 shutdown_connections 2024-04-24T08:24:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 --2- 172.21.15.146:0/621098421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8684101720 0x7f868410e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 --2- 172.21.15.146:0/621098421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f868410e9b0 0x7f8684110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 --2- 172.21.15.146:0/621098421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841011e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 -- 172.21.15.146:0/621098421 >> 172.21.15.146:0/621098421 conn(0x7f8684078000 msgr2=0x7f8684078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 -- 172.21.15.146:0/621098421 shutdown_connections 2024-04-24T08:24:04.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.634+0000 7f868b000640 1 -- 172.21.15.146:0/621098421 wait complete. 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.635+0000 7f868b000640 1 Processor -- start 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.635+0000 7f868b000640 1 -- start start 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f868b000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f868b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8684101720 0x7f86841a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f868b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f868410e9b0 0x7f868419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f868b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8684114620 con 0x7f868410e9b0 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f868b000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86841144a0 con 0x7f8684100e00 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f868b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f86841147a0 con 0x7f8684101720 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39362/0 (socket says 172.21.15.146:39362) 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 -- 172.21.15.146:0/7131264 learned_addr learned my addr 172.21.15.146:0/7131264 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8689000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f868410e9b0 0x7f868419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683400640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8684101720 0x7f86841a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 -- 172.21.15.146:0/7131264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8684101720 msgr2=0x7f86841a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8684101720 0x7f86841a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 -- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f868410e9b0 msgr2=0x7f868419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f868410e9b0 0x7f868419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8683e00640 1 -- 172.21.15.146:0/7131264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868419b080 con 0x7f8684100e00 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.636+0000 7f8689000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f868410e9b0 0x7f868419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.637+0000 7f8683e00640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841a0750 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8674004770 tx=0x7f8674003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.637+0000 7f8680c00640 1 -- 172.21.15.146:0/7131264 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867403f070 con 0x7f8684100e00 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.637+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868419b310 con 0x7f8684100e00 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.637+0000 7f8680c00640 1 -- 172.21.15.146:0/7131264 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8674031ca0 con 0x7f8684100e00 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.637+0000 7f8680c00640 1 -- 172.21.15.146:0/7131264 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674035030 con 0x7f8684100e00 2024-04-24T08:24:04.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.637+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86841a7610 con 0x7f8684100e00 2024-04-24T08:24:04.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.640+0000 7f868de2d740 1 Processor -- start 2024-04-24T08:24:04.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f8680c00640 1 -- 172.21.15.146:0/7131264 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f8674042070 con 0x7f8684100e00 2024-04-24T08:24:04.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f868de2d740 1 -- start start 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f868de2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a38829d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f868de2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a3882a270 0x562a38834400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f868de2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38836d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f868de2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x562a3878f1d0 con 0x562a38834940 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f868de2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562a38735450 con 0x562a38829950 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f868de2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562a3878b190 con 0x562a3882a270 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f8683e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a3882a270 0x562a38834400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f8689000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a38829d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f8683e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a3882a270 0x562a38834400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54252/0 (socket says 172.21.15.146:54252) 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f8689000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a38829d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39366/0 (socket says 172.21.15.146:39366) 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f8683e00640 1 -- 172.21.15.146:0/168664596 learned_addr learned my addr 172.21.15.146:0/168664596 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.641+0000 7f8683400640 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38836d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.642+0000 7f8683e00640 1 -- 172.21.15.146:0/168664596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 msgr2=0x562a38829d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.642+0000 7f8683e00640 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a38829d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.642+0000 7f8683e00640 1 -- 172.21.15.146:0/168664596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 msgr2=0x562a38836d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.642+0000 7f8683e00640 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38836d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.642+0000 7f8683e00640 1 -- 172.21.15.146:0/168664596 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a387f2310 con 0x562a3882a270 2024-04-24T08:24:04.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.642+0000 7f8683e00640 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a3882a270 0x562a38834400 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86740ee440 tx=0x7f86740f7470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca7a8ff73f1c6294 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.642+0000 7f8679600640 1 -- 172.21.15.146:0/168664596 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86740f78d0 con 0x562a3882a270 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.643+0000 7f8679600640 1 -- 172.21.15.146:0/168664596 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86740f5450 con 0x562a3882a270 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.643+0000 7f8679600640 1 -- 172.21.15.146:0/168664596 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674101610 con 0x562a3882a270 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.643+0000 7f8680c00640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f86680a3d20 0x7f86680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.643+0000 7f8680c00640 1 -- 172.21.15.146:0/7131264 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f86740f0e00 con 0x7f8684100e00 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.643+0000 7f8689000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f86680a3d20 0x7f86680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.643+0000 7f868de2d740 1 -- 172.21.15.146:0/168664596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a3882a270 msgr2=0x562a38834400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.643+0000 7f868de2d740 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a3882a270 0x562a38834400 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86740ee440 tx=0x7f86740f7470 comp rx=0 tx=0).stop 2024-04-24T08:24:04.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 -- 172.21.15.146:0/168664596 shutdown_connections 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a3882a270 0x562a38834400 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38836d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 --2- 172.21.15.146:0/168664596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a38829d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 -- 172.21.15.146:0/168664596 >> 172.21.15.146:0/168664596 conn(0x562a38825c70 msgr2=0x562a38826050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f8689000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f86680a3d20 0x7f86680a61e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f867c009fd0 tx=0x7f867c009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 -- 172.21.15.146:0/168664596 shutdown_connections 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 -- 172.21.15.146:0/168664596 wait complete. 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 -- start start 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.644+0000 7f868de2d740 1 Processor -- start 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f868de2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a38834370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f868de2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38831660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f868de2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38834940 0x562a38831da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f868de2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x562a388374f0 con 0x562a3882a270 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f868de2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562a38748810 con 0x562a38834940 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f868de2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562a38837670 con 0x562a38829950 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f868de2d740 10 client.0 _ll_register_callbacks cb 0x562a38716140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f8683e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a38834370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f8689000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38834940 0x562a38831da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f8683e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a38834370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54266/0 (socket says 172.21.15.146:54266) 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f8683e00640 1 -- 172.21.15.146:0/1819824667 learned_addr learned my addr 172.21.15.146:0/1819824667 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.645+0000 7f8683400640 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38831660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f8683e00640 1 -- 172.21.15.146:0/1819824667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38834940 msgr2=0x562a38831da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f8683e00640 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38834940 0x562a38831da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f8683e00640 1 -- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 msgr2=0x562a38831660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f8683e00640 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38831660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f8683e00640 1 -- 172.21.15.146:0/1819824667 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a387593f0 con 0x562a38829950 2024-04-24T08:24:04.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f8683e00640 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a38834370 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f86740b9890 tx=0x7f86740f5bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa9abc2dff08ab7c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f866e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f866e000640 1 -- 172.21.15.146:0/1819824667 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674049030 con 0x562a38829950 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f866e000640 1 -- 172.21.15.146:0/1819824667 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f86740bb1c0 con 0x562a38829950 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86740f5450 con 0x562a38829950 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f866e000640 1 -- 172.21.15.146:0/1819824667 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f86740b9160 con 0x562a38829950 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f868de2d740 10 client.25918 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.646+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a384a13f0 con 0x562a38829950 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f866e000640 1 -- 172.21.15.146:0/1819824667 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86740f5450 con 0x562a38829950 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f866e000640 1 -- 172.21.15.146:0/1819824667 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f86740b9530 con 0x562a38829950 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f868de2d740 10 client.25918 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f868de2d740 10 client.25918 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f868de2d740 20 client.25918 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:04.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f868de2d740 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a38844690 0x562a38846a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f868de2d740 4 client.25918 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:04.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a3879cbf0 con 0x562a38844690 2024-04-24T08:24:04.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.647+0000 7f8683400640 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a38844690 0x562a38846a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.648+0000 7f866e000640 10 client.25918 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.648+0000 7f8683400640 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a38844690 0x562a38846a80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=59c5b782201035e8 server_cookie=6b337b72d50f3e73 in_seq=0 out_seq=0 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f866e000640 1 -- 172.21.15.146:0/1819824667 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a3879cbf0 con 0x562a38844690 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f866e000640 10 client.25918 handle_command_reply: tid=0 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 client.25918 shutdown 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 msgr2=0x562a38834370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a38834370 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f86740b9890 tx=0x7f86740f5bf0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 shutdown_connections 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a38844690 0x562a38846a80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38834940 0x562a38831da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38831660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f866e000640 0 client.25918 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 --2- 172.21.15.146:0/1819824667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a38834370 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 >> 172.21.15.146:0/1819824667 conn(0x562a38825c70 msgr2=0x562a3882ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 shutdown_connections 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.649+0000 7f868de2d740 1 -- 172.21.15.146:0/1819824667 wait complete. 2024-04-24T08:24:04.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.650+0000 7f868de2d740 20 client.25918 trim_cache size 0 max 16384 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 Processor -- start 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 -- start start 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a388403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38830640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38834940 0x562a38833ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f86841b27c0 con 0x562a3882a270 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86841b0090 con 0x562a38829950 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f868de2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f86841adee0 con 0x562a38834940 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f8683e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a388403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f8683e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a388403b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39380/0 (socket says 172.21.15.146:39380) 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f8683400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38830640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f8683e00640 1 -- 172.21.15.146:0/903236427 learned_addr learned my addr 172.21.15.146:0/903236427 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.665+0000 7f8689000640 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38834940 0x562a38833ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8683e00640 1 -- 172.21.15.146:0/903236427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38834940 msgr2=0x562a38833ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8683e00640 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38834940 0x562a38833ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8683e00640 1 -- 172.21.15.146:0/903236427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 msgr2=0x562a38830640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8683e00640 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38830640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8683e00640 1 -- 172.21.15.146:0/903236427 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a38832110 con 0x562a38829950 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8683e00640 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a388403b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f86740bce50 tx=0x7f867403b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3ac970f81340298 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8678c00640 1 -- 172.21.15.146:0/903236427 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674049070 con 0x562a38829950 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8678c00640 1 -- 172.21.15.146:0/903236427 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86740f4680 con 0x562a38829950 2024-04-24T08:24:04.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.666+0000 7f8678c00640 1 -- 172.21.15.146:0/903236427 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674005d50 con 0x562a38829950 2024-04-24T08:24:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 -- 172.21.15.146:0/903236427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 msgr2=0x562a388403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a388403b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f86740bce50 tx=0x7f867403b890 comp rx=0 tx=0).stop 2024-04-24T08:24:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 -- 172.21.15.146:0/903236427 shutdown_connections 2024-04-24T08:24:04.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38834940 0x562a38833ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a3882a270 0x562a38830640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 --2- 172.21.15.146:0/903236427 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a38829950 0x562a388403b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 -- 172.21.15.146:0/903236427 >> 172.21.15.146:0/903236427 conn(0x562a38825c70 msgr2=0x562a3882ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 -- 172.21.15.146:0/903236427 shutdown_connections 2024-04-24T08:24:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.667+0000 7f868de2d740 1 -- 172.21.15.146:0/903236427 wait complete. 2024-04-24T08:24:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 -- start start 2024-04-24T08:24:04.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 Processor -- start 2024-04-24T08:24:04.670 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/attributes.h usability... no 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a3883cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a3882a270 0x562a388353c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38835900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f86841ae550 con 0x562a38834940 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f86841af9c0 con 0x562a3882a270 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f86841aec20 con 0x562a38829950 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f868de2d740 10 client.0 _ll_register_callbacks cb 0x7f8670009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f8683400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a3882a270 0x562a388353c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f8683e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a3883cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f8683400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a3882a270 0x562a388353c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39394/0 (socket says 172.21.15.146:39394) 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f8683e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a3883cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54288/0 (socket says 172.21.15.146:54288) 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.668+0000 7f8683400640 1 -- 172.21.15.146:0/279102149 learned_addr learned my addr 172.21.15.146:0/279102149 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8689000640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38835900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8683e00640 1 -- 172.21.15.146:0/279102149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a3882a270 msgr2=0x562a388353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8683e00640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a3882a270 0x562a388353c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8683e00640 1 -- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 msgr2=0x562a38835900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8683e00640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38835900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8683e00640 1 -- 172.21.15.146:0/279102149 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3856b7d0 con 0x562a38829950 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8689000640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38835900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8683400640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a3882a270 0x562a388353c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f8683e00640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a3883cbc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f86740c5550 tx=0x7f86740f4fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce5193ede24af0eb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f866e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f866e000640 1 -- 172.21.15.146:0/279102149 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674049070 con 0x562a38829950 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f866e000640 1 -- 172.21.15.146:0/279102149 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f86740b9530 con 0x562a38829950 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.669+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86740f4680 con 0x562a38829950 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f866e000640 1 -- 172.21.15.146:0/279102149 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f867407b050 con 0x562a38829950 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 10 client.25924 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a3856b7d0 con 0x562a38829950 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f866e000640 1 -- 172.21.15.146:0/279102149 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86740f4680 con 0x562a38829950 2024-04-24T08:24:04.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f866e000640 1 -- 172.21.15.146:0/279102149 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f86740ee780 con 0x562a38829950 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 10 client.25924 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 10 client.25924 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 20 client.25924 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a388aab20 0x562a388acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 4 client.25924 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a38796a60 con 0x562a388aab20 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.670+0000 7f8683400640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a388aab20 0x562a388acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.671+0000 7f8683400640 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a388aab20 0x562a388acf10 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe3140f32e7d37c3 server_cookie=d7958988d69e4a3a in_seq=0 out_seq=0 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.671+0000 7f866e000640 10 client.25924 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.671+0000 7f866e000640 1 -- 172.21.15.146:0/279102149 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a38796a60 con 0x562a388aab20 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.671+0000 7f866e000640 10 client.25924 handle_command_reply: tid=0 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.671+0000 7f868de2d740 1 client.25924 shutdown 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.671+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 msgr2=0x562a3883cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.671+0000 7f868de2d740 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a3883cbc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f86740c5550 tx=0x7f86740f4fb0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 shutdown_connections 2024-04-24T08:24:04.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562a388aab20 0x562a388acf10 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562a38834940 0x562a38835900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562a3882a270 0x562a388353c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f866e000640 0 client.25924 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 --2- 172.21.15.146:0/279102149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562a38829950 0x562a3883cbc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 >> 172.21.15.146:0/279102149 conn(0x562a38825c70 msgr2=0x562a38824560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 shutdown_connections 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 1 -- 172.21.15.146:0/279102149 wait complete. 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.672+0000 7f868de2d740 20 client.25924 trim_cache size 0 max 16384 2024-04-24T08:24:04.677 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f86680a3d20 msgr2=0x7f86680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f86680a3d20 0x7f86680a61e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f867c009fd0 tx=0x7f867c009500 comp rx=0 tx=0).stop 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 msgr2=0x7f86841a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841a0750 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8674004770 tx=0x7f8674003c10 comp rx=0 tx=0).stop 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 shutdown_connections 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f86680a3d20 0x7f86680a61e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8684101720 0x7f86841a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f868410e9b0 0x7f868419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 --2- 172.21.15.146:0/7131264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8684100e00 0x7f86841a0750 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 >> 172.21.15.146:0/7131264 conn(0x7f8684078000 msgr2=0x7f868410fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 shutdown_connections 2024-04-24T08:24:04.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.673+0000 7f868b000640 1 -- 172.21.15.146:0/7131264 wait complete. 2024-04-24T08:24:04.690 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:04 smithi146 ceph-mon[27805]: pgmap v89: 97 pgs: 97 active+clean; 65 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 1.6 MiB/s wr, 379 op/s 2024-04-24T08:24:04.690 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:04 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2818094171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:24:04.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:24:04.715 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/attributes.h presence... no 2024-04-24T08:24:04.715 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/attributes.h... no 2024-04-24T08:24:04.800 INFO:tasks.workunit.client.0.smithi146.stdout:checking for attr_list in -lattr... yes 2024-04-24T08:24:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:04 smithi177 ceph-mon[33046]: pgmap v89: 97 pgs: 97 active+clean; 65 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 1.6 MiB/s wr, 379 op/s 2024-04-24T08:24:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:04 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2818094171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.873+0000 7f95bc200640 1 -- 172.21.15.146:0/3394006894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 msgr2=0x7f95b4104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.873+0000 7f95bc200640 1 --2- 172.21.15.146:0/3394006894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b4104890 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009900 tx=0x7f95a802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.874+0000 7f95bc200640 1 -- 172.21.15.146:0/3394006894 shutdown_connections 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.874+0000 7f95bc200640 1 --2- 172.21.15.146:0/3394006894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95b4104e60 0x7f95b4108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.874+0000 7f95bc200640 1 --2- 172.21.15.146:0/3394006894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95b4109580 0x7f95b4111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.874+0000 7f95bc200640 1 --2- 172.21.15.146:0/3394006894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b4104890 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.874+0000 7f95bc200640 1 -- 172.21.15.146:0/3394006894 >> 172.21.15.146:0/3394006894 conn(0x7f95b40ffd90 msgr2=0x7f95b41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.874+0000 7f95bc200640 1 -- 172.21.15.146:0/3394006894 shutdown_connections 2024-04-24T08:24:04.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.874+0000 7f95bc200640 1 -- 172.21.15.146:0/3394006894 wait complete. 2024-04-24T08:24:04.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 Processor -- start 2024-04-24T08:24:04.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 -- start start 2024-04-24T08:24:04.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b41a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95b4109580 0x7f95b41a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95b419a9e0 0x7f95b419ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f95b4114680 con 0x7f95b419a9e0 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95b4114500 con 0x7f95b41044b0 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.875+0000 7f95bc200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f95b4114800 con 0x7f95b4109580 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b41a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b41a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39410/0 (socket says 172.21.15.146:39410) 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95ba200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95b419a9e0 0x7f95b419ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 -- 172.21.15.146:0/2971680851 learned_addr learned my addr 172.21.15.146:0/2971680851 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b8e00640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95b4109580 0x7f95b41a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 -- 172.21.15.146:0/2971680851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95b4109580 msgr2=0x7f95b41a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95b4109580 0x7f95b41a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 -- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95b419a9e0 msgr2=0x7f95b419ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95b419a9e0 0x7f95b419ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 -- 172.21.15.146:0/2971680851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b419b500 con 0x7f95b41044b0 2024-04-24T08:24:04.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b8e00640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95b4109580 0x7f95b41a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:04.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.876+0000 7f95b9800640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b41a0830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f95a8004770 tx=0x7f95a8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.877+0000 7f95b2000640 1 -- 172.21.15.146:0/2971680851 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a803f070 con 0x7f95b41044b0 2024-04-24T08:24:04.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.877+0000 7f95b2000640 1 -- 172.21.15.146:0/2971680851 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95a8031ca0 con 0x7f95b41044b0 2024-04-24T08:24:04.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.877+0000 7f95b2000640 1 -- 172.21.15.146:0/2971680851 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8035030 con 0x7f95b41044b0 2024-04-24T08:24:04.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.877+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b419b790 con 0x7f95b41044b0 2024-04-24T08:24:04.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.877+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b41a7980 con 0x7f95b41044b0 2024-04-24T08:24:04.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.880+0000 7f95bef88740 1 Processor -- start 2024-04-24T08:24:04.881 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.880+0000 7f95b2000640 1 -- 172.21.15.146:0/2971680851 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f95a8042070 con 0x7f95b41044b0 2024-04-24T08:24:04.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95bef88740 1 -- start start 2024-04-24T08:24:04.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95bef88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e50d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95bef88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e5b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95bef88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e5dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95bef88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556b34db61d0 con 0x556b34e5b940 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95bef88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556b34d5c450 con 0x556b34e50950 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95bef88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556b34db2190 con 0x556b34e51270 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95ba200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e50d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95ba200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e50d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39426/0 (socket says 172.21.15.146:39426) 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95b9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e5b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95ba200640 1 -- 172.21.15.146:0/1449372074 learned_addr learned my addr 172.21.15.146:0/1449372074 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.881+0000 7f95b8e00640 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e5dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95ba200640 1 -- 172.21.15.146:0/1449372074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 msgr2=0x556b34e5b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95ba200640 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e5b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95ba200640 1 -- 172.21.15.146:0/1449372074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 msgr2=0x556b34e5dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95ba200640 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e5dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95ba200640 1 -- 172.21.15.146:0/1449372074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b34e19310 con 0x556b34e50950 2024-04-24T08:24:04.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95b2000640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f959c0a3d20 0x7f959c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95ba200640 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e50d30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009f50 tx=0x7f95ac0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e104b043f19136f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95b2000640 1 -- 172.21.15.146:0/2971680851 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f95a80f0df0 con 0x7f95b41044b0 2024-04-24T08:24:04.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.883+0000 7f95b9800640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f959c0a3d20 0x7f959c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.882+0000 7f95a6a00640 1 -- 172.21.15.146:0/1449372074 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac004d20 con 0x556b34e50950 2024-04-24T08:24:04.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.883+0000 7f95a6a00640 1 -- 172.21.15.146:0/1449372074 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ac004050 con 0x556b34e50950 2024-04-24T08:24:04.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.883+0000 7f95a6a00640 1 -- 172.21.15.146:0/1449372074 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac005280 con 0x556b34e50950 2024-04-24T08:24:04.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.883+0000 7f95bef88740 1 -- 172.21.15.146:0/1449372074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 msgr2=0x556b34e50d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.883+0000 7f95b9800640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f959c0a3d20 0x7f959c0a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x556b34e51f10 tx=0x7f95a80f48a0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.883+0000 7f95bef88740 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e50d30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009f50 tx=0x7f95ac0118d0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 -- 172.21.15.146:0/1449372074 shutdown_connections 2024-04-24T08:24:04.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e5b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e5dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 --2- 172.21.15.146:0/1449372074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e50d30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 -- 172.21.15.146:0/1449372074 >> 172.21.15.146:0/1449372074 conn(0x556b34e4cc70 msgr2=0x556b34e4d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 -- 172.21.15.146:0/1449372074 shutdown_connections 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 -- 172.21.15.146:0/1449372074 wait complete. 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 -- start start 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.884+0000 7f95bef88740 1 Processor -- start 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95bef88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e5b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95bef88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95bef88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e58e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95bef88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556b34e5e7d0 con 0x556b34e5b940 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95bef88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556b34d6f810 con 0x556b34e50950 2024-04-24T08:24:04.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95bef88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556b34e5e950 con 0x556b34e51270 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95bef88740 10 client.0 _ll_register_callbacks cb 0x556b34d3d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95ba200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e5b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95b8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95ba200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e5b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39432/0 (socket says 172.21.15.146:39432) 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95ba200640 1 -- 172.21.15.146:0/834479459 learned_addr learned my addr 172.21.15.146:0/834479459 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.885+0000 7f95ba200640 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e58e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f95ba200640 1 -- 172.21.15.146:0/834479459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 msgr2=0x556b34e586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f95ba200640 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f95ba200640 1 -- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 msgr2=0x556b34e58e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f95ba200640 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e58e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f95ba200640 1 -- 172.21.15.146:0/834479459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b34d803f0 con 0x556b34e50950 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f95ba200640 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e5b3f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009250 tx=0x7f95ac0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90587dd1022cbfb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f959b400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.886+0000 7f959b400640 1 -- 172.21.15.146:0/834479459 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac02b070 con 0x556b34e50950 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f959b400640 1 -- 172.21.15.146:0/834479459 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f95ac027640 con 0x556b34e50950 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95ac004050 con 0x556b34e50950 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f959b400640 1 -- 172.21.15.146:0/834479459 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f95ac037050 con 0x556b34e50950 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 10 client.25952 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556b34ac83f0 con 0x556b34e50950 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f959b400640 1 -- 172.21.15.146:0/834479459 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ac004050 con 0x556b34e50950 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f959b400640 1 -- 172.21.15.146:0/834479459 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f95ac00f780 con 0x556b34e50950 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 10 client.25952 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 10 client.25952 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 20 client.25952 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34e6ba60 0x556b34e6de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 4 client.25952 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.887+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b34dc3bf0 con 0x556b34e6ba60 2024-04-24T08:24:04.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.888+0000 7f95b8e00640 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34e6ba60 0x556b34e6de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.888+0000 7f959b400640 10 client.25952 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.888+0000 7f95b8e00640 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34e6ba60 0x556b34e6de50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be902548d8c1f2c3 server_cookie=888b54794482ad94 in_seq=0 out_seq=0 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f959b400640 1 -- 172.21.15.146:0/834479459 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b34dc3bf0 con 0x556b34e6ba60 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f959b400640 10 client.25952 handle_command_reply: tid=0 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 client.25952 shutdown 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 msgr2=0x556b34e5b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e5b3f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009250 tx=0x7f95ac0047b0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 shutdown_connections 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34e6ba60 0x556b34e6de50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e58e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 --2- 172.21.15.146:0/834479459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e5b3f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f959b400640 0 client.25952 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:04.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.889+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 >> 172.21.15.146:0/834479459 conn(0x556b34e4cc70 msgr2=0x556b34e52a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.890+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 shutdown_connections 2024-04-24T08:24:04.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.890+0000 7f95bef88740 1 -- 172.21.15.146:0/834479459 wait complete. 2024-04-24T08:24:04.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.890+0000 7f95bef88740 20 client.25952 trim_cache size 0 max 16384 2024-04-24T08:24:04.894 INFO:tasks.workunit.client.0.smithi146.stdout:checking for attr_set in -lattr... yes 2024-04-24T08:24:04.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.905+0000 7f95bef88740 1 Processor -- start 2024-04-24T08:24:04.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.905+0000 7f95bef88740 1 -- start start 2024-04-24T08:24:04.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.905+0000 7f95bef88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e67690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.905+0000 7f95bef88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e53040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95bef88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e56e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95bef88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41b28f0 con 0x556b34e5b940 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95bef88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41b01c0 con 0x556b34e50950 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95bef88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41ae010 con 0x556b34e51270 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e67690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e67690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39442/0 (socket says 172.21.15.146:39442) 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 -- 172.21.15.146:0/2466926538 learned_addr learned my addr 172.21.15.146:0/2466926538 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e56e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95b8e00640 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e53040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 -- 172.21.15.146:0/2466926538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 msgr2=0x556b34e53040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e53040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 -- 172.21.15.146:0/2466926538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 msgr2=0x556b34e56e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e56e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.906+0000 7f95ba200640 1 -- 172.21.15.146:0/2466926538 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b34e6e1c0 con 0x556b34e50950 2024-04-24T08:24:04.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.907+0000 7f95ba200640 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e67690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009560 tx=0x7f95ac0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b571611d778eb15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.907+0000 7f95a6000640 1 -- 172.21.15.146:0/2466926538 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac02b050 con 0x556b34e50950 2024-04-24T08:24:04.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.907+0000 7f95a6000640 1 -- 172.21.15.146:0/2466926538 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ac00f960 con 0x556b34e50950 2024-04-24T08:24:04.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.907+0000 7f95a6000640 1 -- 172.21.15.146:0/2466926538 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac027990 con 0x556b34e50950 2024-04-24T08:24:04.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 -- 172.21.15.146:0/2466926538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 msgr2=0x556b34e67690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e67690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f95ac009560 tx=0x7f95ac0118d0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 -- 172.21.15.146:0/2466926538 shutdown_connections 2024-04-24T08:24:04.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e51270 0x556b34e53040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e5b940 0x556b34e56e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 --2- 172.21.15.146:0/2466926538 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e50950 0x556b34e67690 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 -- 172.21.15.146:0/2466926538 >> 172.21.15.146:0/2466926538 conn(0x556b34e4cc70 msgr2=0x556b34e52930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 -- 172.21.15.146:0/2466926538 shutdown_connections 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 -- 172.21.15.146:0/2466926538 wait complete. 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 -- start start 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.908+0000 7f95bef88740 1 Processor -- start 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95bef88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e50950 0x556b34e5ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95bef88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e51270 0x556b34e5cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95bef88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e5b940 0x556b34e62060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95bef88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41ae680 con 0x556b34e51270 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95bef88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41afaf0 con 0x556b34e5b940 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95bef88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41aed50 con 0x556b34e50950 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95bef88740 10 client.0 _ll_register_callbacks cb 0x7f95a0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95ba200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e50950 0x556b34e5ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95b8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e51270 0x556b34e5cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95ba200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e50950 0x556b34e5ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54360/0 (socket says 172.21.15.146:54360) 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95ba200640 1 -- 172.21.15.146:0/3185775285 learned_addr learned my addr 172.21.15.146:0/3185775285 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.909+0000 7f95ba200640 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e5b940 0x556b34e62060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95ba200640 1 -- 172.21.15.146:0/3185775285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e5b940 msgr2=0x556b34e62060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95ba200640 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e5b940 0x556b34e62060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95ba200640 1 -- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e51270 msgr2=0x556b34e5cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95ba200640 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e51270 0x556b34e5cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95ba200640 1 -- 172.21.15.146:0/3185775285 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b34e59190 con 0x556b34e50950 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95b8e00640 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e51270 0x556b34e5cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95ba200640 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e50950 0x556b34e5ca20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95ac032040 tx=0x7f95ac0096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b37ee5b3d2478fe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f959b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:24:04.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f959b400640 1 -- 172.21.15.146:0/3185775285 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac02b070 con 0x556b34e50950 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95ac00f960 con 0x556b34e50950 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.910+0000 7f959b400640 1 -- 172.21.15.146:0/3185775285 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f95ac003e70 con 0x556b34e50950 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f959b400640 1 -- 172.21.15.146:0/3185775285 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f95ac00f480 con 0x556b34e50950 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 10 client.25942 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556b34b927d0 con 0x556b34e50950 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f959b400640 1 -- 172.21.15.146:0/3185775285 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ac00f960 con 0x556b34e50950 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f959b400640 1 -- 172.21.15.146:0/3185775285 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f95ac019500 con 0x556b34e50950 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 10 client.25942 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 10 client.25942 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 20 client.25942 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34ed26c0 0x556b34ed4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 4 client.25942 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:24:04.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.911+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b34dbda60 con 0x556b34ed26c0 2024-04-24T08:24:04.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.912+0000 7f95b8e00640 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34ed26c0 0x556b34ed4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.912+0000 7f959b400640 10 client.25942 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.912+0000 7f95b8e00640 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34ed26c0 0x556b34ed4ae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e1ccb3670520efb server_cookie=570a327960c82380 in_seq=0 out_seq=0 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.912+0000 7f959b400640 1 -- 172.21.15.146:0/3185775285 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b34dbda60 con 0x556b34ed26c0 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.912+0000 7f959b400640 10 client.25942 handle_command_reply: tid=0 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 client.25942 shutdown 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e50950 msgr2=0x556b34e5ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e50950 0x556b34e5ca20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f95ac032040 tx=0x7f95ac0096c0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 shutdown_connections 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556b34ed26c0 0x556b34ed4ae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556b34e5b940 0x556b34e62060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556b34e51270 0x556b34e5cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 --2- 172.21.15.146:0/3185775285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b34e50950 0x556b34e5ca20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f959b400640 0 client.25942 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:04.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 >> 172.21.15.146:0/3185775285 conn(0x556b34e4cc70 msgr2=0x556b34e66680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 shutdown_connections 2024-04-24T08:24:04.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 1 -- 172.21.15.146:0/3185775285 wait complete. 2024-04-24T08:24:04.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.913+0000 7f95bef88740 20 client.25942 trim_cache size 0 max 16384 2024-04-24T08:24:04.918 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f959c0a3d20 msgr2=0x7f959c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f959c0a3d20 0x7f959c0a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x556b34e51f10 tx=0x7f95a80f48a0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 msgr2=0x7f95b41a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b41a0830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f95a8004770 tx=0x7f95a8003c10 comp rx=0 tx=0).stop 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 shutdown_connections 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f959c0a3d20 0x7f959c0a61e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95b4109580 0x7f95b41a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95b419a9e0 0x7f95b419ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 --2- 172.21.15.146:0/2971680851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f95b41044b0 0x7f95b41a0830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 >> 172.21.15.146:0/2971680851 conn(0x7f95b40ffd90 msgr2=0x7f95b41010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 shutdown_connections 2024-04-24T08:24:04.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:04.914+0000 7f95bc200640 1 -- 172.21.15.146:0/2971680851 wait complete. 2024-04-24T08:24:04.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:24:04.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:04 smithi153 ceph-mon[32746]: pgmap v89: 97 pgs: 97 active+clean; 65 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 1.6 MiB/s wr, 379 op/s 2024-04-24T08:24:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:04 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2818094171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:24:04.973 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/acl.h usability... yes 2024-04-24T08:24:05.014 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/acl.h presence... yes 2024-04-24T08:24:05.014 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/acl.h... yes 2024-04-24T08:24:05.086 INFO:tasks.workunit.client.0.smithi146.stdout:checking acl/libacl.h usability... yes 2024-04-24T08:24:05.128 INFO:tasks.workunit.client.0.smithi146.stdout:checking acl/libacl.h presence... yes 2024-04-24T08:24:05.129 INFO:tasks.workunit.client.0.smithi146.stdout:checking for acl/libacl.h... yes 2024-04-24T08:24:05.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 -- 172.21.15.146:0/2345735995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 msgr2=0x7fcbb4104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 --2- 172.21.15.146:0/2345735995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb4104890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009860 tx=0x7fcba802a340 comp rx=0 tx=0).stop 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 -- 172.21.15.146:0/2345735995 shutdown_connections 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 --2- 172.21.15.146:0/2345735995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb4109580 0x7fcbb4111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 --2- 172.21.15.146:0/2345735995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb4104e60 0x7fcbb4108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 --2- 172.21.15.146:0/2345735995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb4104890 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 -- 172.21.15.146:0/2345735995 >> 172.21.15.146:0/2345735995 conn(0x7fcbb40ffd90 msgr2=0x7fcbb41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.162+0000 7fcbbc400640 1 -- 172.21.15.146:0/2345735995 shutdown_connections 2024-04-24T08:24:05.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.163+0000 7fcbbc400640 1 -- 172.21.15.146:0/2345735995 wait complete. 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.163+0000 7fcbbc400640 1 Processor -- start 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbbc400640 1 -- start start 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbbc400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb41a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbbc400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb4109580 0x7fcbb41a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbbc400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb419a9e0 0x7fcbb419ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbbc400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb4114640 con 0x7fcbb4109580 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbbc400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41144c0 con 0x7fcbb419a9e0 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbbc400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41147c0 con 0x7fcbb41044b0 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbb9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb4109580 0x7fcbb41a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbb9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb41a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbb9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb41a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54368/0 (socket says 172.21.15.146:54368) 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbb9a00640 1 -- 172.21.15.146:0/634999478 learned_addr learned my addr 172.21.15.146:0/634999478 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.164+0000 7fcbb9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb4109580 0x7fcbb41a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33520/0 (socket says 172.21.15.146:33520) 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbba400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb419a9e0 0x7fcbb419ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb9a00640 1 -- 172.21.15.146:0/634999478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb419a9e0 msgr2=0x7fcbb419ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb419a9e0 0x7fcbb419ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb9a00640 1 -- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb4109580 msgr2=0x7fcbb41a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb4109580 0x7fcbb41a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb9a00640 1 -- 172.21.15.146:0/634999478 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb419b500 con 0x7fcbb41044b0 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb41a07e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcba8035040 tx=0x7fcba8033b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb2000640 1 -- 172.21.15.146:0/634999478 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8037040 con 0x7fcbb41044b0 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb2000640 1 -- 172.21.15.146:0/634999478 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba8040070 con 0x7fcbb41044b0 2024-04-24T08:24:05.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb419b790 con 0x7fcbb41044b0 2024-04-24T08:24:05.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.165+0000 7fcbb2000640 1 -- 172.21.15.146:0/634999478 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80039b0 con 0x7fcbb41044b0 2024-04-24T08:24:05.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.166+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb41a7990 con 0x7fcbb41044b0 2024-04-24T08:24:05.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.169+0000 7fcbb2000640 1 -- 172.21.15.146:0/634999478 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fcba80305e0 con 0x7fcbb41044b0 2024-04-24T08:24:05.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.169+0000 7fcbbf370740 1 Processor -- start 2024-04-24T08:24:05.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.169+0000 7fcbbf370740 1 -- start start 2024-04-24T08:24:05.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dc950 0x5610a14dcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14e7940 0x5610a14e9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbbf370740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5610a14421d0 con 0x5610a14e7940 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbbf370740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5610a13e8450 con 0x5610a14dd270 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbbf370740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5610a143e190 con 0x5610a14dc950 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbba400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dc950 0x5610a14dcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39498/0 (socket says 172.21.15.146:39498) 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbba400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dc950 0x5610a14dcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54370/0 (socket says 172.21.15.146:54370) 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 -- 172.21.15.146:0/219501135 learned_addr learned my addr 172.21.15.146:0/219501135 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9000640 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14e7940 0x5610a14e9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 -- 172.21.15.146:0/219501135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dc950 msgr2=0x5610a14dcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dc950 0x5610a14dcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 -- 172.21.15.146:0/219501135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14e7940 msgr2=0x5610a14e9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14e7940 0x5610a14e9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.170+0000 7fcbb9a00640 1 -- 172.21.15.146:0/219501135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a14a5310 con 0x5610a14dd270 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.171+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e7400 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcba80ede20 tx=0x7fcba80f6b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba271622cfa31b41 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:05.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.171+0000 7fcbb2000640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb9c0a3da0 0x7fcb9c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.171+0000 7fcbb2000640 1 -- 172.21.15.146:0/634999478 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcba80f06d0 con 0x7fcbb41044b0 2024-04-24T08:24:05.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.171+0000 7fcba6a00640 1 -- 172.21.15.146:0/219501135 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80ff070 con 0x5610a14dd270 2024-04-24T08:24:05.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.171+0000 7fcbba400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb9c0a3da0 0x7fcb9c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.171+0000 7fcba6a00640 1 -- 172.21.15.146:0/219501135 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba80bce30 con 0x5610a14dd270 2024-04-24T08:24:05.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.171+0000 7fcba6a00640 1 -- 172.21.15.146:0/219501135 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80bc500 con 0x5610a14dd270 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 -- 172.21.15.146:0/219501135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 msgr2=0x5610a14e7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbba400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb9c0a3da0 0x7fcb9c0a6260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcbac005fd0 tx=0x7fcbac009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e7400 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcba80ede20 tx=0x7fcba80f6b20 comp rx=0 tx=0).stop 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 -- 172.21.15.146:0/219501135 shutdown_connections 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14e7940 0x5610a14e9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e7400 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 --2- 172.21.15.146:0/219501135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dc950 0x5610a14dcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 -- 172.21.15.146:0/219501135 >> 172.21.15.146:0/219501135 conn(0x5610a14d8c70 msgr2=0x5610a14d9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 -- 172.21.15.146:0/219501135 shutdown_connections 2024-04-24T08:24:05.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.172+0000 7fcbbf370740 1 -- 172.21.15.146:0/219501135 wait complete. 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.173+0000 7fcbbf370740 1 -- start start 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.173+0000 7fcbbf370740 1 Processor -- start 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dd270 0x5610a14e4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14e7940 0x5610a14e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbbf370740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5610a14ea4f0 con 0x5610a14dc950 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbbf370740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5610a13fb810 con 0x5610a14e7940 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbbf370740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5610a14ea670 con 0x5610a14dd270 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbbf370740 10 client.0 _ll_register_callbacks cb 0x5610a13c9140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbb9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbb9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33530/0 (socket says 172.21.15.146:33530) 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbb9a00640 1 -- 172.21.15.146:0/23582167 learned_addr learned my addr 172.21.15.146:0/23582167 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbba400640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14e7940 0x5610a14e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbb9000640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dd270 0x5610a14e4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbba400640 1 -- 172.21.15.146:0/23582167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dd270 msgr2=0x5610a14e4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbba400640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dd270 0x5610a14e4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbba400640 1 -- 172.21.15.146:0/23582167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 msgr2=0x5610a14e7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbba400640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbba400640 1 -- 172.21.15.146:0/23582167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a140c3f0 con 0x5610a14e7940 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.174+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcbba400640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14e7940 0x5610a14e4c50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcbac01f6d0 tx=0x7fcbac004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9df48c37ccae574 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcb9b400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcb9b400640 1 -- 172.21.15.146:0/23582167 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac02c060 con 0x5610a14e7940 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcb9b400640 1 -- 172.21.15.146:0/23582167 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcbac017030 con 0x5610a14e7940 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcba80bce30 con 0x5610a14e7940 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcb9b400640 1 -- 172.21.15.146:0/23582167 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcbac011890 con 0x5610a14e7940 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcbbf370740 10 client.25970 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.175+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5610a11543f0 con 0x5610a14e7940 2024-04-24T08:24:05.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcb9b400640 1 -- 172.21.15.146:0/23582167 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba80bce30 con 0x5610a14e7940 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcb9b400640 1 -- 172.21.15.146:0/23582167 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fcbac027c30 con 0x5610a14e7940 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcbbf370740 10 client.25970 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcbbf370740 10 client.25970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcbbf370740 20 client.25970 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcbbf370740 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a14f7870 0x5610a14f9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcbbf370740 4 client.25970 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.176+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610a144fbf0 con 0x5610a14f7870 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.177+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a14f7870 0x5610a14f9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.177+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a14f7870 0x5610a14f9cf0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbd1b5051a42f1f0 server_cookie=457b8861cc809e8d in_seq=0 out_seq=0 2024-04-24T08:24:05.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.177+0000 7fcb9b400640 10 client.25970 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:05.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcb9b400640 1 -- 172.21.15.146:0/23582167 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610a144fbf0 con 0x5610a14f7870 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcb9b400640 10 client.25970 handle_command_reply: tid=0 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 client.25970 shutdown 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14e7940 msgr2=0x5610a14e4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14e7940 0x5610a14e4c50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcbac01f6d0 tx=0x7fcbac004ab0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 shutdown_connections 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a14f7870 0x5610a14f9cf0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14dd270 0x5610a14e4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14e7940 0x5610a14e4c50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcb9b400640 0 client.25970 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 --2- 172.21.15.146:0/23582167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 >> 172.21.15.146:0/23582167 conn(0x5610a14d8c70 msgr2=0x5610a14dea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 shutdown_connections 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.178+0000 7fcbbf370740 1 -- 172.21.15.146:0/23582167 wait complete. 2024-04-24T08:24:05.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.179+0000 7fcbbf370740 20 client.25970 trim_cache size 0 max 16384 2024-04-24T08:24:05.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.195+0000 7fcbbf370740 1 Processor -- start 2024-04-24T08:24:05.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.195+0000 7fcbbf370740 1 -- start start 2024-04-24T08:24:05.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14d7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbbf370740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41b28e0 con 0x5610a14dc950 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbbf370740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41b01b0 con 0x5610a14dd270 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbbf370740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41ae000 con 0x5610a14e7940 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14d7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14d7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39524/0 (socket says 172.21.15.146:39524) 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 -- 172.21.15.146:0/152950001 learned_addr learned my addr 172.21.15.146:0/152950001 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbba400640 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 -- 172.21.15.146:0/152950001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 msgr2=0x5610a14e6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 -- 172.21.15.146:0/152950001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 msgr2=0x5610a14e37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.196+0000 7fcbb9000640 1 -- 172.21.15.146:0/152950001 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610a121e7d0 con 0x5610a14dd270 2024-04-24T08:24:05.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.197+0000 7fcbb9000640 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14d7c70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcbb406b750 tx=0x7fcba0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835088f7ab609048 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:05.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.197+0000 7fcba6000640 1 -- 172.21.15.146:0/152950001 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0004020 con 0x5610a14dd270 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.197+0000 7fcba6000640 1 -- 172.21.15.146:0/152950001 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba0005500 con 0x5610a14dd270 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.197+0000 7fcba6000640 1 -- 172.21.15.146:0/152950001 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0004bb0 con 0x5610a14dd270 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 -- 172.21.15.146:0/152950001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 msgr2=0x5610a14d7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14d7c70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcbb406b750 tx=0x7fcba0009260 comp rx=0 tx=0).stop 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 -- 172.21.15.146:0/152950001 shutdown_connections 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14d7c70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 --2- 172.21.15.146:0/152950001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14e37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 -- 172.21.15.146:0/152950001 >> 172.21.15.146:0/152950001 conn(0x5610a14d8c70 msgr2=0x5610a14d7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 -- 172.21.15.146:0/152950001 shutdown_connections 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 -- 172.21.15.146:0/152950001 wait complete. 2024-04-24T08:24:05.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 -- start start 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.198+0000 7fcbbf370740 1 Processor -- start 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14efab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbbf370740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbbf370740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41ae670 con 0x5610a14dc950 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbbf370740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41afae0 con 0x5610a14dd270 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbbf370740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb41aed40 con 0x5610a14e7940 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbbf370740 10 client.0 _ll_register_callbacks cb 0x7fcbac003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbb9000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbb9000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39534/0 (socket says 172.21.15.146:39534) 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbba400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbb9000640 1 -- 172.21.15.146:0/540529636 learned_addr learned my addr 172.21.15.146:0/540529636 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.199+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbb9000640 1 -- 172.21.15.146:0/540529636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 msgr2=0x5610a14e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbb9000640 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbb9000640 1 -- 172.21.15.146:0/540529636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 msgr2=0x5610a14efab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbb9000640 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14efab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbb9000640 1 -- 172.21.15.146:0/540529636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610a14e2340 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbb9000640 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e18c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009530 tx=0x7fcba0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f23368405138eeb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcb9b400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcb9b400640 1 -- 172.21.15.146:0/540529636 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0004020 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcba0005500 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.200+0000 7fcb9b400640 1 -- 172.21.15.146:0/540529636 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcba00041c0 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcb9b400640 1 -- 172.21.15.146:0/540529636 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcba0036050 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 10 client.25976 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5610a14e8e60 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcb9b400640 1 -- 172.21.15.146:0/540529636 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba0005500 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcb9b400640 1 -- 172.21.15.146:0/540529636 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fcba0003b90 con 0x5610a14dd270 2024-04-24T08:24:05.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 10 client.25976 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:05.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 10 client.25976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:05.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 20 client.25976 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:05.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a155e220 0x5610a1560640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:05.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 4 client.25976 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:24:05.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610a1449a60 con 0x5610a155e220 2024-04-24T08:24:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.201+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a155e220 0x5610a1560640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcb9b400640 10 client.25976 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:05.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbb9a00640 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a155e220 0x5610a1560640 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a21a2241bb181ee6 server_cookie=e8fa510befd85295 in_seq=0 out_seq=0 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcb9b400640 1 -- 172.21.15.146:0/540529636 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610a1449a60 con 0x5610a155e220 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcb9b400640 10 client.25976 handle_command_reply: tid=0 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 client.25976 shutdown 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 msgr2=0x5610a14e18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e18c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009530 tx=0x7fcba0005740 comp rx=0 tx=0).stop 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 shutdown_connections 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5610a155e220 0x5610a1560640 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5610a14e7940 0x5610a14e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5610a14dd270 0x5610a14e18c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 --2- 172.21.15.146:0/540529636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5610a14dc950 0x5610a14efab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcb9b400640 0 client.25976 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 >> 172.21.15.146:0/540529636 conn(0x5610a14d8c70 msgr2=0x5610a14d7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.202+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 shutdown_connections 2024-04-24T08:24:05.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbf370740 1 -- 172.21.15.146:0/540529636 wait complete. 2024-04-24T08:24:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbf370740 20 client.25976 trim_cache size 0 max 16384 2024-04-24T08:24:05.208 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:24:05.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb9c0a3da0 msgr2=0x7fcb9c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb9c0a3da0 0x7fcb9c0a6260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcbac005fd0 tx=0x7fcbac009500 comp rx=0 tx=0).stop 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 msgr2=0x7fcbb41a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb41a07e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcba8035040 tx=0x7fcba8033b60 comp rx=0 tx=0).stop 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 shutdown_connections 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb9c0a3da0 0x7fcb9c0a6260 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcbb419a9e0 0x7fcbb419ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb4109580 0x7fcbb41a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 --2- 172.21.15.146:0/634999478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcbb41044b0 0x7fcbb41a07e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.203+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 >> 172.21.15.146:0/634999478 conn(0x7fcbb40ffd90 msgr2=0x7fcbb41010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.204+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 shutdown_connections 2024-04-24T08:24:05.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:05.204+0000 7fcbbc400640 1 -- 172.21.15.146:0/634999478 wait complete. 2024-04-24T08:24:05.225 INFO:tasks.workunit.client.0.smithi146.stdout:checking for acl_init in -lacl... yes 2024-04-24T08:24:05.326 INFO:tasks.workunit.client.0.smithi146.stdout:checking gdbm-ndbm.h usability... no 2024-04-24T08:24:05.391 INFO:tasks.workunit.client.0.smithi146.stdout:checking gdbm-ndbm.h presence... no 2024-04-24T08:24:05.391 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gdbm-ndbm.h... no 2024-04-24T08:24:05.478 INFO:tasks.workunit.client.0.smithi146.stdout:checking gdbm/ndbm.h usability... no 2024-04-24T08:24:05.543 INFO:tasks.workunit.client.0.smithi146.stdout:checking gdbm/ndbm.h presence... no 2024-04-24T08:24:05.543 INFO:tasks.workunit.client.0.smithi146.stdout:checking for gdbm/ndbm.h... no 2024-04-24T08:24:05.627 INFO:tasks.workunit.client.0.smithi146.stdout:checking ndbm.h usability... no 2024-04-24T08:24:05.690 INFO:tasks.workunit.client.0.smithi146.stdout:checking ndbm.h presence... no 2024-04-24T08:24:05.690 INFO:tasks.workunit.client.0.smithi146.stdout:checking for ndbm.h... no 2024-04-24T08:24:05.767 INFO:tasks.workunit.client.0.smithi146.stdout:checking libaio.h usability... yes 2024-04-24T08:24:05.809 INFO:tasks.workunit.client.0.smithi146.stdout:checking libaio.h presence... yes 2024-04-24T08:24:05.809 INFO:tasks.workunit.client.0.smithi146.stdout:checking for libaio.h... yes 2024-04-24T08:24:05.922 INFO:tasks.workunit.client.0.smithi146.stdout:checking for dm_make_handle in -ldm... no 2024-04-24T08:24:05.922 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-04-24T08:24:05.922 INFO:tasks.workunit.client.0.smithi146.stdout:WARNING: could not find a valid DMAPI base library. 2024-04-24T08:24:05.922 INFO:tasks.workunit.client.0.smithi146.stdout:If you want DMAPI support please install the data migration 2024-04-24T08:24:05.922 INFO:tasks.workunit.client.0.smithi146.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-04-24T08:24:05.922 INFO:tasks.workunit.client.0.smithi146.stdout:from the dmapi source. 2024-04-24T08:24:05.923 INFO:tasks.workunit.client.0.smithi146.stdout: 2024-04-24T08:24:05.998 INFO:tasks.workunit.client.0.smithi146.stdout:checking linux/fiemap.h usability... yes 2024-04-24T08:24:06.034 INFO:tasks.workunit.client.0.smithi146.stdout:checking linux/fiemap.h presence... yes 2024-04-24T08:24:06.034 INFO:tasks.workunit.client.0.smithi146.stdout:checking for linux/fiemap.h... yes 2024-04-24T08:24:06.129 INFO:tasks.workunit.client.0.smithi146.stdout:checking for fallocate... yes 2024-04-24T08:24:06.223 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:24:06.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:24:06.224 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:24:06.241 INFO:tasks.workunit.client.0.smithi146.stdout:checking for open_by_handle_at... yes 2024-04-24T08:24:06.318 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/prctl.h usability... yes 2024-04-24T08:24:06.355 INFO:tasks.workunit.client.0.smithi146.stdout:checking sys/prctl.h presence... yes 2024-04-24T08:24:06.355 INFO:tasks.workunit.client.0.smithi146.stdout:checking for sys/prctl.h... yes 2024-04-24T08:24:06.425 INFO:tasks.workunit.client.0.smithi146.stdout:checking linux/fs.h usability... yes 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 -- 172.21.15.146:0/395283353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 msgr2=0x7f8794111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 --2- 172.21.15.146:0/395283353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f8794111150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f878c009ca0 tx=0x7f878c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 -- 172.21.15.146:0/395283353 shutdown_connections 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 --2- 172.21.15.146:0/395283353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f8794111150 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 --2- 172.21.15.146:0/395283353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8794104e60 0x7f8794108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 --2- 172.21.15.146:0/395283353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f87941044b0 0x7f8794104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 -- 172.21.15.146:0/395283353 >> 172.21.15.146:0/395283353 conn(0x7f87940ffd90 msgr2=0x7f87941021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 -- 172.21.15.146:0/395283353 shutdown_connections 2024-04-24T08:24:06.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.426+0000 7f879c000640 1 -- 172.21.15.146:0/395283353 wait complete. 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 Processor -- start 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 -- start start 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87941044b0 0x7f87941a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f87941a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879419a9e0 0x7f879419ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8794114640 con 0x7f87941044b0 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87941144c0 con 0x7f8794109580 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.427+0000 7f879c000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f87941147c0 con 0x7f879419a9e0 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f87941a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8799600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87941044b0 0x7f87941a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f87941a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39546/0 (socket says 172.21.15.146:39546) 2024-04-24T08:24:06.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 -- 172.21.15.146:0/3224339660 learned_addr learned my addr 172.21.15.146:0/3224339660 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f879a000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879419a9e0 0x7f879419ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 -- 172.21.15.146:0/3224339660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879419a9e0 msgr2=0x7f879419ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879419a9e0 0x7f879419ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 -- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87941044b0 msgr2=0x7f87941a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87941044b0 0x7f87941a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.428+0000 7f8798c00640 1 -- 172.21.15.146:0/3224339660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879419b500 con 0x7f8794109580 2024-04-24T08:24:06.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.429+0000 7f8798c00640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f87941a0d20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f878000e9c0 tx=0x7f878000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.429+0000 7f8792000640 1 -- 172.21.15.146:0/3224339660 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878000cd80 con 0x7f8794109580 2024-04-24T08:24:06.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.429+0000 7f8792000640 1 -- 172.21.15.146:0/3224339660 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8780005500 con 0x7f8794109580 2024-04-24T08:24:06.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.429+0000 7f8792000640 1 -- 172.21.15.146:0/3224339660 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8780010690 con 0x7f8794109580 2024-04-24T08:24:06.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.429+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879419b7f0 con 0x7f8794109580 2024-04-24T08:24:06.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.429+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87941a79a0 con 0x7f8794109580 2024-04-24T08:24:06.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.432+0000 7f879ee11740 1 Processor -- start 2024-04-24T08:24:06.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f8792000640 1 -- 172.21.15.146:0/3224339660 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f87800046c0 con 0x7f8794109580 2024-04-24T08:24:06.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879ee11740 1 -- start start 2024-04-24T08:24:06.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879ee11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd090 0x55c138cdd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879ee11740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879ee11740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138cea470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879ee11740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c138c42e70 con 0x55c138ce8080 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879ee11740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c138be9380 con 0x55c138cdd9b0 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879ee11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c138c3ee30 con 0x55c138cdd090 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f8799600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f8798c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138cea470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd090 0x55c138cdd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f8799600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39560/0 (socket says 172.21.15.146:39560) 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f8799600640 1 -- 172.21.15.146:0/3682487518 learned_addr learned my addr 172.21.15.146:0/3682487518 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.433+0000 7f879a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd090 0x55c138cdd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54410/0 (socket says 172.21.15.146:54410) 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f879a000640 1 -- 172.21.15.146:0/3682487518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 msgr2=0x55c138ce7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f879a000640 1 --2- 172.21.15.146:0/3682487518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f879a000640 1 -- 172.21.15.146:0/3682487518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 msgr2=0x55c138cea470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f879a000640 1 --2- 172.21.15.146:0/3682487518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138cea470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f879a000640 1 -- 172.21.15.146:0/3682487518 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c138a1d7e0 con 0x55c138cdd090 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f8792000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f877c0a3da0 0x7f877c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f879a000640 1 --2- 172.21.15.146:0/3682487518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd090 0x55c138cdd470 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f878c006010 tx=0x7f878c005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f722c6e63bfc220 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f8792000640 1 -- 172.21.15.146:0/3224339660 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f87800d0a80 con 0x7f8794109580 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.434+0000 7f8786600640 1 -- 172.21.15.146:0/3682487518 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c031db0 con 0x55c138cdd090 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.435+0000 7f8786600640 1 -- 172.21.15.146:0/3682487518 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878c036040 con 0x55c138cdd090 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.435+0000 7f8799600640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f877c0a3da0 0x7f877c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.435+0000 7f8786600640 1 -- 172.21.15.146:0/3682487518 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c004240 con 0x55c138cdd090 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.435+0000 7f879ee11740 1 -- 172.21.15.146:0/3682487518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd090 msgr2=0x55c138cdd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.435+0000 7f8799600640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f877c0a3da0 0x7f877c0a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55c138cde650 tx=0x7f8788005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.435+0000 7f879ee11740 1 --2- 172.21.15.146:0/3682487518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd090 0x55c138cdd470 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f878c006010 tx=0x7f878c005f10 comp rx=0 tx=0).stop 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 -- 172.21.15.146:0/3682487518 shutdown_connections 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 --2- 172.21.15.146:0/3682487518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138cea470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 --2- 172.21.15.146:0/3682487518 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 --2- 172.21.15.146:0/3682487518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd090 0x55c138cdd470 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 -- 172.21.15.146:0/3682487518 >> 172.21.15.146:0/3682487518 conn(0x55c138cd93b0 msgr2=0x55c138cd9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 -- 172.21.15.146:0/3682487518 shutdown_connections 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 -- 172.21.15.146:0/3682487518 wait complete. 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 -- start start 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.436+0000 7f879ee11740 1 Processor -- start 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879ee11740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138cdd090 0x55c138ce5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879ee11740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879ee11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138ce8080 0x55c138ce5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879ee11740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c138cfbca0 con 0x55c138cdd090 2024-04-24T08:24:06.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879ee11740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c138bfc660 con 0x55c138cdd9b0 2024-04-24T08:24:06.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879ee11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c138cfbe20 con 0x55c138ce8080 2024-04-24T08:24:06.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879ee11740 10 client.0 _ll_register_callbacks cb 0x55c138beaaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:06.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138ce8080 0x55c138ce5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f8799600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138cdd090 0x55c138ce5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138ce8080 0x55c138ce5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54418/0 (socket says 172.21.15.146:54418) 2024-04-24T08:24:06.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879a000640 1 -- 172.21.15.146:0/967042726 learned_addr learned my addr 172.21.15.146:0/967042726 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f8798c00640 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879a000640 1 -- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 msgr2=0x55c138ce5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.437+0000 7f879a000640 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f879a000640 1 -- 172.21.15.146:0/967042726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138cdd090 msgr2=0x55c138ce5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f879a000640 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138cdd090 0x55c138ce5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f879a000640 1 -- 172.21.15.146:0/967042726 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c138a7df70 con 0x55c138ce8080 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f879a000640 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138ce8080 0x55c138ce5ac0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f878c02a900 tx=0x7f878c032ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d207550a76acc5b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f877aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f877aa00640 1 -- 172.21.15.146:0/967042726 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c003b80 con 0x55c138ce8080 2024-04-24T08:24:06.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f878c036040 con 0x55c138ce8080 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.438+0000 7f877aa00640 1 -- 172.21.15.146:0/967042726 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f878c003d20 con 0x55c138ce8080 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f877aa00640 1 -- 172.21.15.146:0/967042726 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f878c05b050 con 0x55c138ce8080 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 10 client.25972 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c138915170 con 0x55c138ce8080 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f877aa00640 1 -- 172.21.15.146:0/967042726 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878c036040 con 0x55c138ce8080 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f877aa00640 1 -- 172.21.15.146:0/967042726 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f878c04c070 con 0x55c138ce8080 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 10 client.25972 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 10 client.25972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 20 client.25972 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d090e0 0x55c138d0b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 4 client.25972 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.439+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c138c50890 con 0x55c138d090e0 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.440+0000 7f8799600640 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d090e0 0x55c138d0b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.440+0000 7f8799600640 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d090e0 0x55c138d0b4d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42cef62cb7cd397d server_cookie=1e33071e57650af2 in_seq=0 out_seq=0 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.440+0000 7f877aa00640 10 client.25972 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f877aa00640 1 -- 172.21.15.146:0/967042726 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c138c50890 con 0x55c138d090e0 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f877aa00640 10 client.25972 handle_command_reply: tid=0 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 client.25972 shutdown 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138ce8080 msgr2=0x55c138ce5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138ce8080 0x55c138ce5ac0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f878c02a900 tx=0x7f878c032ef0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 shutdown_connections 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d090e0 0x55c138d0b4d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138ce8080 0x55c138ce5ac0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd9b0 0x55c138ce5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f877aa00640 0 client.25972 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 --2- 172.21.15.146:0/967042726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138cdd090 0x55c138ce5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 >> 172.21.15.146:0/967042726 conn(0x55c138cd93b0 msgr2=0x55c138cdf1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 shutdown_connections 2024-04-24T08:24:06.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.441+0000 7f879ee11740 1 -- 172.21.15.146:0/967042726 wait complete. 2024-04-24T08:24:06.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.442+0000 7f879ee11740 20 client.25972 trim_cache size 0 max 16384 2024-04-24T08:24:06.458 INFO:tasks.workunit.client.0.smithi146.stdout:checking linux/fs.h presence... yes 2024-04-24T08:24:06.458 INFO:tasks.workunit.client.0.smithi146.stdout:checking for linux/fs.h... yes 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.463+0000 7f879ee11740 1 Processor -- start 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.463+0000 7f879ee11740 1 -- start start 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.463+0000 7f879ee11740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138d04b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f879ee11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138cd8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f879ee11740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d10bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f879ee11740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f87941aa030 con 0x55c138ce8080 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f879ee11740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8794073ec0 con 0x55c138cdd090 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f879ee11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8794071d10 con 0x55c138cdd9b0 2024-04-24T08:24:06.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138d04b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138d04b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39578/0 (socket says 172.21.15.146:39578) 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 -- 172.21.15.146:0/2098346490 learned_addr learned my addr 172.21.15.146:0/2098346490 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8798c00640 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138cd8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f879a000640 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d10bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 -- 172.21.15.146:0/2098346490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 msgr2=0x55c138cd8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138cd8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 -- 172.21.15.146:0/2098346490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 msgr2=0x55c138d10bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d10bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.464+0000 7f8799600640 1 -- 172.21.15.146:0/2098346490 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c138cd99c0 con 0x55c138cdd090 2024-04-24T08:24:06.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.465+0000 7f8799600640 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138d04b60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8788012cb0 tx=0x7f878801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f2fd839bf8ef38 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.465+0000 7f8785c00640 1 -- 172.21.15.146:0/2098346490 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788030070 con 0x55c138cdd090 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.465+0000 7f8785c00640 1 -- 172.21.15.146:0/2098346490 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878802b420 con 0x55c138cdd090 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.465+0000 7f8785c00640 1 -- 172.21.15.146:0/2098346490 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878801f8c0 con 0x55c138cdd090 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 -- 172.21.15.146:0/2098346490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 msgr2=0x55c138d04b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138d04b60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8788012cb0 tx=0x7f878801b000 comp rx=0 tx=0).stop 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 -- 172.21.15.146:0/2098346490 shutdown_connections 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138cd8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d10bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 --2- 172.21.15.146:0/2098346490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138d04b60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 -- 172.21.15.146:0/2098346490 >> 172.21.15.146:0/2098346490 conn(0x55c138cd93b0 msgr2=0x55c138cd7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 -- 172.21.15.146:0/2098346490 shutdown_connections 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 -- 172.21.15.146:0/2098346490 wait complete. 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.466+0000 7f879ee11740 1 -- start start 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 1 Processor -- start 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138ce20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138ce2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d6e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8794072380 con 0x55c138ce8080 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87940737f0 con 0x55c138cdd090 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8794072a50 con 0x55c138cdd9b0 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f879ee11740 10 client.0 _ll_register_callbacks cb 0x7f87800d44a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f8799600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138ce20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f8799600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138ce20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39592/0 (socket says 172.21.15.146:39592) 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.467+0000 7f8799600640 1 -- 172.21.15.146:0/2929231724 learned_addr learned my addr 172.21.15.146:0/2929231724 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8798c00640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138ce2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f879a000640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d6e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8799600640 1 -- 172.21.15.146:0/2929231724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 msgr2=0x55c138ce2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8799600640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138ce2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8799600640 1 -- 172.21.15.146:0/2929231724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 msgr2=0x55c138d6e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8799600640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d6e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8799600640 1 -- 172.21.15.146:0/2929231724 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c138d6e9a0 con 0x55c138cdd090 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8798c00640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138ce2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f8799600640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138ce20d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8788004650 tx=0x7f878801f4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5289d0fe61ca0054 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.468+0000 7f877aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:06.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.471+0000 7f877aa00640 1 -- 172.21.15.146:0/2929231724 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8788030070 con 0x55c138cdd090 2024-04-24T08:24:06.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.471+0000 7f877aa00640 1 -- 172.21.15.146:0/2929231724 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8788003a70 con 0x55c138cdd090 2024-04-24T08:24:06.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.471+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f878802b420 con 0x55c138cdd090 2024-04-24T08:24:06.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.472+0000 7f877aa00640 1 -- 172.21.15.146:0/2929231724 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8788043050 con 0x55c138cdd090 2024-04-24T08:24:06.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.472+0000 7f879ee11740 10 client.25994 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.472+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c138d6fb40 con 0x55c138cdd090 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.472+0000 7f877aa00640 1 -- 172.21.15.146:0/2929231724 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878802b420 con 0x55c138cdd090 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.472+0000 7f877aa00640 1 -- 172.21.15.146:0/2929231724 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f8788004750 con 0x55c138cdd090 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f879ee11740 10 client.25994 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f879ee11740 10 client.25994 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f879ee11740 20 client.25994 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f879ee11740 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d701d0 0x55c138d725f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f879ee11740 4 client.25994 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c138c4a700 con 0x55c138d701d0 2024-04-24T08:24:06.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f8798c00640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d701d0 0x55c138d725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f8798c00640 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d701d0 0x55c138d725f0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285c909388bdf1d6 server_cookie=b00478552ae1094f in_seq=0 out_seq=0 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.473+0000 7f877aa00640 10 client.25994 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.474+0000 7f877aa00640 1 -- 172.21.15.146:0/2929231724 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c138c4a700 con 0x55c138d701d0 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.474+0000 7f877aa00640 10 client.25994 handle_command_reply: tid=0 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.474+0000 7f879ee11740 1 client.25994 shutdown 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 msgr2=0x55c138ce20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138ce20d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8788004650 tx=0x7f878801f4c0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 shutdown_connections 2024-04-24T08:24:06.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c138d701d0 0x55c138d725f0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c138cdd9b0 0x55c138ce2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c138ce8080 0x55c138d6e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f877aa00640 0 client.25994 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 --2- 172.21.15.146:0/2929231724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c138cdd090 0x55c138ce20d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 >> 172.21.15.146:0/2929231724 conn(0x55c138cd93b0 msgr2=0x55c138ce7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 shutdown_connections 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 1 -- 172.21.15.146:0/2929231724 wait complete. 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.475+0000 7f879ee11740 20 client.25994 trim_cache size 0 max 16384 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:24:06.477 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:24:06.478 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "d2ad23be-2826-4f3b-b8a4-f1c6ce2101c1", 2024-04-24T08:24:06.478 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:24:06.478 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:24:06.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f877c0a3da0 msgr2=0x7f877c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f877c0a3da0 0x7f877c0a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55c138cde650 tx=0x7f8788005f90 comp rx=0 tx=0).stop 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 msgr2=0x7f87941a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f87941a0d20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f878000e9c0 tx=0x7f878000ee80 comp rx=0 tx=0).stop 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 shutdown_connections 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f877c0a3da0 0x7f877c0a6260 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879419a9e0 0x7f879419ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8794109580 0x7f87941a0d20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 --2- 172.21.15.146:0/3224339660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87941044b0 0x7f87941a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.476+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 >> 172.21.15.146:0/3224339660 conn(0x7f87940ffd90 msgr2=0x7f8794102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.477+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 shutdown_connections 2024-04-24T08:24:06.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.477+0000 7f879c000640 1 -- 172.21.15.146:0/3224339660 wait complete. 2024-04-24T08:24:06.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:24:06.498 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:24:06.588 INFO:tasks.workunit.client.0.smithi146.stdout:checking btrfsutil.h usability... no 2024-04-24T08:24:06.603 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:06 smithi146 ceph-mon[27805]: pgmap v90: 97 pgs: 97 active+clean; 65 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 735 KiB/s wr, 137 op/s 2024-04-24T08:24:06.661 INFO:tasks.workunit.client.0.smithi146.stdout:checking btrfsutil.h presence... no 2024-04-24T08:24:06.662 INFO:tasks.workunit.client.0.smithi146.stdout:checking for btrfsutil.h... no 2024-04-24T08:24:06.798 INFO:tasks.workunit.client.0.smithi146.stdout:checking for copy_file_range... yes 2024-04-24T08:24:06.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.803+0000 7f3298a00640 1 -- 172.21.15.146:0/3356864340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3294102240 msgr2=0x7f3294102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.803+0000 7f3298a00640 1 --2- 172.21.15.146:0/3356864340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3294102240 0x7f3294102620 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3280009900 tx=0x7f328002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.803+0000 7f3298a00640 1 -- 172.21.15.146:0/3356864340 shutdown_connections 2024-04-24T08:24:06.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.803+0000 7f3298a00640 1 --2- 172.21.15.146:0/3356864340 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3294107310 0x7f329410eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.803+0000 7f3298a00640 1 --2- 172.21.15.146:0/3356864340 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3294102bf0 0x7f3294106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.803+0000 7f3298a00640 1 --2- 172.21.15.146:0/3356864340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3294102240 0x7f3294102620 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.803+0000 7f3298a00640 1 -- 172.21.15.146:0/3356864340 >> 172.21.15.146:0/3356864340 conn(0x7f3294079450 msgr2=0x7f3294079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.804+0000 7f3298a00640 1 -- 172.21.15.146:0/3356864340 shutdown_connections 2024-04-24T08:24:06.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.804+0000 7f3298a00640 1 -- 172.21.15.146:0/3356864340 wait complete. 2024-04-24T08:24:06.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.806+0000 7f3298a00640 1 Processor -- start 2024-04-24T08:24:06.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.806+0000 7f3298a00640 1 -- start start 2024-04-24T08:24:06.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.807+0000 7f3298a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3294102240 0x7f329410f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.807+0000 7f3298a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3294107310 0x7f329410fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.807+0000 7f3298a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32941100f0 0x7f32941ab020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.807+0000 7f3298a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3294114650 con 0x7f3294102240 2024-04-24T08:24:06.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.807+0000 7f3298a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32941144d0 con 0x7f3294107310 2024-04-24T08:24:06.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.807+0000 7f3298a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32941147d0 con 0x7f32941100f0 2024-04-24T08:24:06.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.808+0000 7f3292800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32941100f0 0x7f32941ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.808+0000 7f3291400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3294107310 0x7f329410fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.808+0000 7f3292800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32941100f0 0x7f32941ab020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54460/0 (socket says 172.21.15.146:54460) 2024-04-24T08:24:06.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.808+0000 7f3292800640 1 -- 172.21.15.146:0/128535665 learned_addr learned my addr 172.21.15.146:0/128535665 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f3292800640 1 -- 172.21.15.146:0/128535665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3294107310 msgr2=0x7f329410fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f3292800640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3294107310 0x7f329410fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f3292800640 1 -- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3294102240 msgr2=0x7f329410f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f3292800640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3294102240 0x7f329410f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f3292800640 1 -- 172.21.15.146:0/128535665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32941ab560 con 0x7f32941100f0 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f3291400640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3294107310 0x7f329410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f3292800640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32941100f0 0x7f32941ab020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f328c00eaa0 tx=0x7f328c00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f328aa00640 1 -- 172.21.15.146:0/128535665 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c004d20 con 0x7f32941100f0 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f328aa00640 1 -- 172.21.15.146:0/128535665 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f328c005020 con 0x7f32941100f0 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.809+0000 7f328aa00640 1 -- 172.21.15.146:0/128535665 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c0116d0 con 0x7f32941100f0 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.810+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32941ab850 con 0x7f32941100f0 2024-04-24T08:24:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.810+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32941abd20 con 0x7f32941100f0 2024-04-24T08:24:06.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:06 smithi177 ceph-mon[33046]: pgmap v90: 97 pgs: 97 active+clean; 65 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 735 KiB/s wr, 137 op/s 2024-04-24T08:24:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.816+0000 7f329b7c0740 1 Processor -- start 2024-04-24T08:24:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.816+0000 7f328aa00640 1 -- 172.21.15.146:0/128535665 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f328c0051d0 con 0x7f32941100f0 2024-04-24T08:24:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.817+0000 7f329b7c0740 1 -- start start 2024-04-24T08:24:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.817+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d547c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.817+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d548180 0x55800d552310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.817+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d552850 0x55800d554c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.817+0000 7f329b7c0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55800d4ad1f0 con 0x55800d547860 2024-04-24T08:24:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.817+0000 7f329b7c0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55800d453350 con 0x55800d548180 2024-04-24T08:24:06.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.817+0000 7f329b7c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55800d4a91b0 con 0x55800d552850 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d548180 0x55800d552310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d548180 0x55800d552310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39628/0 (socket says 172.21.15.146:39628) 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 -- 172.21.15.146:0/2415098974 learned_addr learned my addr 172.21.15.146:0/2415098974 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291400640 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d547c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 -- 172.21.15.146:0/2415098974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d552850 msgr2=0x55800d554c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d552850 0x55800d554c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 -- 172.21.15.146:0/2415098974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 msgr2=0x55800d547c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d547c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291e00640 1 -- 172.21.15.146:0/2415098974 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800d4a7f50 con 0x55800d548180 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.818+0000 7f3291400640 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d547c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:24:06.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.819+0000 7f3291e00640 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d548180 0x55800d552310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3280009230 tx=0x7f32800314b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb3c1acd611abbbe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.819+0000 7f328aa00640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32780a3e60 0x7f32780a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.819+0000 7f328aa00640 1 -- 172.21.15.146:0/128535665 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f328c0d0c40 con 0x7f32941100f0 2024-04-24T08:24:06.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.819+0000 7f327f000640 1 -- 172.21.15.146:0/2415098974 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328003e030 con 0x55800d548180 2024-04-24T08:24:06.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.819+0000 7f327f000640 1 -- 172.21.15.146:0/2415098974 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32800035f0 con 0x55800d548180 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.819+0000 7f327f000640 1 -- 172.21.15.146:0/2415098974 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3280035040 con 0x55800d548180 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f3291400640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32780a3e60 0x7f32780a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f329b7c0740 1 -- 172.21.15.146:0/2415098974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d548180 msgr2=0x55800d552310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f329b7c0740 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d548180 0x55800d552310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3280009230 tx=0x7f32800314b0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f329b7c0740 1 -- 172.21.15.146:0/2415098974 shutdown_connections 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f329b7c0740 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d552850 0x55800d554c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f329b7c0740 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d548180 0x55800d552310 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f3291400640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32780a3e60 0x7f32780a6320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55800d544d10 tx=0x7f328400a2d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f329b7c0740 1 --2- 172.21.15.146:0/2415098974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d547c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.820+0000 7f329b7c0740 1 -- 172.21.15.146:0/2415098974 >> 172.21.15.146:0/2415098974 conn(0x55800d543b80 msgr2=0x55800d543f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.821+0000 7f329b7c0740 1 -- 172.21.15.146:0/2415098974 shutdown_connections 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.821+0000 7f329b7c0740 1 -- 172.21.15.146:0/2415098974 wait complete. 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.821+0000 7f329b7c0740 1 -- start start 2024-04-24T08:24:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.821+0000 7f329b7c0740 1 Processor -- start 2024-04-24T08:24:06.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d54f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d54fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.824 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d550230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f3291e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d54f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f329b7c0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55800d5555e0 con 0x55800d548180 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f3291e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d54f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54462/0 (socket says 172.21.15.146:54462) 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f3291e00640 1 -- 172.21.15.146:0/1276689102 learned_addr learned my addr 172.21.15.146:0/1276689102 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f3291400640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d54fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f3292800640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d550230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55800d466690 con 0x55800d552850 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55800d555760 con 0x55800d547860 2024-04-24T08:24:06.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.822+0000 7f329b7c0740 10 client.0 _ll_register_callbacks cb 0x55800d4a2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3292800640 1 -- 172.21.15.146:0/1276689102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 msgr2=0x55800d54f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3292800640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d54f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3292800640 1 -- 172.21.15.146:0/1276689102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 msgr2=0x55800d54fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3292800640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d54fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3292800640 1 -- 172.21.15.146:0/1276689102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800d465920 con 0x55800d552850 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3291e00640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d54f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3292800640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d550230 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f328c0ce340 tx=0x7f328c09c570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d5c80e772b46b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3273400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3273400640 1 -- 172.21.15.146:0/1276689102 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c0a0f40 con 0x55800d552850 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32800035f0 con 0x55800d552850 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.823+0000 7f3273400640 1 -- 172.21.15.146:0/1276689102 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f328c0ce5c0 con 0x55800d552850 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f3273400640 1 -- 172.21.15.146:0/1276689102 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f328c0ce7c0 con 0x55800d552850 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 10 client.26003 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:06.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55800d2897d0 con 0x55800d552850 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f3273400640 1 -- 172.21.15.146:0/1276689102 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32800035f0 con 0x55800d552850 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f3273400640 1 -- 172.21.15.146:0/1276689102 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f328c00f040 con 0x55800d552850 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 10 client.26003 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 10 client.26003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 20 client.26003 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d562ac0 0x55800d564f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 4 client.26003 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.824+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55800d4bac10 con 0x55800d562ac0 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.825+0000 7f3291e00640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d562ac0 0x55800d564f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.825+0000 7f3291e00640 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d562ac0 0x55800d564f20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de0f78261b10c2d server_cookie=df1c7a2a10c28522 in_seq=0 out_seq=0 2024-04-24T08:24:06.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.825+0000 7f3273400640 10 client.26003 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f3273400640 1 -- 172.21.15.146:0/1276689102 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55800d4bac10 con 0x55800d562ac0 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f3273400640 10 client.26003 handle_command_reply: tid=0 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 client.26003 shutdown 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 msgr2=0x55800d550230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d550230 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f328c0ce340 tx=0x7f328c09c570 comp rx=0 tx=0).stop 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 shutdown_connections 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d562ac0 0x55800d564f20 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d550230 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d54fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 --2- 172.21.15.146:0/1276689102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d54f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.826+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 >> 172.21.15.146:0/1276689102 conn(0x55800d543b80 msgr2=0x55800d549840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.827+0000 7f3273400640 0 client.26003 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.827+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 shutdown_connections 2024-04-24T08:24:06.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.827+0000 7f329b7c0740 1 -- 172.21.15.146:0/1276689102 wait complete. 2024-04-24T08:24:06.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.827+0000 7f329b7c0740 20 client.26003 trim_cache size 0 max 16384 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 Processor -- start 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 -- start start 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d55e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d53ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d5496c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f32941b6c10 con 0x55800d548180 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32941b44e0 con 0x55800d552850 2024-04-24T08:24:06.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.862+0000 7f329b7c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32941b2330 con 0x55800d547860 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d55e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d55e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:54472/0 (socket says 172.21.15.146:54472) 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 -- 172.21.15.146:0/3706483132 learned_addr learned my addr 172.21.15.146:0/3706483132 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3292800640 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d5496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291400640 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d53ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 -- 172.21.15.146:0/3706483132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 msgr2=0x55800d5496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d5496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 -- 172.21.15.146:0/3706483132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 msgr2=0x55800d53ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d53ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 -- 172.21.15.146:0/3706483132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800d547fb0 con 0x55800d547860 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291400640 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d53ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f3291e00640 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d55e4d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f32800058c0 tx=0x7f3280005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bb5ba6700eaf4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f327e600640 1 -- 172.21.15.146:0/3706483132 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3280051070 con 0x55800d547860 2024-04-24T08:24:06.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f327e600640 1 -- 172.21.15.146:0/3706483132 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f328004c8b0 con 0x55800d547860 2024-04-24T08:24:06.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.863+0000 7f327e600640 1 -- 172.21.15.146:0/3706483132 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328003e040 con 0x55800d547860 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 -- 172.21.15.146:0/3706483132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 msgr2=0x55800d55e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d55e4d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f32800058c0 tx=0x7f3280005850 comp rx=0 tx=0).stop 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 -- 172.21.15.146:0/3706483132 shutdown_connections 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d5496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d548180 0x55800d53ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3706483132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d547860 0x55800d55e4d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 -- 172.21.15.146:0/3706483132 >> 172.21.15.146:0/3706483132 conn(0x55800d543b80 msgr2=0x55800d548fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 -- 172.21.15.146:0/3706483132 shutdown_connections 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.864+0000 7f329b7c0740 1 -- 172.21.15.146:0/3706483132 wait complete. 2024-04-24T08:24:06.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.865+0000 7f329b7c0740 1 -- start start 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.865+0000 7f329b7c0740 1 Processor -- start 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.865+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d5563f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d548180 0x55800d556930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f329b7c0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d556e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f329b7c0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f32941b29a0 con 0x55800d547860 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f329b7c0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f32941b3e10 con 0x55800d552850 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f329b7c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32941b3070 con 0x55800d548180 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f329b7c0740 10 client.0 _ll_register_callbacks cb 0x7f328c005b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3291e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d5563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d556e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3291400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d548180 0x55800d556930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d556e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39654/0 (socket says 172.21.15.146:39654) 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3291e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d5563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:33666/0 (socket says 172.21.15.146:33666) 2024-04-24T08:24:06.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 -- 172.21.15.146:0/3552163931 learned_addr learned my addr 172.21.15.146:0/3552163931 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 -- 172.21.15.146:0/3552163931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d548180 msgr2=0x55800d556930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d548180 0x55800d556930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 -- 172.21.15.146:0/3552163931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 msgr2=0x55800d5563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d5563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.866+0000 7f3292800640 1 -- 172.21.15.146:0/3552163931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800d1bf3f0 con 0x55800d552850 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f3292800640 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d556e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f328c099560 tx=0x7f328c009710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1570158e8da03e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f3273400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f3273400640 1 -- 172.21.15.146:0/3552163931 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c09c6d0 con 0x55800d552850 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f328004c8b0 con 0x55800d552850 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f3273400640 1 -- 172.21.15.146:0/3552163931 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f328c09bfe0 con 0x55800d552850 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f3273400640 1 -- 172.21.15.146:0/3552163931 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f328c098e30 con 0x55800d552850 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f329b7c0740 10 client.26009 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55800d17f0f0 con 0x55800d552850 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.867+0000 7f3273400640 1 -- 172.21.15.146:0/3552163931 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f328004c8b0 con 0x55800d552850 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f3273400640 1 -- 172.21.15.146:0/3552163931 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f328c0b11f0 con 0x55800d552850 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f329b7c0740 10 client.26009 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f329b7c0740 10 client.26009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f329b7c0740 20 client.26009 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d5c9720 0x55800d5cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f329b7c0740 4 client.26009 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55800d4b4a80 con 0x55800d5c9720 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f3291e00640 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d5c9720 0x55800d5cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f3273400640 10 client.26009 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.868+0000 7f3291e00640 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d5c9720 0x55800d5cbb10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3841c4b66aacfd42 server_cookie=5639f6e49619aa98 in_seq=0 out_seq=0 2024-04-24T08:24:06.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f3273400640 1 -- 172.21.15.146:0/3552163931 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55800d4b4a80 con 0x55800d5c9720 2024-04-24T08:24:06.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f3273400640 10 client.26009 handle_command_reply: tid=0 2024-04-24T08:24:06.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 client.26009 shutdown 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 msgr2=0x55800d556e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d556e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f328c099560 tx=0x7f328c009710 comp rx=0 tx=0).stop 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 shutdown_connections 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55800d5c9720 0x55800d5cbb10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55800d548180 0x55800d556930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55800d552850 0x55800d556e70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 --2- 172.21.15.146:0/3552163931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55800d547860 0x55800d5563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 >> 172.21.15.146:0/3552163931 conn(0x55800d543b80 msgr2=0x55800d549840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f3273400640 0 client.26009 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 shutdown_connections 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 1 -- 172.21.15.146:0/3552163931 wait complete. 2024-04-24T08:24:06.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.869+0000 7f329b7c0740 20 client.26009 trim_cache size 0 max 16384 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "scrub active (175 inodes in the stack)", 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: "d2ad23be-2826-4f3b-b8a4-f1c6ce2101c1": { 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "d2ad23be-2826-4f3b-b8a4-f1c6ce2101c1", 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:24:06.875 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:24:06.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32780a3e60 msgr2=0x7f32780a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32780a3e60 0x7f32780a6320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55800d544d10 tx=0x7f328400a2d0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32941100f0 msgr2=0x7f32941ab020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:06.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32941100f0 0x7f32941ab020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f328c00eaa0 tx=0x7f328c00eea0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 shutdown_connections 2024-04-24T08:24:06.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f32780a3e60 0x7f32780a6320 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32941100f0 0x7f32941ab020 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3294107310 0x7f329410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 --2- 172.21.15.146:0/128535665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3294102240 0x7f329410f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:06.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 >> 172.21.15.146:0/128535665 conn(0x7f3294079450 msgr2=0x7f32940fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:06.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 shutdown_connections 2024-04-24T08:24:06.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:06.870+0000 7f3298a00640 1 -- 172.21.15.146:0/128535665 wait complete. 2024-04-24T08:24:06.899 INFO:tasks.cephfs.filesystem:scrub status for tag:d2ad23be-2826-4f3b-b8a4-f1c6ce2101c1 - {'path': '/', 'tag': 'd2ad23be-2826-4f3b-b8a4-f1c6ce2101c1', 'options': 'recursive,force'} 2024-04-24T08:24:06.918 INFO:tasks.workunit.client.0.smithi146.stdout:checking for renameat2... yes 2024-04-24T08:24:06.952 INFO:tasks.workunit.client.0.smithi146.stdout:configure: creating ./config.status 2024-04-24T08:24:06.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:06 smithi153 ceph-mon[32746]: pgmap v90: 97 pgs: 97 active+clean; 65 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 735 KiB/s wr, 137 op/s 2024-04-24T08:24:07.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:24:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:24:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:24:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:24:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:07 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:24:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:24:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:24:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:24:07.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:07 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:24:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:24:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:24:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:24:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:07 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:08.097 INFO:tasks.workunit.client.0.smithi146.stdout:config.status: creating include/builddefs 2024-04-24T08:24:08.150 INFO:tasks.workunit.client.0.smithi146.stdout:config.status: creating include/config.h 2024-04-24T08:24:08.191 INFO:tasks.workunit.client.0.smithi146.stdout:config.status: executing libtool commands 2024-04-24T08:24:08.359 INFO:tasks.workunit.client.0.smithi146.stdout:Building include 2024-04-24T08:24:08.360 INFO:tasks.workunit.client.0.smithi146.stdout:Building lib 2024-04-24T08:24:08.360 INFO:tasks.workunit.client.0.smithi146.stdout:Building tests 2024-04-24T08:24:08.373 INFO:tasks.workunit.client.0.smithi146.stdout: [LTDEP] 2024-04-24T08:24:08.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:08 smithi146 ceph-mon[27805]: pgmap v91: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 904 KiB/s wr, 145 op/s 2024-04-24T08:24:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:08 smithi177 ceph-mon[33046]: pgmap v91: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 904 KiB/s wr, 145 op/s 2024-04-24T08:24:08.842 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] dataascii.lo 2024-04-24T08:24:08.842 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] databin.lo 2024-04-24T08:24:08.843 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] datapid.lo 2024-04-24T08:24:08.843 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] file_lock.lo 2024-04-24T08:24:08.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:08 smithi153 ceph-mon[32746]: pgmap v91: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 904 KiB/s wr, 145 op/s 2024-04-24T08:24:09.128 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] forker.lo 2024-04-24T08:24:09.128 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] pattern.lo 2024-04-24T08:24:09.191 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] open_flags.lo 2024-04-24T08:24:09.208 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] random_range.lo 2024-04-24T08:24:09.406 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] string_to_tokens.lo 2024-04-24T08:24:09.453 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] str_to_bytes.lo 2024-04-24T08:24:09.504 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] tlibio.lo 2024-04-24T08:24:09.666 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] write_log.lo 2024-04-24T08:24:09.679 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] random.lo 2024-04-24T08:24:10.098 INFO:tasks.workunit.client.0.smithi146.stdout: [LD] 2024-04-24T08:24:10.394 INFO:tasks.workunit.client.0.smithi146.stdout:Building ltp 2024-04-24T08:24:10.395 INFO:tasks.workunit.client.0.smithi146.stdout:Building src 2024-04-24T08:24:10.395 INFO:tasks.workunit.client.0.smithi146.stdout:Building m4 2024-04-24T08:24:10.396 INFO:tasks.workunit.client.0.smithi146.stdout:Building common 2024-04-24T08:24:10.409 INFO:tasks.workunit.client.0.smithi146.stdout: [DEP] 2024-04-24T08:24:10.434 INFO:tasks.workunit.client.0.smithi146.stdout: [DEP] 2024-04-24T08:24:10.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:10 smithi146 ceph-mon[27805]: pgmap v92: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 316 KiB/s wr, 32 op/s 2024-04-24T08:24:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:10 smithi177 ceph-mon[33046]: pgmap v92: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 316 KiB/s wr, 32 op/s 2024-04-24T08:24:10.821 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] doio 2024-04-24T08:24:10.822 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] fsstress 2024-04-24T08:24:10.822 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] fsx 2024-04-24T08:24:10.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:10 smithi153 ceph-mon[32746]: pgmap v92: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 316 KiB/s wr, 32 op/s 2024-04-24T08:24:11.099 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c: In function 'attr_list_path': 2024-04-24T08:24:11.099 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-04-24T08:24:11.099 INFO:tasks.workunit.client.0.smithi146.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~ 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:In file included from ../src/global.h:25, 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr: from fsstress.c:12: 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~ 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c: In function 'attr_remove_path': 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~ 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:In file included from ../src/global.h:25, 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr: from fsstress.c:12: 2024-04-24T08:24:11.100 INFO:tasks.workunit.client.0.smithi146.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~ 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c: In function 'attr_set_path': 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~ 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr:In file included from ../src/global.h:25, 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr: from fsstress.c:12: 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-04-24T08:24:11.101 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~ 2024-04-24T08:24:11.112 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c: In function 'allocsp_f': 2024-04-24T08:24:11.112 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-04-24T08:24:11.112 INFO:tasks.workunit.client.0.smithi146.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-04-24T08:24:11.112 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~~~~~~~ 2024-04-24T08:24:11.112 INFO:tasks.workunit.client.0.smithi146.stderr: | XFS_IOC_RESVSP64 2024-04-24T08:24:11.112 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-04-24T08:24:11.131 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c: In function 'freesp_f': 2024-04-24T08:24:11.132 INFO:tasks.workunit.client.0.smithi146.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-04-24T08:24:11.132 INFO:tasks.workunit.client.0.smithi146.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-04-24T08:24:11.132 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~~~~~~ 2024-04-24T08:24:11.132 INFO:tasks.workunit.client.0.smithi146.stderr: | XFS_IOC_RESVSP64 2024-04-24T08:24:11.175 INFO:tasks.workunit.client.0.smithi146.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-04-24T08:24:11.175 INFO:tasks.workunit.client.0.smithi146.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-04-24T08:24:11.567 INFO:tasks.workunit.client.0.smithi146.stderr:doio.c: In function 'parse_cmdline': 2024-04-24T08:24:11.567 INFO:tasks.workunit.client.0.smithi146.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-04-24T08:24:11.567 INFO:tasks.workunit.client.0.smithi146.stderr: 3113 | fprintf(stderr, 2024-04-24T08:24:11.567 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~~~~~ 2024-04-24T08:24:11.568 INFO:tasks.workunit.client.0.smithi146.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-04-24T08:24:11.568 INFO:tasks.workunit.client.0.smithi146.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-24T08:24:11.568 INFO:tasks.workunit.client.0.smithi146.stderr: 3115 | Prog, TagName, tok); 2024-04-24T08:24:11.568 INFO:tasks.workunit.client.0.smithi146.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-04-24T08:24:11.596 INFO:tasks.workunit.client.0.smithi146.stderr:fsx.c: In function 'main': 2024-04-24T08:24:11.596 INFO:tasks.workunit.client.0.smithi146.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-04-24T08:24:11.596 INFO:tasks.workunit.client.0.smithi146.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-04-24T08:24:11.596 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-24T08:24:11.596 INFO:tasks.workunit.client.0.smithi146.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-04-24T08:24:11.596 INFO:tasks.workunit.client.0.smithi146.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-04-24T08:24:11.596 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-24T08:24:11.732 INFO:tasks.workunit.client.0.smithi146.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-04-24T08:24:11.732 INFO:tasks.workunit.client.0.smithi146.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-04-24T08:24:12.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:12 smithi146 ceph-mon[27805]: pgmap v93: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 316 KiB/s wr, 32 op/s 2024-04-24T08:24:12.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:12 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:24:12.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:12 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:24:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:12 smithi177 ceph-mon[33046]: pgmap v93: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 316 KiB/s wr, 32 op/s 2024-04-24T08:24:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:12 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:24:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:12 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:24:12.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:12 smithi153 ceph-mon[32746]: pgmap v93: 97 pgs: 97 active+clean; 67 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 316 KiB/s wr, 32 op/s 2024-04-24T08:24:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:12 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:24:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:12 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:24:13.750 INFO:tasks.workunit.client.0.smithi146.stdout: [DEP] 2024-04-24T08:24:13.750 INFO:tasks.workunit.client.0.smithi146.stdout: [DEP] 2024-04-24T08:24:13.751 INFO:tasks.workunit.client.0.smithi146.stdout: [DEP] 2024-04-24T08:24:14.266 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] dirstress 2024-04-24T08:24:14.271 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] fill 2024-04-24T08:24:14.272 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] fill2 2024-04-24T08:24:14.272 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] getpagesize 2024-04-24T08:24:14.578 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] holes 2024-04-24T08:24:14.581 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] lstat64 2024-04-24T08:24:14.689 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] nametest 2024-04-24T08:24:14.714 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] permname 2024-04-24T08:24:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:14 smithi146 ceph-mon[27805]: pgmap v94: 97 pgs: 97 active+clean; 67 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 46 op/s 2024-04-24T08:24:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:14 smithi177 ceph-mon[33046]: pgmap v94: 97 pgs: 97 active+clean; 67 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 46 op/s 2024-04-24T08:24:14.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:14 smithi153 ceph-mon[32746]: pgmap v94: 97 pgs: 97 active+clean; 67 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 46 op/s 2024-04-24T08:24:14.991 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] randholes 2024-04-24T08:24:14.991 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] runas 2024-04-24T08:24:15.130 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] truncfile 2024-04-24T08:24:15.136 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] usemem 2024-04-24T08:24:15.306 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] mmapcat 2024-04-24T08:24:15.407 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] append_reader 2024-04-24T08:24:15.418 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] append_writer 2024-04-24T08:24:15.456 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] dirperf 2024-04-24T08:24:15.587 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] metaperf 2024-04-24T08:24:15.690 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] devzero 2024-04-24T08:24:15.698 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] feature 2024-04-24T08:24:15.860 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] alloc 2024-04-24T08:24:16.000 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] fault 2024-04-24T08:24:16.039 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] fstest 2024-04-24T08:24:16.039 INFO:tasks.workunit.client.0.smithi146.stdout: [CC] t_access_root 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr:alloc.c: In function 'main': 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: 154 | bozo! 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~ 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: 154 | bozo! 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: | ^ 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: | ; 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: 184 | rflag = 1; 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~ 2024-04-24T08:24:16.109 INFO:tasks.workunit.client.0.smithi146.stderr: | nflag 2024-04-24T08:24:16.110 INFO:tasks.workunit.client.0.smithi146.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-04-24T08:24:16.110 INFO:tasks.workunit.client.0.smithi146.stderr: 320 | bozo! 2024-04-24T08:24:16.110 INFO:tasks.workunit.client.0.smithi146.stderr: | ^ 2024-04-24T08:24:16.110 INFO:tasks.workunit.client.0.smithi146.stderr: | ; 2024-04-24T08:24:16.110 INFO:tasks.workunit.client.0.smithi146.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-04-24T08:24:16.110 INFO:tasks.workunit.client.0.smithi146.stderr: 128 | struct flock64 f; 2024-04-24T08:24:16.110 INFO:tasks.workunit.client.0.smithi146.stderr: | ^ 2024-04-24T08:24:16.116 INFO:tasks.workunit.client.0.smithi146.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-04-24T08:24:16.116 INFO:tasks.workunit.client.0.smithi146.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr:fstest.c: In function 'run_child': 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~ 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~ 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-24T08:24:16.295 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~ 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~ 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-04-24T08:24:16.296 INFO:tasks.workunit.client.0.smithi146.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-24T08:24:16.402 INFO:tasks.workunit.client.0.smithi146.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-04-24T08:24:16.402 INFO:tasks.workunit.client.0.smithi146.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-04-24T08:24:16.403 INFO:tasks.workunit.client.0.smithi146.stderr:make: *** [Makefile:51: default] Error 2 2024-04-24T08:24:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-24T08:24:16.405 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-04-24T08:24:16.405 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-24T08:24:16.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:16 smithi146 ceph-mon[27805]: pgmap v95: 97 pgs: 97 active+clean; 67 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 219 KiB/s wr, 22 op/s 2024-04-24T08:24:16.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:16 smithi177 ceph-mon[33046]: pgmap v95: 97 pgs: 97 active+clean; 67 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 219 KiB/s wr, 22 op/s 2024-04-24T08:24:16.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:16 smithi153 ceph-mon[32746]: pgmap v95: 97 pgs: 97 active+clean; 67 MiB data, 655 MiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 219 KiB/s wr, 22 op/s 2024-04-24T08:24:17.473 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi146 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-04-24T08:24:17.728 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1b14c3b5a7cc45bbb1d079bdc51b16bf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi146 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-04-24T08:24:17.731 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-24T08:24:17.742 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-24T08:24:17.742 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:24:17.742 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:24:17.742 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-24T08:24:17.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-24T08:24:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:18 smithi177 ceph-mon[33046]: pgmap v96: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 861 KiB/s wr, 98 op/s 2024-04-24T08:24:18.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:18 smithi153 ceph-mon[32746]: pgmap v96: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 861 KiB/s wr, 98 op/s 2024-04-24T08:24:18.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:18 smithi146 ceph-mon[27805]: pgmap v96: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 861 KiB/s wr, 98 op/s 2024-04-24T08:24:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:20 smithi177 ceph-mon[33046]: pgmap v97: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 691 KiB/s wr, 90 op/s 2024-04-24T08:24:20.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:20 smithi153 ceph-mon[32746]: pgmap v97: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 691 KiB/s wr, 90 op/s 2024-04-24T08:24:20.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:20 smithi146 ceph-mon[27805]: pgmap v97: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 691 KiB/s wr, 90 op/s 2024-04-24T08:24:22.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:22 smithi177 ceph-mon[33046]: pgmap v98: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 691 KiB/s wr, 90 op/s 2024-04-24T08:24:22.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:22 smithi153 ceph-mon[32746]: pgmap v98: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 691 KiB/s wr, 90 op/s 2024-04-24T08:24:22.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:22 smithi146 ceph-mon[27805]: pgmap v98: 97 pgs: 97 active+clean; 73 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 691 KiB/s wr, 90 op/s 2024-04-24T08:24:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:23 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:24:23.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:23 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:24:23.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:23 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:24:24.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:24 smithi177 ceph-mon[33046]: pgmap v99: 97 pgs: 97 active+clean; 73 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 700 KiB/s wr, 95 op/s 2024-04-24T08:24:24.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:24 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:24:24.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:24 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:24:24.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:24 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:24:24.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:24 smithi153 ceph-mon[32746]: pgmap v99: 97 pgs: 97 active+clean; 73 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 700 KiB/s wr, 95 op/s 2024-04-24T08:24:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:24 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:24:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:24 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:24:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:24 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:24:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:24 smithi146 ceph-mon[27805]: pgmap v99: 97 pgs: 97 active+clean; 73 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 700 KiB/s wr, 95 op/s 2024-04-24T08:24:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:24 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:24:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:24 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:24:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:24 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:24:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:26 smithi177 ceph-mon[33046]: pgmap v100: 97 pgs: 97 active+clean; 73 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 649 KiB/s wr, 81 op/s 2024-04-24T08:24:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:26 smithi153 ceph-mon[32746]: pgmap v100: 97 pgs: 97 active+clean; 73 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 649 KiB/s wr, 81 op/s 2024-04-24T08:24:26.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:26 smithi146 ceph-mon[27805]: pgmap v100: 97 pgs: 97 active+clean; 73 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 649 KiB/s wr, 81 op/s 2024-04-24T08:24:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:28 smithi177 ceph-mon[33046]: pgmap v101: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 654 KiB/s wr, 84 op/s 2024-04-24T08:24:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:28 smithi153 ceph-mon[32746]: pgmap v101: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 654 KiB/s wr, 84 op/s 2024-04-24T08:24:28.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:28 smithi146 ceph-mon[27805]: pgmap v101: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 654 KiB/s wr, 84 op/s 2024-04-24T08:24:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:30 smithi177 ceph-mon[33046]: pgmap v102: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:24:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:30 smithi153 ceph-mon[32746]: pgmap v102: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:24:30.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:30 smithi146 ceph-mon[27805]: pgmap v102: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:24:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:32 smithi177 ceph-mon[33046]: pgmap v103: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:24:32.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:32 smithi153 ceph-mon[32746]: pgmap v103: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:24:32.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:32 smithi146 ceph-mon[27805]: pgmap v103: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:24:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:34 smithi177 ceph-mon[33046]: pgmap v104: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-24T08:24:34.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:34 smithi153 ceph-mon[32746]: pgmap v104: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-24T08:24:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:34 smithi146 ceph-mon[27805]: pgmap v104: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-24T08:24:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:36 smithi177 ceph-mon[33046]: pgmap v105: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:24:36.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:24:36.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:24:36.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:36 smithi153 ceph-mon[32746]: pgmap v105: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:24:36.977 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:36 smithi146 ceph-mon[27805]: pgmap v105: 97 pgs: 97 active+clean; 73 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:24:37.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 -- 172.21.15.146:0/1057299266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f221410e660 msgr2=0x7f221410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 --2- 172.21.15.146:0/1057299266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f221410e660 0x7f221410ea40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2204009cd0 tx=0x7f220402a8f0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 -- 172.21.15.146:0/1057299266 shutdown_connections 2024-04-24T08:24:37.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 --2- 172.21.15.146:0/1057299266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2214108fd0 0x7f2214115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 --2- 172.21.15.146:0/1057299266 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2214108610 0x7f2214108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 --2- 172.21.15.146:0/1057299266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f221410e660 0x7f221410ea40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 -- 172.21.15.146:0/1057299266 >> 172.21.15.146:0/1057299266 conn(0x7f2214076540 msgr2=0x7f2214078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 -- 172.21.15.146:0/1057299266 shutdown_connections 2024-04-24T08:24:37.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.113+0000 7f221bc00640 1 -- 172.21.15.146:0/1057299266 wait complete. 2024-04-24T08:24:37.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 Processor -- start 2024-04-24T08:24:37.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 -- start start 2024-04-24T08:24:37.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2214108610 0x7f221419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2214108fd0 0x7f221419f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221410e660 0x7f22141a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2214118a30 con 0x7f2214108610 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22141188b0 con 0x7f221410e660 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.114+0000 7f221bc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2214118bb0 con 0x7f2214108fd0 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2214108610 0x7f221419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2214108610 0x7f221419eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54582/0 (socket says 172.21.15.146:54582) 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 -- 172.21.15.146:0/4217717219 learned_addr learned my addr 172.21.15.146:0/4217717219 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f220be00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2214108fd0 0x7f221419f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219c00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221410e660 0x7f22141a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 -- 172.21.15.146:0/4217717219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2214108fd0 msgr2=0x7f221419f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2214108fd0 0x7f221419f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 -- 172.21.15.146:0/4217717219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221410e660 msgr2=0x7f22141a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221410e660 0x7f22141a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 -- 172.21.15.146:0/4217717219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22141a3ea0 con 0x7f2214108610 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f220be00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2214108fd0 0x7f221419f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219c00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221410e660 0x7f22141a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.115+0000 7f2219200640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2214108610 0x7f221419eeb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2204000f80 tx=0x7f2204003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.116+0000 7f2212000640 1 -- 172.21.15.146:0/4217717219 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204030e40 con 0x7f2214108610 2024-04-24T08:24:37.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.116+0000 7f2212000640 1 -- 172.21.15.146:0/4217717219 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2204040070 con 0x7f2214108610 2024-04-24T08:24:37.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.116+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22141a4130 con 0x7f2214108610 2024-04-24T08:24:37.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.116+0000 7f2212000640 1 -- 172.21.15.146:0/4217717219 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204031b10 con 0x7f2214108610 2024-04-24T08:24:37.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.116+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22141aba20 con 0x7f2214108610 2024-04-24T08:24:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.119+0000 7f221ea28740 1 Processor -- start 2024-04-24T08:24:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.119+0000 7f2212000640 1 -- 172.21.15.146:0/4217717219 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f2204031cb0 con 0x7f2214108610 2024-04-24T08:24:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.119+0000 7f221ea28740 1 -- start start 2024-04-24T08:24:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f221ea28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c57657ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f221ea28740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576585310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f221ea28740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576587c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f221ea28740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c5764e01f0 con 0x55c576585850 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f221ea28740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c576486350 con 0x55c57657b180 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f221ea28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c5764dc1b0 con 0x55c57657a860 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c57657ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c57657ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59534/0 (socket says 172.21.15.146:59534) 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219c00640 1 -- 172.21.15.146:0/1061486022 learned_addr learned my addr 172.21.15.146:0/1061486022 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f220be00640 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576587c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219200640 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576585310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219c00640 1 -- 172.21.15.146:0/1061486022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 msgr2=0x55c576585310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219c00640 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576585310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219c00640 1 -- 172.21.15.146:0/1061486022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 msgr2=0x55c576587c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.120+0000 7f2219c00640 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576587c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f2219c00640 1 -- 172.21.15.146:0/1061486022 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5764daf50 con 0x55c57657a860 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f2219200640 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576585310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:37.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f2219c00640 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c57657ac40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f220c011540 tx=0x7f220c011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e094642cda3ef03 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f220a000640 1 -- 172.21.15.146:0/1061486022 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c00f5c0 con 0x55c57657a860 2024-04-24T08:24:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f220a000640 1 -- 172.21.15.146:0/1061486022 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f220c00fc40 con 0x55c57657a860 2024-04-24T08:24:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f220a000640 1 -- 172.21.15.146:0/1061486022 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c019cc0 con 0x55c57657a860 2024-04-24T08:24:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f2212000640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f21f80a3da0 0x7f21f80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f2212000640 1 -- 172.21.15.146:0/4217717219 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f22040f1720 con 0x7f2214108610 2024-04-24T08:24:37.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.121+0000 7f2219200640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f21f80a3da0 0x7f21f80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 -- 172.21.15.146:0/1061486022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 msgr2=0x55c57657ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c57657ac40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f220c011540 tx=0x7f220c011a00 comp rx=0 tx=0).stop 2024-04-24T08:24:37.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 -- 172.21.15.146:0/1061486022 shutdown_connections 2024-04-24T08:24:37.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576587c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576585310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 --2- 172.21.15.146:0/1061486022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c57657ac40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 -- 172.21.15.146:0/1061486022 >> 172.21.15.146:0/1061486022 conn(0x55c576576b80 msgr2=0x55c576576f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 -- 172.21.15.146:0/1061486022 shutdown_connections 2024-04-24T08:24:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f221ea28740 1 -- 172.21.15.146:0/1061486022 wait complete. 2024-04-24T08:24:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.122+0000 7f2219200640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f21f80a3da0 0x7f21f80a6260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55c57657be20 tx=0x7f220402a8f0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 -- start start 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 Processor -- start 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576582800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576582d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576583280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c576588640 con 0x55c576585850 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c576499690 con 0x55c57657b180 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c5765887c0 con 0x55c57657a860 2024-04-24T08:24:37.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f221ea28740 10 client.0 _ll_register_callbacks cb 0x55c5764d5650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.123+0000 7f2219c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576582800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f220be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576582d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576582800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59536/0 (socket says 172.21.15.146:59536) 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 -- 172.21.15.146:0/2464502675 learned_addr learned my addr 172.21.15.146:0/2464502675 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576583280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 -- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 msgr2=0x55c576582d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576582d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 -- 172.21.15.146:0/2464502675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 msgr2=0x55c576583280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576583280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 -- 172.21.15.146:0/2464502675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c576498920 con 0x55c57657a860 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f220be00640 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576582d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2219c00640 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576582800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f220c019760 tx=0x7f220c019880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6decb3f5e684a01c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.124+0000 7f2202a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.125+0000 7f2202a00640 1 -- 172.21.15.146:0/2464502675 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c004280 con 0x55c57657a860 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.125+0000 7f2202a00640 1 -- 172.21.15.146:0/2464502675 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f220c016070 con 0x55c57657a860 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.125+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f220c00fc40 con 0x55c57657a860 2024-04-24T08:24:37.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.125+0000 7f2202a00640 1 -- 172.21.15.146:0/2464502675 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f220c034050 con 0x55c57657a860 2024-04-24T08:24:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.125+0000 7f221ea28740 10 client.26005 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.125+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c5762bc7d0 con 0x55c57657a860 2024-04-24T08:24:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.125+0000 7f2202a00640 1 -- 172.21.15.146:0/2464502675 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f220c00fc40 con 0x55c57657a860 2024-04-24T08:24:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f2202a00640 1 -- 172.21.15.146:0/2464502675 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f220c0046c0 con 0x55c57657a860 2024-04-24T08:24:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f221ea28740 10 client.26005 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f221ea28740 10 client.26005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:37.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f221ea28740 20 client.26005 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f221ea28740 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c576595b90 0x55c576597f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f221ea28740 4 client.26005 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5764edc10 con 0x55c576595b90 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.126+0000 7f220be00640 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c576595b90 0x55c576597f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.127+0000 7f220be00640 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c576595b90 0x55c576597f80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a28b4962c2d351e4 server_cookie=f8f1dbd7396ca449 in_seq=0 out_seq=0 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.127+0000 7f2202a00640 10 client.26005 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f2202a00640 1 -- 172.21.15.146:0/2464502675 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5764edc10 con 0x55c576595b90 2024-04-24T08:24:37.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f2202a00640 10 client.26005 handle_command_reply: tid=0 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 client.26005 shutdown 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 msgr2=0x55c576582800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576582800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f220c019760 tx=0x7f220c019880 comp rx=0 tx=0).stop 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 shutdown_connections 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c576595b90 0x55c576597f80 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c576583280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576582d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 --2- 172.21.15.146:0/2464502675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576582800 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 >> 172.21.15.146:0/2464502675 conn(0x55c576576b80 msgr2=0x55c57657c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f2202a00640 0 client.26005 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:37.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 shutdown_connections 2024-04-24T08:24:37.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.128+0000 7f221ea28740 1 -- 172.21.15.146:0/2464502675 wait complete. 2024-04-24T08:24:37.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.129+0000 7f221ea28740 20 client.26005 trim_cache size 0 max 16384 2024-04-24T08:24:37.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 Processor -- start 2024-04-24T08:24:37.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 -- start start 2024-04-24T08:24:37.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576591500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576581960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c5765830a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b6af0 con 0x55c576585850 2024-04-24T08:24:37.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b43c0 con 0x55c57657b180 2024-04-24T08:24:37.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f221ea28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b2210 con 0x55c57657a860 2024-04-24T08:24:37.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f220be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576581960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.148+0000 7f220be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576581960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39590/0 (socket says 172.21.15.146:39590) 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f220be00640 1 -- 172.21.15.146:0/2658247693 learned_addr learned my addr 172.21.15.146:0/2658247693 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f2219c00640 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c5765830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f2219c00640 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576591500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f220be00640 1 -- 172.21.15.146:0/2658247693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 msgr2=0x55c576591500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f220be00640 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576591500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f220be00640 1 -- 172.21.15.146:0/2658247693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 msgr2=0x55c5765830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f220be00640 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c5765830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f220be00640 1 -- 172.21.15.146:0/2658247693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5761f23f0 con 0x55c57657b180 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f220be00640 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576581960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55c576577d10 tx=0x7f21fc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31d05d6506066497 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f2209600640 1 -- 172.21.15.146:0/2658247693 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc00fc80 con 0x55c57657b180 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f2209600640 1 -- 172.21.15.146:0/2658247693 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21fc00fe20 con 0x55c57657b180 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.149+0000 7f2209600640 1 -- 172.21.15.146:0/2658247693 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc0047a0 con 0x55c57657b180 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 -- 172.21.15.146:0/2658247693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 msgr2=0x55c576581960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576581960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55c576577d10 tx=0x7f21fc005f90 comp rx=0 tx=0).stop 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 -- 172.21.15.146:0/2658247693 shutdown_connections 2024-04-24T08:24:37.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c576585850 0x55c5765830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c57657b180 0x55c576581960 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 --2- 172.21.15.146:0/2658247693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c576591500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 -- 172.21.15.146:0/2658247693 >> 172.21.15.146:0/2658247693 conn(0x55c576576b80 msgr2=0x55c576575390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 -- 172.21.15.146:0/2658247693 shutdown_connections 2024-04-24T08:24:37.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.150+0000 7f221ea28740 1 -- 172.21.15.146:0/2658247693 wait complete. 2024-04-24T08:24:37.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 -- start start 2024-04-24T08:24:37.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 Processor -- start 2024-04-24T08:24:37.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c5765889d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c57657b180 0x55c576588f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c576585850 0x55c576589450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b2880 con 0x55c57657b180 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b3cf0 con 0x55c576585850 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22141b2f50 con 0x55c57657a860 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.151+0000 7f221ea28740 10 client.0 _ll_register_callbacks cb 0x7f21fc009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c5765889d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c5765889d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59560/0 (socket says 172.21.15.146:59560) 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 -- 172.21.15.146:0/1925095631 learned_addr learned my addr 172.21.15.146:0/1925095631 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f220be00640 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c57657b180 0x55c576588f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c576585850 0x55c576589450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 -- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c576585850 msgr2=0x55c576589450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c576585850 0x55c576589450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 -- 172.21.15.146:0/1925095631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c57657b180 msgr2=0x55c576588f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c57657b180 0x55c576588f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 -- 172.21.15.146:0/1925095631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5761b20f0 con 0x55c57657a860 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2219c00640 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c5765889d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f220c004ac0 tx=0x7f220c0197e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67b83b0a4178e186 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2202a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.152+0000 7f2202a00640 1 -- 172.21.15.146:0/1925095631 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c005090 con 0x55c57657a860 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f2202a00640 1 -- 172.21.15.146:0/1925095631 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f220c005230 con 0x55c57657a860 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21fc00fe20 con 0x55c57657a860 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f2202a00640 1 -- 172.21.15.146:0/1925095631 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f220c00f5c0 con 0x55c57657a860 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f221ea28740 10 client.26011 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c5764a5810 con 0x55c57657a860 2024-04-24T08:24:37.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f2202a00640 1 -- 172.21.15.146:0/1925095631 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21fc00fe20 con 0x55c57657a860 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f2202a00640 1 -- 172.21.15.146:0/1925095631 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f220c004730 con 0x55c57657a860 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f221ea28740 10 client.26011 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f221ea28740 10 client.26011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.153+0000 7f221ea28740 20 client.26011 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.154+0000 7f221ea28740 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c5765fb7e0 0x55c5765fdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.154+0000 7f221ea28740 4 client.26011 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.154+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c5764e7a80 con 0x55c5765fb7e0 2024-04-24T08:24:37.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.154+0000 7f220be00640 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c5765fb7e0 0x55c5765fdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f220be00640 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c5765fb7e0 0x55c5765fdc00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d9e1303b36e5441 server_cookie=8bd1792b1b906264 in_seq=0 out_seq=0 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f2202a00640 10 client.26011 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f2202a00640 1 -- 172.21.15.146:0/1925095631 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c5764e7a80 con 0x55c5765fb7e0 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f2202a00640 10 client.26011 handle_command_reply: tid=0 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 client.26011 shutdown 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 msgr2=0x55c5765889d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c5765889d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f220c004ac0 tx=0x7f220c0197e0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 shutdown_connections 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c5765fb7e0 0x55c5765fdc00 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c576585850 0x55c576589450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c57657b180 0x55c576588f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f2202a00640 0 client.26011 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 --2- 172.21.15.146:0/1925095631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c57657a860 0x55c5765889d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 >> 172.21.15.146:0/1925095631 conn(0x55c576576b80 msgr2=0x55c5765830e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 shutdown_connections 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 1 -- 172.21.15.146:0/1925095631 wait complete. 2024-04-24T08:24:37.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.155+0000 7f221ea28740 20 client.26011 trim_cache size 0 max 16384 2024-04-24T08:24:37.165 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:24:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:24:37.165 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:24:37.165 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:24:37.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f21f80a3da0 msgr2=0x7f21f80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f21f80a3da0 0x7f21f80a6260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55c57657be20 tx=0x7f220402a8f0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2214108610 msgr2=0x7f221419eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2214108610 0x7f221419eeb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2204000f80 tx=0x7f2204003ea0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 shutdown_connections 2024-04-24T08:24:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f21f80a3da0 0x7f21f80a6260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2214108fd0 0x7f221419f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f221410e660 0x7f22141a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 --2- 172.21.15.146:0/4217717219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2214108610 0x7f221419eeb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 >> 172.21.15.146:0/4217717219 conn(0x7f2214076540 msgr2=0x7f2214077c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 shutdown_connections 2024-04-24T08:24:37.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.156+0000 7f221bc00640 1 -- 172.21.15.146:0/4217717219 wait complete. 2024-04-24T08:24:37.175 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:24:37.176 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:24:37.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 -- 172.21.15.146:0/1039462824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4076e50 msgr2=0x7f92b4112760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 --2- 172.21.15.146:0/1039462824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4076e50 0x7f92b4112760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f929c009890 tx=0x7f929c02a370 comp rx=0 tx=0).stop 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 -- 172.21.15.146:0/1039462824 shutdown_connections 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 --2- 172.21.15.146:0/1039462824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4076e50 0x7f92b4112760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 --2- 172.21.15.146:0/1039462824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4112ca0 0x7f92b4115090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 --2- 172.21.15.146:0/1039462824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b4076530 0x7f92b4076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 -- 172.21.15.146:0/1039462824 >> 172.21.15.146:0/1039462824 conn(0x7f92b40ffa90 msgr2=0x7f92b4101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 -- 172.21.15.146:0/1039462824 shutdown_connections 2024-04-24T08:24:37.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.361+0000 7f92b9e00640 1 -- 172.21.15.146:0/1039462824 wait complete. 2024-04-24T08:24:37.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 Processor -- start 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 -- start start 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b4076530 0x7f92b419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4076e50 0x7f92b419f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4112ca0 0x7f92b41a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f92b4118800 con 0x7f92b4076530 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f92b4118680 con 0x7f92b4076e50 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b9e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92b4118980 con 0x7f92b4112ca0 2024-04-24T08:24:37.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.363+0000 7f92b3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4076e50 0x7f92b419f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4076e50 0x7f92b419f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39610/0 (socket says 172.21.15.146:39610) 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 -- 172.21.15.146:0/3134906527 learned_addr learned my addr 172.21.15.146:0/3134906527 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b9400640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4112ca0 0x7f92b41a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b8a00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b4076530 0x7f92b419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 -- 172.21.15.146:0/3134906527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4112ca0 msgr2=0x7f92b41a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4112ca0 0x7f92b41a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 -- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b4076530 msgr2=0x7f92b419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b4076530 0x7f92b419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 -- 172.21.15.146:0/3134906527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b41a3d30 con 0x7f92b4076e50 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b9400640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4112ca0 0x7f92b41a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b8a00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b4076530 0x7f92b419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b3e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4076e50 0x7f92b419f280 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f929c030510 tx=0x7f929c033ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.364+0000 7f92b1600640 1 -- 172.21.15.146:0/3134906527 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c037040 con 0x7f92b4076e50 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.365+0000 7f92b1600640 1 -- 172.21.15.146:0/3134906527 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f929c040070 con 0x7f92b4076e50 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.365+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b41a3fc0 con 0x7f92b4076e50 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.365+0000 7f92b1600640 1 -- 172.21.15.146:0/3134906527 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c003a10 con 0x7f92b4076e50 2024-04-24T08:24:37.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.365+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b41ab8b0 con 0x7f92b4076e50 2024-04-24T08:24:37.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.366+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b4069190 con 0x7f92b4076e50 2024-04-24T08:24:37.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.368+0000 7f92b1600640 1 -- 172.21.15.146:0/3134906527 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f929c004170 con 0x7f92b4076e50 2024-04-24T08:24:37.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.369+0000 7f92b1600640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f92980a3da0 0x7f92980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.370+0000 7f92b1600640 1 -- 172.21.15.146:0/3134906527 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f929c0f1230 con 0x7f92b4076e50 2024-04-24T08:24:37.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.370+0000 7f92b8a00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f92980a3da0 0x7f92980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.370+0000 7f92b8a00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f92980a3da0 0x7f92980a6260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f92a4005fd0 tx=0x7f92a4009260 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.372+0000 7f92b1600640 1 -- 172.21.15.146:0/3134906527 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f929c043360 con 0x7f92b4076e50 2024-04-24T08:24:37.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.618+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92b41a0090 con 0x7f92b4076e50 2024-04-24T08:24:37.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.620+0000 7f92b1600640 1 -- 172.21.15.146:0/3134906527 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11892 (secure 0 0 0) 0x7f92b41a0090 con 0x7f92b4076e50 2024-04-24T08:24:37.622 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:24:37.623 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:23:07.211437+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24568,24569,24553]},"id":1}]} 2024-04-24T08:24:37.623 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 25 2024-04-24T08:24:37.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f92980a3da0 msgr2=0x7f92980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f92980a3da0 0x7f92980a6260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f92a4005fd0 tx=0x7f92a4009260 comp rx=0 tx=0).stop 2024-04-24T08:24:37.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4076e50 msgr2=0x7f92b419f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4076e50 0x7f92b419f280 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f929c030510 tx=0x7f929c033ae0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 shutdown_connections 2024-04-24T08:24:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f92980a3da0 0x7f92980a6260 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92b4112ca0 0x7f92b41a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92b4076e50 0x7f92b419f280 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 --2- 172.21.15.146:0/3134906527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92b4076530 0x7f92b419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 >> 172.21.15.146:0/3134906527 conn(0x7f92b40ffa90 msgr2=0x7f92b41143c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.623+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 shutdown_connections 2024-04-24T08:24:37.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.624+0000 7f92b9e00640 1 -- 172.21.15.146:0/3134906527 wait complete. 2024-04-24T08:24:37.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:24:37.856 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 -- 172.21.15.146:0/3857655329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f039410e660 msgr2=0x7f039410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 --2- 172.21.15.146:0/3857655329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f039410e660 0x7f039410ea40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0388009930 tx=0x7f038802a420 comp rx=0 tx=0).stop 2024-04-24T08:24:37.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 -- 172.21.15.146:0/3857655329 shutdown_connections 2024-04-24T08:24:37.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 --2- 172.21.15.146:0/3857655329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0394108fd0 0x7f0394115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 --2- 172.21.15.146:0/3857655329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108610 0x7f0394108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.857 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 --2- 172.21.15.146:0/3857655329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f039410e660 0x7f039410ea40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 -- 172.21.15.146:0/3857655329 >> 172.21.15.146:0/3857655329 conn(0x7f0394076540 msgr2=0x7f0394078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 -- 172.21.15.146:0/3857655329 shutdown_connections 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.851+0000 7f039bc00640 1 -- 172.21.15.146:0/3857655329 wait complete. 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.852+0000 7f039bc00640 1 Processor -- start 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.852+0000 7f039bc00640 1 -- start start 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f039bc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0394108610 0x7f039419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f039bc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108fd0 0x7f039419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f039bc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f039410e660 0x7f03941a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f039bc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03941189f0 con 0x7f039410e660 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f039bc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0394118870 con 0x7f0394108fd0 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f039bc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0394118b70 con 0x7f0394108610 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f0393e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108fd0 0x7f039419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f0393e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108fd0 0x7f039419f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39634/0 (socket says 172.21.15.146:39634) 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f0399200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0394108610 0x7f039419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f0393e00640 1 -- 172.21.15.146:0/1592550970 learned_addr learned my addr 172.21.15.146:0/1592550970 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.853+0000 7f0399c00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f039410e660 0x7f03941a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0393e00640 1 -- 172.21.15.146:0/1592550970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0394108610 msgr2=0x7f039419eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0393e00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0394108610 0x7f039419eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0393e00640 1 -- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f039410e660 msgr2=0x7f03941a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0393e00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f039410e660 0x7f03941a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0393e00640 1 -- 172.21.15.146:0/1592550970 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03941a3ec0 con 0x7f0394108fd0 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0393e00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108fd0 0x7f039419f410 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f038000d950 tx=0x7f038000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0391600640 1 -- 172.21.15.146:0/1592550970 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0380017070 con 0x7f0394108fd0 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03941a41b0 con 0x7f0394108fd0 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0391600640 1 -- 172.21.15.146:0/1592550970 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0380009c30 con 0x7f0394108fd0 2024-04-24T08:24:37.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f0391600640 1 -- 172.21.15.146:0/1592550970 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038000f6b0 con 0x7f0394108fd0 2024-04-24T08:24:37.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.854+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03941aba40 con 0x7f0394108fd0 2024-04-24T08:24:37.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.857+0000 7f039e9e1740 1 Processor -- start 2024-04-24T08:24:37.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f0391600640 1 -- 172.21.15.146:0/1592550970 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f03800109d0 con 0x7f0394108fd0 2024-04-24T08:24:37.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f039e9e1740 1 -- start start 2024-04-24T08:24:37.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc8747d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc8752400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc8754d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f039e9e1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555fc86ad1d0 con 0x555fc8752940 2024-04-24T08:24:37.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f039e9e1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555fc8653450 con 0x555fc8748270 2024-04-24T08:24:37.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f039e9e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fc86a9190 con 0x555fc8747950 2024-04-24T08:24:37.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f0399c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc8747d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f0399c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc8747d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59602/0 (socket says 172.21.15.146:59602) 2024-04-24T08:24:37.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.858+0000 7f0399c00640 1 -- 172.21.15.146:0/1545464504 learned_addr learned my addr 172.21.15.146:0/1545464504 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0399200640 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc8752400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0393e00640 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc8754d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0399c00640 1 -- 172.21.15.146:0/1545464504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 msgr2=0x555fc8752400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0399c00640 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc8752400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0399c00640 1 -- 172.21.15.146:0/1545464504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 msgr2=0x555fc8754d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0399c00640 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc8754d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0399c00640 1 -- 172.21.15.146:0/1545464504 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc8710310 con 0x555fc8747950 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0399c00640 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc8747d30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f038c009fd0 tx=0x7f038c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c79745959a58faef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0391600640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f037c0a3da0 0x7f037c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.859+0000 7f0386000640 1 -- 172.21.15.146:0/1545464504 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c004d20 con 0x555fc8747950 2024-04-24T08:24:37.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.860+0000 7f0386000640 1 -- 172.21.15.146:0/1545464504 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f038c004050 con 0x555fc8747950 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.860+0000 7f0391600640 1 -- 172.21.15.146:0/1592550970 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f03800d06c0 con 0x7f0394108fd0 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.860+0000 7f0386000640 1 -- 172.21.15.146:0/1545464504 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c005260 con 0x555fc8747950 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.860+0000 7f0399200640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f037c0a3da0 0x7f037c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.860+0000 7f039e9e1740 1 -- 172.21.15.146:0/1545464504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 msgr2=0x555fc8747d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.860+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc8747d30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f038c009fd0 tx=0x7f038c0118d0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.860+0000 7f0399200640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f037c0a3da0 0x7f037c0a6260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0388009900 tx=0x7f0388000bc0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 -- 172.21.15.146:0/1545464504 shutdown_connections 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc8754d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc8752400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1545464504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc8747d30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 -- 172.21.15.146:0/1545464504 >> 172.21.15.146:0/1545464504 conn(0x555fc8743c70 msgr2=0x555fc8744050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 -- 172.21.15.146:0/1545464504 shutdown_connections 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 -- 172.21.15.146:0/1545464504 wait complete. 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 -- start start 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.861+0000 7f039e9e1740 1 Processor -- start 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc87523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc874f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc874fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f039e9e1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555fc87557d0 con 0x555fc8752940 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f039e9e1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555fc8666810 con 0x555fc8748270 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f039e9e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fc8755950 con 0x555fc8747950 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f039e9e1740 10 client.0 _ll_register_callbacks cb 0x555fc8634140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc874f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc874f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39656/0 (socket says 172.21.15.146:39656) 2024-04-24T08:24:37.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 -- 172.21.15.146:0/1472012681 learned_addr learned my addr 172.21.15.146:0/1472012681 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0399c00640 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc874fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0399200640 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc87523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 -- 172.21.15.146:0/1472012681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 msgr2=0x555fc87523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc87523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 -- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 msgr2=0x555fc874fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc874fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.862+0000 7f0393e00640 1 -- 172.21.15.146:0/1472012681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc86773f0 con 0x555fc8748270 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f0393e00640 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc874f6c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f038009c3c0 tx=0x7f038009c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16766e6b98bd4644 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f037aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f037aa00640 1 -- 172.21.15.146:0/1472012681 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03800ce240 con 0x555fc8748270 2024-04-24T08:24:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f037aa00640 1 -- 172.21.15.146:0/1472012681 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f03800a4df0 con 0x555fc8748270 2024-04-24T08:24:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f038c004050 con 0x555fc8748270 2024-04-24T08:24:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f037aa00640 1 -- 172.21.15.146:0/1472012681 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0380009de0 con 0x555fc8748270 2024-04-24T08:24:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f039e9e1740 10 client.26045 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555fc83bf3f0 con 0x555fc8748270 2024-04-24T08:24:37.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.863+0000 7f037aa00640 1 -- 172.21.15.146:0/1472012681 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f038c004050 con 0x555fc8748270 2024-04-24T08:24:37.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f037aa00640 1 -- 172.21.15.146:0/1472012681 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f038009bb10 con 0x555fc8748270 2024-04-24T08:24:37.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f039e9e1740 10 client.26045 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:37.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f039e9e1740 10 client.26045 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:24:37.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f039e9e1740 20 client.26045 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:37.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc8762af0 0x555fc8764f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f039e9e1740 4 client.26045 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fc86babf0 con 0x555fc8762af0 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.864+0000 7f0399200640 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc8762af0 0x555fc8764f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.865+0000 7f0399200640 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc8762af0 0x555fc8764f30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a8289e7964e0252 server_cookie=6d94836fac31c560 in_seq=0 out_seq=0 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.865+0000 7f037aa00640 10 client.26045 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.865+0000 7f037aa00640 1 -- 172.21.15.146:0/1472012681 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fc86babf0 con 0x555fc8762af0 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.865+0000 7f037aa00640 10 client.26045 handle_command_reply: tid=0 2024-04-24T08:24:37.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 client.26045 shutdown 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 msgr2=0x555fc874f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc874f6c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f038009c3c0 tx=0x7f038009c200 comp rx=0 tx=0).stop 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 shutdown_connections 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc8762af0 0x555fc8764f30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc874fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8748270 0x555fc874f6c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1472012681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8747950 0x555fc87523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f037aa00640 0 client.26045 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:37.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 >> 172.21.15.146:0/1472012681 conn(0x555fc8743c70 msgr2=0x555fc8749a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 shutdown_connections 2024-04-24T08:24:37.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 1 -- 172.21.15.146:0/1472012681 wait complete. 2024-04-24T08:24:37.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.866+0000 7f039e9e1740 20 client.26045 trim_cache size 0 max 16384 2024-04-24T08:24:37.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 Processor -- start 2024-04-24T08:24:37.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 -- start start 2024-04-24T08:24:37.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc875e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8748270 0x555fc8751f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc876a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03941b6b70 con 0x555fc8752940 2024-04-24T08:24:37.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03941b4440 con 0x555fc8747950 2024-04-24T08:24:37.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f039e9e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03941b2290 con 0x555fc8748270 2024-04-24T08:24:37.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f0399c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc876a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f0399c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc876a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54706/0 (socket says 172.21.15.146:54706) 2024-04-24T08:24:37.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f0399c00640 1 -- 172.21.15.146:0/3926060813 learned_addr learned my addr 172.21.15.146:0/3926060813 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f0399200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc875e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.881+0000 7f0393e00640 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8748270 0x555fc8751f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0393e00640 1 -- 172.21.15.146:0/3926060813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 msgr2=0x555fc875e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0393e00640 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc875e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0393e00640 1 -- 172.21.15.146:0/3926060813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 msgr2=0x555fc876a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0393e00640 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc876a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0393e00640 1 -- 172.21.15.146:0/3926060813 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc8750120 con 0x555fc8748270 2024-04-24T08:24:37.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0399c00640 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc876a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0393e00640 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8748270 0x555fc8751f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x555fc8750c90 tx=0x7f038009c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2015872e4a9d225 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0385600640 1 -- 172.21.15.146:0/3926060813 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03800a09c0 con 0x555fc8748270 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0385600640 1 -- 172.21.15.146:0/3926060813 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03800a5280 con 0x555fc8748270 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.882+0000 7f0385600640 1 -- 172.21.15.146:0/3926060813 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03800ce040 con 0x555fc8748270 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 -- 172.21.15.146:0/3926060813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8748270 msgr2=0x555fc8751f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8748270 0x555fc8751f00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x555fc8750c90 tx=0x7f038009c330 comp rx=0 tx=0).stop 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 -- 172.21.15.146:0/3926060813 shutdown_connections 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8748270 0x555fc8751f00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8752940 0x555fc876a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 --2- 172.21.15.146:0/3926060813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc875e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 -- 172.21.15.146:0/3926060813 >> 172.21.15.146:0/3926060813 conn(0x555fc8743c70 msgr2=0x555fc8744050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 -- 172.21.15.146:0/3926060813 shutdown_connections 2024-04-24T08:24:37.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.883+0000 7f039e9e1740 1 -- 172.21.15.146:0/3926060813 wait complete. 2024-04-24T08:24:37.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 -- start start 2024-04-24T08:24:37.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 Processor -- start 2024-04-24T08:24:37.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc8754090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8748270 0x555fc87545d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8752940 0x555fc87591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f03941b2900 con 0x555fc8748270 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f03941b3d70 con 0x555fc8747950 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03941b2fd0 con 0x555fc8752940 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.884+0000 7f039e9e1740 10 client.0 _ll_register_callbacks cb 0x7f0380098960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0393e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8748270 0x555fc87545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc8754090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8752940 0x555fc87591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc8754090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39676/0 (socket says 172.21.15.146:39676) 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 -- 172.21.15.146:0/1115223742 learned_addr learned my addr 172.21.15.146:0/1115223742 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8752940 0x555fc87591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59626/0 (socket says 172.21.15.146:59626) 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 -- 172.21.15.146:0/1115223742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8752940 msgr2=0x555fc87591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8752940 0x555fc87591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 -- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8748270 msgr2=0x555fc87545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8748270 0x555fc87545d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 -- 172.21.15.146:0/1115223742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc87480a0 con 0x555fc8747950 2024-04-24T08:24:37.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0393e00640 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8748270 0x555fc87545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f0399200640 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc8754090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0388009930 tx=0x7f038802a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=770dbf8272fae596 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.885+0000 7f037aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f037aa00640 1 -- 172.21.15.146:0/1115223742 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0388035040 con 0x555fc8747950 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03800a5280 con 0x555fc8747950 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f037aa00640 1 -- 172.21.15.146:0/1115223742 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0388054710 con 0x555fc8747950 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f037aa00640 1 -- 172.21.15.146:0/1115223742 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0388005630 con 0x555fc8747950 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 10 client.26051 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:37.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555fc84897d0 con 0x555fc8747950 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f037aa00640 1 -- 172.21.15.146:0/1115223742 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03800a5280 con 0x555fc8747950 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f037aa00640 1 -- 172.21.15.146:0/1115223742 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f038805b070 con 0x555fc8747950 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 10 client.26051 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 10 client.26051 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 20 client.26051 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc87c9b70 0x555fc87cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 4 client.26051 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.886+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555fc86b4a60 con 0x555fc87c9b70 2024-04-24T08:24:37.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.887+0000 7f0393e00640 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc87c9b70 0x555fc87cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.887+0000 7f0393e00640 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc87c9b70 0x555fc87cbf90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=972ed8182c216734 server_cookie=b47ea47ffacbd355 in_seq=0 out_seq=0 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.887+0000 7f037aa00640 10 client.26051 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.887+0000 7f037aa00640 1 -- 172.21.15.146:0/1115223742 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555fc86b4a60 con 0x555fc87c9b70 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.887+0000 7f037aa00640 10 client.26051 handle_command_reply: tid=0 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.887+0000 7f039e9e1740 1 client.26051 shutdown 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 msgr2=0x555fc8754090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc8754090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0388009930 tx=0x7f038802a420 comp rx=0 tx=0).stop 2024-04-24T08:24:37.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 shutdown_connections 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555fc87c9b70 0x555fc87cbf90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc8752940 0x555fc87591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc8748270 0x555fc87545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 --2- 172.21.15.146:0/1115223742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc8747950 0x555fc8754090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f037aa00640 0 client.26051 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 >> 172.21.15.146:0/1115223742 conn(0x555fc8743c70 msgr2=0x555fc8742560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 shutdown_connections 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 1 -- 172.21.15.146:0/1115223742 wait complete. 2024-04-24T08:24:37.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.888+0000 7f039e9e1740 20 client.26051 trim_cache size 0 max 16384 2024-04-24T08:24:37.898 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f037c0a3da0 msgr2=0x7f037c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f037c0a3da0 0x7f037c0a6260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0388009900 tx=0x7f0388000bc0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108fd0 msgr2=0x7f039419f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108fd0 0x7f039419f410 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f038000d950 tx=0x7f038000de10 comp rx=0 tx=0).stop 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 shutdown_connections 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f037c0a3da0 0x7f037c0a6260 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f039410e660 0x7f03941a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0394108fd0 0x7f039419f410 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 --2- 172.21.15.146:0/1592550970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0394108610 0x7f039419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:37.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 >> 172.21.15.146:0/1592550970 conn(0x7f0394076540 msgr2=0x7f0394077d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:37.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 shutdown_connections 2024-04-24T08:24:37.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:37.889+0000 7f039bc00640 1 -- 172.21.15.146:0/1592550970 wait complete. 2024-04-24T08:24:37.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:24:38.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 -- 172.21.15.146:0/4079004678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 msgr2=0x7f2ad0104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 --2- 172.21.15.146:0/4079004678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 0x7f2ad0104890 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009860 tx=0x7f2ac402a340 comp rx=0 tx=0).stop 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 -- 172.21.15.146:0/4079004678 shutdown_connections 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 --2- 172.21.15.146:0/4079004678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ad0109580 0x7f2ad0111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 --2- 172.21.15.146:0/4079004678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0104e60 0x7f2ad0108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 --2- 172.21.15.146:0/4079004678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 0x7f2ad0104890 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 -- 172.21.15.146:0/4079004678 >> 172.21.15.146:0/4079004678 conn(0x7f2ad00ffd90 msgr2=0x7f2ad01021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 -- 172.21.15.146:0/4079004678 shutdown_connections 2024-04-24T08:24:38.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.110+0000 7f2ad8800640 1 -- 172.21.15.146:0/4079004678 wait complete. 2024-04-24T08:24:38.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 Processor -- start 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 -- start start 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 0x7f2ad01a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0109580 0x7f2ad01a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ad019a9b0 0x7f2ad019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad0114610 con 0x7f2ad0109580 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad0114490 con 0x7f2ad01044b0 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.111+0000 7f2ad8800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad0114790 con 0x7f2ad019a9b0 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0109580 0x7f2ad01a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0109580 0x7f2ad01a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54736/0 (socket says 172.21.15.146:54736) 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5400640 1 -- 172.21.15.146:0/1906909423 learned_addr learned my addr 172.21.15.146:0/1906909423 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5e00640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 0x7f2ad01a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad6800640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ad019a9b0 0x7f2ad019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5e00640 1 -- 172.21.15.146:0/1906909423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ad019a9b0 msgr2=0x7f2ad019ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5e00640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ad019a9b0 0x7f2ad019ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5e00640 1 -- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0109580 msgr2=0x7f2ad01a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5e00640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0109580 0x7f2ad01a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5e00640 1 -- 172.21.15.146:0/1906909423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad019b4d0 con 0x7f2ad01044b0 2024-04-24T08:24:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5400640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0109580 0x7f2ad01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.112+0000 7f2ad5e00640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 0x7f2ad01a07b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ac402a860 tx=0x7f2ac4033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.113+0000 7f2acaa00640 1 -- 172.21.15.146:0/1906909423 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4037040 con 0x7f2ad01044b0 2024-04-24T08:24:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.113+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad019b760 con 0x7f2ad01044b0 2024-04-24T08:24:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.113+0000 7f2acaa00640 1 -- 172.21.15.146:0/1906909423 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac4033d50 con 0x7f2ad01044b0 2024-04-24T08:24:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.113+0000 7f2acaa00640 1 -- 172.21.15.146:0/1906909423 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4003810 con 0x7f2ad01044b0 2024-04-24T08:24:38.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.113+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad01a7900 con 0x7f2ad01044b0 2024-04-24T08:24:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 Processor -- start 2024-04-24T08:24:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 -- start start 2024-04-24T08:24:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2acaa00640 1 -- 172.21.15.146:0/1906909423 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f2ac4042020 con 0x7f2ad01044b0 2024-04-24T08:24:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980aa950 0x556e980aad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980b5940 0x556e980b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556e980101d0 con 0x556e980aa950 2024-04-24T08:24:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556e97fb6450 con 0x556e980ab270 2024-04-24T08:24:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.116+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e9800c190 con 0x556e980b5940 2024-04-24T08:24:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980aa950 0x556e980aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980aa950 0x556e980aad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54752/0 (socket says 172.21.15.146:54752) 2024-04-24T08:24:38.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 -- 172.21.15.146:0/2840596016 learned_addr learned my addr 172.21.15.146:0/2840596016 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad5e00640 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980b5940 0x556e980b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 -- 172.21.15.146:0/2840596016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980b5940 msgr2=0x556e980b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980b5940 0x556e980b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 -- 172.21.15.146:0/2840596016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 msgr2=0x556e980b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 -- 172.21.15.146:0/2840596016 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e98073310 con 0x556e980aa950 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.117+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980aa950 0x556e980aad30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009f60 tx=0x7f2acc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f4202fc70ae781 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2ac3000640 1 -- 172.21.15.146:0/2840596016 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc01e070 con 0x556e980aa950 2024-04-24T08:24:38.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2ac3000640 1 -- 172.21.15.146:0/2840596016 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2acc003c30 con 0x556e980aa950 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2ac3000640 1 -- 172.21.15.146:0/2840596016 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc00f9c0 con 0x556e980aa950 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2acaa00640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2ab80a3d20 0x7f2ab80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2acaa00640 1 -- 172.21.15.146:0/1906909423 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2ac40efc80 con 0x7f2ad01044b0 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2ad5e00640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2ab80a3d20 0x7f2ab80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2840596016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980aa950 msgr2=0x556e980aad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.118+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980aa950 0x556e980aad30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009f60 tx=0x7f2acc009f90 comp rx=0 tx=0).stop 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2840596016 shutdown_connections 2024-04-24T08:24:38.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980b5940 0x556e980b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2840596016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980aa950 0x556e980aad30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2840596016 >> 172.21.15.146:0/2840596016 conn(0x556e980a6c70 msgr2=0x556e980a7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2840596016 shutdown_connections 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2840596016 wait complete. 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2ad5e00640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2ab80a3d20 0x7f2ab80a61e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x556e980abf10 tx=0x7f2ac40f3730 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.119+0000 7f2adb7ca740 1 -- start start 2024-04-24T08:24:38.125 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 1 Processor -- start 2024-04-24T08:24:38.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556e980b87d0 con 0x556e980b5940 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556e97fc9810 con 0x556e980ab270 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e980b8950 con 0x556e980aa950 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2adb7ca740 10 client.0 _ll_register_callbacks cb 0x556e97f97140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2ad5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980b53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59642/0 (socket says 172.21.15.146:59642) 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2ad5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39714/0 (socket says 172.21.15.146:39714) 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2ad6800640 1 -- 172.21.15.146:0/2828522121 learned_addr learned my addr 172.21.15.146:0/2828522121 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.120+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ad5400640 1 -- 172.21.15.146:0/2828522121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 msgr2=0x556e980b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ad5400640 1 -- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 msgr2=0x556e980b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ad5400640 1 -- 172.21.15.146:0/2828522121 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e97fda3f0 con 0x556e980ab270 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b26a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2ad010ab40 tx=0x7f2abc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a098b5a19118d71b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ab7400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.121+0000 7f2ab7400640 1 -- 172.21.15.146:0/2828522121 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc004050 con 0x556e980ab270 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2acc003c30 con 0x556e980ab270 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2ab7400640 1 -- 172.21.15.146:0/2828522121 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2abc0041f0 con 0x556e980ab270 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2ab7400640 1 -- 172.21.15.146:0/2828522121 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2abc0043f0 con 0x556e980ab270 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 10 client.26063 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556e97d223f0 con 0x556e980ab270 2024-04-24T08:24:38.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2ab7400640 1 -- 172.21.15.146:0/2828522121 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2acc003c30 con 0x556e980ab270 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2ab7400640 1 -- 172.21.15.146:0/2828522121 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f2abc004cc0 con 0x556e980ab270 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 10 client.26063 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 10 client.26063 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 20 client.26063 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e980c5bd0 0x556e980c8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 4 client.26063 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.122+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e9801dbf0 con 0x556e980c5bd0 2024-04-24T08:24:38.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.123+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e980c5bd0 0x556e980c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.123+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e980c5bd0 0x556e980c8010 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7182576b826b0440 server_cookie=6346a4b658b66caf in_seq=0 out_seq=0 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.123+0000 7f2ab7400640 10 client.26063 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2ab7400640 1 -- 172.21.15.146:0/2828522121 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e9801dbf0 con 0x556e980c5bd0 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2ab7400640 10 client.26063 handle_command_reply: tid=0 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 client.26063 shutdown 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 msgr2=0x556e980b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b26a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2ad010ab40 tx=0x7f2abc00f790 comp rx=0 tx=0).stop 2024-04-24T08:24:38.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 shutdown_connections 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e980c5bd0 0x556e980c8010 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980b26a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2ab7400640 0 client.26063 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2828522121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 >> 172.21.15.146:0/2828522121 conn(0x556e980a6c70 msgr2=0x556e980ac930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.124+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 shutdown_connections 2024-04-24T08:24:38.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.125+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2828522121 wait complete. 2024-04-24T08:24:38.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.125+0000 7f2adb7ca740 20 client.26063 trim_cache size 0 max 16384 2024-04-24T08:24:38.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.139+0000 7f2adb7ca740 1 Processor -- start 2024-04-24T08:24:38.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.139+0000 7f2adb7ca740 1 -- start start 2024-04-24T08:24:38.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980aa950 0x556e980b1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980ab270 0x556e980a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad01b2810 con 0x556e980b5940 2024-04-24T08:24:38.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad01b00e0 con 0x556e980aa950 2024-04-24T08:24:38.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad01adf30 con 0x556e980ab270 2024-04-24T08:24:38.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54768/0 (socket says 172.21.15.146:54768) 2024-04-24T08:24:38.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2ad6800640 1 -- 172.21.15.146:0/2807441838 learned_addr learned my addr 172.21.15.146:0/2807441838 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980ab270 0x556e980a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.140+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980aa950 0x556e980b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ad5400640 1 -- 172.21.15.146:0/2807441838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980aa950 msgr2=0x556e980b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980aa950 0x556e980b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ad5400640 1 -- 172.21.15.146:0/2807441838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 msgr2=0x556e980b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ad5400640 1 -- 172.21.15.146:0/2807441838 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e97dec7d0 con 0x556e980ab270 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ad6800640 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ad5400640 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980ab270 0x556e980a5c70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00ffd0 tx=0x7f2abc011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=107d9afd0a9fdf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ac2600640 1 -- 172.21.15.146:0/2807441838 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc003930 con 0x556e980ab270 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ac2600640 1 -- 172.21.15.146:0/2807441838 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2abc003ad0 con 0x556e980ab270 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.141+0000 7f2ac2600640 1 -- 172.21.15.146:0/2807441838 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc004590 con 0x556e980ab270 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2807441838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980ab270 msgr2=0x556e980a5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980ab270 0x556e980a5c70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00ffd0 tx=0x7f2abc011980 comp rx=0 tx=0).stop 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2807441838 shutdown_connections 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980ab270 0x556e980a5c70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/2807441838 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980aa950 0x556e980b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2807441838 >> 172.21.15.146:0/2807441838 conn(0x556e980a6c70 msgr2=0x556e980a5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2807441838 shutdown_connections 2024-04-24T08:24:38.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 -- 172.21.15.146:0/2807441838 wait complete. 2024-04-24T08:24:38.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.142+0000 7f2adb7ca740 1 -- start start 2024-04-24T08:24:38.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 1 Processor -- start 2024-04-24T08:24:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980be110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980afd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad01ae5a0 con 0x556e980b5940 2024-04-24T08:24:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad01afa10 con 0x556e980ab270 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad01aec70 con 0x556e980aa950 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2adb7ca740 10 client.0 _ll_register_callbacks cb 0x7f2abc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980be110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.143+0000 7f2ad6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39724/0 (socket says 172.21.15.146:39724) 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad5400640 1 -- 172.21.15.146:0/4224812813 learned_addr learned my addr 172.21.15.146:0/4224812813 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad6800640 1 -- 172.21.15.146:0/4224812813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 msgr2=0x556e980be110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad6800640 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980be110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad6800640 1 -- 172.21.15.146:0/4224812813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 msgr2=0x556e980afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad6800640 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad6800640 1 -- 172.21.15.146:0/4224812813 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e980b0730 con 0x556e980b5940 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad5400640 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ad6800640 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b6960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00f4f0 tx=0x7f2acc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fdbb8db2533fda server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.144+0000 7f2ab7400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:24:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2ab7400640 1 -- 172.21.15.146:0/4224812813 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc01e070 con 0x556e980b5940 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2abc003ad0 con 0x556e980b5940 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2ab7400640 1 -- 172.21.15.146:0/4224812813 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2acc00fb00 con 0x556e980b5940 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2ab7400640 1 -- 172.21.15.146:0/4224812813 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2acc03c050 con 0x556e980b5940 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 10 client.15927 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556e98134f00 con 0x556e980b5940 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2ab7400640 1 -- 172.21.15.146:0/4224812813 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2abc003ad0 con 0x556e980b5940 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2ab7400640 1 -- 172.21.15.146:0/4224812813 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f2acc019430 con 0x556e980b5940 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 10 client.15927 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 10 client.15927 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:24:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 20 client.15927 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e98135430 0x556e98137820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 4 client.15927 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.145+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e98017a60 con 0x556e98135430 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.146+0000 7f2ad5400640 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e98135430 0x556e98137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.146+0000 7f2ad5400640 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e98135430 0x556e98137820 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7944afd533fdd37 server_cookie=27d420110fbaa411 in_seq=0 out_seq=0 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.146+0000 7f2ab7400640 10 client.15927 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2ab7400640 1 -- 172.21.15.146:0/4224812813 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e98017a60 con 0x556e98135430 2024-04-24T08:24:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2ab7400640 10 client.15927 handle_command_reply: tid=0 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 client.15927 shutdown 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 msgr2=0x556e980b6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b6960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00f4f0 tx=0x7f2acc009500 comp rx=0 tx=0).stop 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 shutdown_connections 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556e98135430 0x556e98137820 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e980b5940 0x556e980b6960 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556e980ab270 0x556e980afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2ab7400640 0 client.15927 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:24:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 --2- 172.21.15.146:0/4224812813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e980aa950 0x556e980be110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 >> 172.21.15.146:0/4224812813 conn(0x556e980a6c70 msgr2=0x556e980a5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 shutdown_connections 2024-04-24T08:24:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 1 -- 172.21.15.146:0/4224812813 wait complete. 2024-04-24T08:24:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.147+0000 7f2adb7ca740 20 client.15927 trim_cache size 0 max 16384 2024-04-24T08:24:38.156 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:24:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2ab80a3d20 msgr2=0x7f2ab80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2ab80a3d20 0x7f2ab80a61e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x556e980abf10 tx=0x7f2ac40f3730 comp rx=0 tx=0).stop 2024-04-24T08:24:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 msgr2=0x7f2ad01a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 0x7f2ad01a07b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ac402a860 tx=0x7f2ac4033b60 comp rx=0 tx=0).stop 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 shutdown_connections 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2ab80a3d20 0x7f2ab80a61e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ad019a9b0 0x7f2ad019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ad0109580 0x7f2ad01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 --2- 172.21.15.146:0/1906909423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2ad01044b0 0x7f2ad01a07b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 >> 172.21.15.146:0/1906909423 conn(0x7f2ad00ffd90 msgr2=0x7f2ad0101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 shutdown_connections 2024-04-24T08:24:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.148+0000 7f2ad8800640 1 -- 172.21.15.146:0/1906909423 wait complete. 2024-04-24T08:24:38.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:24:38.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 -- 172.21.15.146:0/150603648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104cc0 msgr2=0x7fce74108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 --2- 172.21.15.146:0/150603648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104cc0 0x7fce74108d00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009900 tx=0x7fce5c02a400 comp rx=0 tx=0).stop 2024-04-24T08:24:38.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 -- 172.21.15.146:0/150603648 shutdown_connections 2024-04-24T08:24:38.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 --2- 172.21.15.146:0/150603648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 0x7fce74110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 --2- 172.21.15.146:0/150603648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104cc0 0x7fce74108d00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 --2- 172.21.15.146:0/150603648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce74104310 0x7fce741046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 -- 172.21.15.146:0/150603648 >> 172.21.15.146:0/150603648 conn(0x7fce740ffb70 msgr2=0x7fce74101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 -- 172.21.15.146:0/150603648 shutdown_connections 2024-04-24T08:24:38.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.360+0000 7fce79800640 1 -- 172.21.15.146:0/150603648 wait complete. 2024-04-24T08:24:38.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.361+0000 7fce79800640 1 Processor -- start 2024-04-24T08:24:38.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.361+0000 7fce79800640 1 -- start start 2024-04-24T08:24:38.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.361+0000 7fce79800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104310 0x7fce741a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.361+0000 7fce79800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 0x7fce741a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce79800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7419a820 0x7fce7419acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce79800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fce741144a0 con 0x7fce741093e0 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce79800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fce74114320 con 0x7fce74104310 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce79800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fce74114620 con 0x7fce7419a820 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 0x7fce741a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce73e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104310 0x7fce741a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 0x7fce741a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54796/0 (socket says 172.21.15.146:54796) 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce73e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104310 0x7fce741a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39746/0 (socket says 172.21.15.146:39746) 2024-04-24T08:24:38.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 -- 172.21.15.146:0/2465135528 learned_addr learned my addr 172.21.15.146:0/2465135528 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce78e00640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7419a820 0x7fce7419acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 -- 172.21.15.146:0/2465135528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7419a820 msgr2=0x7fce7419acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7419a820 0x7fce7419acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 -- 172.21.15.146:0/2465135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104310 msgr2=0x7fce741a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104310 0x7fce741a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce6be00640 1 -- 172.21.15.146:0/2465135528 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7419b3b0 con 0x7fce741093e0 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.362+0000 7fce73e00640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104310 0x7fce741a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.363+0000 7fce6be00640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 0x7fce741a0bf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fce5c02a920 tx=0x7fce5c003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.363+0000 7fce71600640 1 -- 172.21.15.146:0/2465135528 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c03f070 con 0x7fce741093e0 2024-04-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.363+0000 7fce71600640 1 -- 172.21.15.146:0/2465135528 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce5c031c90 con 0x7fce741093e0 2024-04-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.363+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7419b640 con 0x7fce741093e0 2024-04-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.363+0000 7fce71600640 1 -- 172.21.15.146:0/2465135528 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c035030 con 0x7fce741093e0 2024-04-24T08:24:38.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.363+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce741a7820 con 0x7fce741093e0 2024-04-24T08:24:38.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.366+0000 7fce7dbf7740 1 Processor -- start 2024-04-24T08:24:38.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.366+0000 7fce71600640 1 -- 172.21.15.146:0/2465135528 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fce5c042070 con 0x7fce741093e0 2024-04-24T08:24:38.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.366+0000 7fce7dbf7740 1 -- start start 2024-04-24T08:24:38.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bd5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd6460 0x557550be05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550be0b30 0x557550be2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557550b3b1d0 con 0x557550bd6460 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557550ae1450 con 0x557550bd5b40 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557550b37190 con 0x557550be0b30 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce73e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd6460 0x557550be05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce78e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bd5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce73e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd6460 0x557550be05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54808/0 (socket says 172.21.15.146:54808) 2024-04-24T08:24:38.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce73e00640 1 -- 172.21.15.146:0/3509671918 learned_addr learned my addr 172.21.15.146:0/3509671918 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.367+0000 7fce6be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550be0b30 0x557550be2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce78e00640 1 -- 172.21.15.146:0/3509671918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550be0b30 msgr2=0x557550be2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce78e00640 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550be0b30 0x557550be2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce78e00640 1 -- 172.21.15.146:0/3509671918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd6460 msgr2=0x557550be05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce78e00640 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd6460 0x557550be05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce78e00640 1 -- 172.21.15.146:0/3509671918 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557550b9e310 con 0x557550bd5b40 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce73e00640 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd6460 0x557550be05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:38.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce78e00640 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bd5f20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009f10 tx=0x7fce6c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c9991cdba02a9aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce71600640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fce580a3da0 0x7fce580a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce69600640 1 -- 172.21.15.146:0/3509671918 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c004d20 con 0x557550bd5b40 2024-04-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce69600640 1 -- 172.21.15.146:0/3509671918 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c004050 con 0x557550bd5b40 2024-04-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce71600640 1 -- 172.21.15.146:0/2465135528 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fce5c0f1100 con 0x7fce741093e0 2024-04-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce69600640 1 -- 172.21.15.146:0/3509671918 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c005260 con 0x557550bd5b40 2024-04-24T08:24:38.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.368+0000 7fce73e00640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fce580a3da0 0x7fce580a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3509671918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 msgr2=0x557550bd5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bd5f20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009f10 tx=0x7fce6c011810 comp rx=0 tx=0).stop 2024-04-24T08:24:38.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce73e00640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fce580a3da0 0x7fce580a6260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x557550bd7100 tx=0x7fce64005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3509671918 shutdown_connections 2024-04-24T08:24:38.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550be0b30 0x557550be2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd6460 0x557550be05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3509671918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bd5f20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3509671918 >> 172.21.15.146:0/3509671918 conn(0x557550bd1e60 msgr2=0x557550bd2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3509671918 shutdown_connections 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.369+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3509671918 wait complete. 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.370+0000 7fce7dbf7740 1 -- start start 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.370+0000 7fce7dbf7740 1 Processor -- start 2024-04-24T08:24:38.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.370+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd5b40 0x557550bdd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.370+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bdde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.370+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550be0b30 0x557550bde370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.370+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557550be39c0 con 0x557550bd5b40 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557550af4810 con 0x557550be0b30 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557550be3b40 con 0x557550bd6460 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce7dbf7740 10 client.0 _ll_register_callbacks cb 0x557550ac2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd5b40 0x557550bdd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd5b40 0x557550bdd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54810/0 (socket says 172.21.15.146:54810) 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 -- 172.21.15.146:0/3398870596 learned_addr learned my addr 172.21.15.146:0/3398870596 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce78e00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550be0b30 0x557550bde370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce6be00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bdde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 -- 172.21.15.146:0/3398870596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 msgr2=0x557550bdde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bdde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 -- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550be0b30 msgr2=0x557550bde370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550be0b30 0x557550bde370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce73e00640 1 -- 172.21.15.146:0/3398870596 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557550b053f0 con 0x557550bd5b40 2024-04-24T08:24:38.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.371+0000 7fce78e00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550be0b30 0x557550bde370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce73e00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd5b40 0x557550bdd8f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce64004620 tx=0x7fce64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bdccb4549ddf94 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce62000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce62000640 1 -- 172.21.15.146:0/3398870596 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64003af0 con 0x557550bd5b40 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce62000640 1 -- 172.21.15.146:0/3398870596 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fce64003c90 con 0x557550bd5b40 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce6c004050 con 0x557550bd5b40 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce62000640 1 -- 172.21.15.146:0/3398870596 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fce64005020 con 0x557550bd5b40 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce7dbf7740 10 client.15939 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:38.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55755084d3f0 con 0x557550bd5b40 2024-04-24T08:24:38.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.372+0000 7fce62000640 1 -- 172.21.15.146:0/3398870596 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c004050 con 0x557550bd5b40 2024-04-24T08:24:38.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce62000640 1 -- 172.21.15.146:0/3398870596 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fce64027070 con 0x557550bd5b40 2024-04-24T08:24:38.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce7dbf7740 10 client.15939 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:38.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce7dbf7740 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:38.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce7dbf7740 20 client.15939 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:38.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550bf0d80 0x557550bf3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce7dbf7740 4 client.15939 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557550b48bf0 con 0x557550bf0d80 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.373+0000 7fce6be00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550bf0d80 0x557550bf3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.374+0000 7fce6be00640 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550bf0d80 0x557550bf3200 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fcb68ce076ebbee server_cookie=bbde339282a1adbe in_seq=0 out_seq=0 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.374+0000 7fce62000640 10 client.15939 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce62000640 1 -- 172.21.15.146:0/3398870596 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557550b48bf0 con 0x557550bf0d80 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce62000640 10 client.15939 handle_command_reply: tid=0 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 client.15939 shutdown 2024-04-24T08:24:38.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd5b40 msgr2=0x557550bdd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd5b40 0x557550bdd8f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce64004620 tx=0x7fce64004730 comp rx=0 tx=0).stop 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 shutdown_connections 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550bf0d80 0x557550bf3200 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bdde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550be0b30 0x557550bde370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3398870596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550bd5b40 0x557550bdd8f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce62000640 0 client.15939 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 >> 172.21.15.146:0/3398870596 conn(0x557550bd1e60 msgr2=0x557550bd7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 shutdown_connections 2024-04-24T08:24:38.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3398870596 wait complete. 2024-04-24T08:24:38.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.375+0000 7fce7dbf7740 20 client.15939 trim_cache size 0 max 16384 2024-04-24T08:24:38.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 Processor -- start 2024-04-24T08:24:38.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 -- start start 2024-04-24T08:24:38.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bf8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550bf9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fce741b2670 con 0x557550be0b30 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fce741aff40 con 0x557550bd5b40 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.390+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fce741addb0 con 0x557550bd6460 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce78e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550bf9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce6be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bf8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce73e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce6be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bf8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59696/0 (socket says 172.21.15.146:59696) 2024-04-24T08:24:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce6be00640 1 -- 172.21.15.146:0/3481724145 learned_addr learned my addr 172.21.15.146:0/3481724145 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce78e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550bf9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54826/0 (socket says 172.21.15.146:54826) 2024-04-24T08:24:38.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce73e00640 1 -- 172.21.15.146:0/3481724145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 msgr2=0x557550bf8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce73e00640 1 --2- 172.21.15.146:0/3481724145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bf8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce73e00640 1 -- 172.21.15.146:0/3481724145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 msgr2=0x557550bf9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce73e00640 1 --2- 172.21.15.146:0/3481724145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550bf9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce73e00640 1 -- 172.21.15.146:0/3481724145 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575509177d0 con 0x557550bd5b40 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.391+0000 7fce73e00640 1 --2- 172.21.15.146:0/3481724145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bec880 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce64004e10 tx=0x7fce640092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292aa2d237a5343a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce68c00640 1 -- 172.21.15.146:0/3481724145 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64034020 con 0x557550bd5b40 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce68c00640 1 -- 172.21.15.146:0/3481724145 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce640049a0 con 0x557550bd5b40 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce68c00640 1 -- 172.21.15.146:0/3481724145 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6401f430 con 0x557550bd5b40 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3481724145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 msgr2=0x557550bec880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3481724145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bec880 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce64004e10 tx=0x7fce640092a0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3481724145 shutdown_connections 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3481724145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550bf8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3481724145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550bf9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/3481724145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550bec880 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.392+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3481724145 >> 172.21.15.146:0/3481724145 conn(0x557550bd1e60 msgr2=0x557550bdc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.393+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3481724145 shutdown_connections 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.393+0000 7fce7dbf7740 1 -- 172.21.15.146:0/3481724145 wait complete. 2024-04-24T08:24:38.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.393+0000 7fce7dbf7740 1 -- start start 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.393+0000 7fce7dbf7740 1 Processor -- start 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550be16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550be1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce7dbf7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550be7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fce741ae400 con 0x557550be0b30 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fce741af870 con 0x557550bd5b40 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce7dbf7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fce741aead0 con 0x557550bd6460 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce7dbf7740 10 client.0 _ll_register_callbacks cb 0x7fce64012f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:38.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce78e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550be7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce73e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550be16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce78e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550be7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54832/0 (socket says 172.21.15.146:54832) 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce78e00640 1 -- 172.21.15.146:0/236514447 learned_addr learned my addr 172.21.15.146:0/236514447 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce6be00640 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550be1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce73e00640 1 -- 172.21.15.146:0/236514447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 msgr2=0x557550be1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce73e00640 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550be1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce73e00640 1 -- 172.21.15.146:0/236514447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 msgr2=0x557550be7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce73e00640 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550be7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce73e00640 1 -- 172.21.15.146:0/236514447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557550be8400 con 0x557550bd5b40 2024-04-24T08:24:38.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.394+0000 7fce78e00640 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550be7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:38.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce73e00640 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550be16e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fce6400f800 tx=0x7fce64012740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb8473c59b94e92 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:38.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce62000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:38.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce62000640 1 -- 172.21.15.146:0/236514447 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64034030 con 0x557550bd5b40 2024-04-24T08:24:38.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce62000640 1 -- 172.21.15.146:0/236514447 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fce640039c0 con 0x557550bd5b40 2024-04-24T08:24:38.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce640049a0 con 0x557550bd5b40 2024-04-24T08:24:38.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce62000640 1 -- 172.21.15.146:0/236514447 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fce6403d070 con 0x557550bd5b40 2024-04-24T08:24:38.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce7dbf7740 10 client.26087 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:38.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.395+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557550c5fb50 con 0x557550bd5b40 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce62000640 1 -- 172.21.15.146:0/236514447 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce640049a0 con 0x557550bd5b40 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce62000640 1 -- 172.21.15.146:0/236514447 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fce64027070 con 0x557550bd5b40 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce7dbf7740 10 client.26087 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce7dbf7740 10 client.26087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce7dbf7740 20 client.26087 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550c5ff70 0x557550c62360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce7dbf7740 4 client.26087 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:24:38.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557550b42a60 con 0x557550c5ff70 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.396+0000 7fce6be00640 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550c5ff70 0x557550c62360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce6be00640 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550c5ff70 0x557550c62360 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67c6e4ee46a5a97d server_cookie=688c94a028b289bc in_seq=0 out_seq=0 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce62000640 10 client.26087 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce62000640 1 -- 172.21.15.146:0/236514447 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557550b42a60 con 0x557550c5ff70 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce62000640 10 client.26087 handle_command_reply: tid=0 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 client.26087 shutdown 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 msgr2=0x557550be16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550be16e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fce6400f800 tx=0x7fce64012740 comp rx=0 tx=0).stop 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 shutdown_connections 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557550c5ff70 0x557550c62360 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557550bd6460 0x557550be1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557550be0b30 0x557550be7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 --2- 172.21.15.146:0/236514447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557550bd5b40 0x557550be16e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce62000640 0 client.26087 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.397+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 >> 172.21.15.146:0/236514447 conn(0x557550bd1e60 msgr2=0x557550bd06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 shutdown_connections 2024-04-24T08:24:38.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce7dbf7740 1 -- 172.21.15.146:0/236514447 wait complete. 2024-04-24T08:24:38.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce7dbf7740 20 client.26087 trim_cache size 0 max 16384 2024-04-24T08:24:38.407 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:24:38.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fce580a3da0 msgr2=0x7fce580a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fce580a3da0 0x7fce580a6260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x557550bd7100 tx=0x7fce64005f90 comp rx=0 tx=0).stop 2024-04-24T08:24:38.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 msgr2=0x7fce741a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:38.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 0x7fce741a0bf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fce5c02a920 tx=0x7fce5c003b20 comp rx=0 tx=0).stop 2024-04-24T08:24:38.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 shutdown_connections 2024-04-24T08:24:38.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fce580a3da0 0x7fce580a6260 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce7419a820 0x7fce7419acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce741093e0 0x7fce741a0bf0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 --2- 172.21.15.146:0/2465135528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce74104310 0x7fce741a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:38.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.398+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 >> 172.21.15.146:0/2465135528 conn(0x7fce740ffb70 msgr2=0x7fce74100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:38.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.399+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 shutdown_connections 2024-04-24T08:24:38.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:38.399+0000 7fce79800640 1 -- 172.21.15.146:0/2465135528 wait complete. 2024-04-24T08:24:38.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:38 smithi146 ceph-mon[27805]: pgmap v106: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 72 KiB/s wr, 11 op/s 2024-04-24T08:24:38.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:38 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3134906527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:24:38.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:38 smithi177 ceph-mon[33046]: pgmap v106: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 72 KiB/s wr, 11 op/s 2024-04-24T08:24:38.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:38 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3134906527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:24:38.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:38 smithi153 ceph-mon[32746]: pgmap v106: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 72 KiB/s wr, 11 op/s 2024-04-24T08:24:38.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:38 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3134906527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:24:39.420 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:24:39.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:24:39.421 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:24:39.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 -- 172.21.15.146:0/263264678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb834104e60 msgr2=0x7fb834108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 --2- 172.21.15.146:0/263264678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb834104e60 0x7fb834108ea0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb820009900 tx=0x7fb82002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 -- 172.21.15.146:0/263264678 shutdown_connections 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 --2- 172.21.15.146:0/263264678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb834109580 0x7fb834111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 --2- 172.21.15.146:0/263264678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb834104e60 0x7fb834108ea0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 --2- 172.21.15.146:0/263264678 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8341044b0 0x7fb834104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 -- 172.21.15.146:0/263264678 >> 172.21.15.146:0/263264678 conn(0x7fb8340ffd90 msgr2=0x7fb8341021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 -- 172.21.15.146:0/263264678 shutdown_connections 2024-04-24T08:24:39.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.636+0000 7fb83c400640 1 -- 172.21.15.146:0/263264678 wait complete. 2024-04-24T08:24:39.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 Processor -- start 2024-04-24T08:24:39.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 -- start start 2024-04-24T08:24:39.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8341044b0 0x7fb8341a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb834109580 0x7fb8341a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb83419a9a0 0x7fb83419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb834114660 con 0x7fb83419a9a0 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341144e0 con 0x7fb834109580 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.638+0000 7fb83c400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341147e0 con 0x7fb8341044b0 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb83419a9a0 0x7fb83419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb839a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8341044b0 0x7fb8341a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb83419a9a0 0x7fb83419ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54860/0 (socket says 172.21.15.146:54860) 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb83a400640 1 -- 172.21.15.146:0/615701722 learned_addr learned my addr 172.21.15.146:0/615701722 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb839000640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb834109580 0x7fb8341a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb839a00640 1 -- 172.21.15.146:0/615701722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb834109580 msgr2=0x7fb8341a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb839a00640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb834109580 0x7fb8341a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb839a00640 1 -- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb83419a9a0 msgr2=0x7fb83419ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb839a00640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb83419a9a0 0x7fb83419ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb839a00640 1 -- 172.21.15.146:0/615701722 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83419b4c0 con 0x7fb8341044b0 2024-04-24T08:24:39.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.639+0000 7fb83a400640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb83419a9a0 0x7fb83419ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:39.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.640+0000 7fb839a00640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8341044b0 0x7fb8341a07f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb82400eb00 tx=0x7fb82400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.640+0000 7fb832000640 1 -- 172.21.15.146:0/615701722 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824010430 con 0x7fb8341044b0 2024-04-24T08:24:39.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.640+0000 7fb832000640 1 -- 172.21.15.146:0/615701722 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb824010ab0 con 0x7fb8341044b0 2024-04-24T08:24:39.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.640+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83419b7b0 con 0x7fb8341044b0 2024-04-24T08:24:39.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.640+0000 7fb832000640 1 -- 172.21.15.146:0/615701722 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824017bb0 con 0x7fb8341044b0 2024-04-24T08:24:39.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.640+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8341a7a50 con 0x7fb8341044b0 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.643+0000 7fb832000640 1 -- 172.21.15.146:0/615701722 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fb824010c60 con 0x7fb8341044b0 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.643+0000 7fb83f167740 1 Processor -- start 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.643+0000 7fb83f167740 1 -- start start 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83f167740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f1090 0x55db2f6f1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83f167740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f19b0 0x55db2f6fbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83f167740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f6fe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83f167740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55db2f656e70 con 0x55db2f6fc080 2024-04-24T08:24:39.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83f167740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55db2f5fd380 con 0x55db2f6f19b0 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83f167740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55db2f652e30 con 0x55db2f6f1090 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f1090 0x55db2f6f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb839a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f19b0 0x55db2f6fbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f1090 0x55db2f6f1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59750/0 (socket says 172.21.15.146:59750) 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83a400640 1 -- 172.21.15.146:0/2513292645 learned_addr learned my addr 172.21.15.146:0/2513292645 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb839000640 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f6fe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83a400640 1 -- 172.21.15.146:0/2513292645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f19b0 msgr2=0x55db2f6fbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.644+0000 7fb83a400640 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f19b0 0x55db2f6fbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb83a400640 1 -- 172.21.15.146:0/2513292645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 msgr2=0x55db2f6fe470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb83a400640 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f6fe470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb83a400640 1 -- 172.21.15.146:0/2513292645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db2f4317e0 con 0x55db2f6f1090 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb839000640 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f6fe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb83a400640 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f1090 0x55db2f6f1470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb82c011570 tx=0x7fb82c011a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53d4ddd3290c7494 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb82aa00640 1 -- 172.21.15.146:0/2513292645 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c01e070 con 0x55db2f6f1090 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb82aa00640 1 -- 172.21.15.146:0/2513292645 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb82c00fb20 con 0x55db2f6f1090 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb832000640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb8180a3da0 0x7fb8180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb82aa00640 1 -- 172.21.15.146:0/2513292645 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c019c10 con 0x55db2f6f1090 2024-04-24T08:24:39.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb832000640 1 -- 172.21.15.146:0/615701722 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb8240d1130 con 0x7fb8341044b0 2024-04-24T08:24:39.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.645+0000 7fb839a00640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb8180a3da0 0x7fb8180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 -- 172.21.15.146:0/2513292645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f1090 msgr2=0x55db2f6f1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f1090 0x55db2f6f1470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb82c011570 tx=0x7fb82c011a30 comp rx=0 tx=0).stop 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 -- 172.21.15.146:0/2513292645 shutdown_connections 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f6fe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f19b0 0x55db2f6fbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 --2- 172.21.15.146:0/2513292645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f1090 0x55db2f6f1470 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 -- 172.21.15.146:0/2513292645 >> 172.21.15.146:0/2513292645 conn(0x55db2f6ed3b0 msgr2=0x55db2f6ed790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb839a00640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb8180a3da0 0x7fb8180a6260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55db2f6f2650 tx=0x7fb8240d4be0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 -- 172.21.15.146:0/2513292645 shutdown_connections 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.646+0000 7fb83f167740 1 -- 172.21.15.146:0/2513292645 wait complete. 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 -- start start 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 Processor -- start 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f6f9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6f9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f6f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55db2f6fee70 con 0x55db2f6f1090 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55db2f610660 con 0x55db2f6fc080 2024-04-24T08:24:39.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55db2f6feff0 con 0x55db2f6f19b0 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.647+0000 7fb83f167740 10 client.0 _ll_register_callbacks cb 0x55db2f5feaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f6f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb839000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6f9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f6f9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54874/0 (socket says 172.21.15.146:54874) 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 -- 172.21.15.146:0/630211665 learned_addr learned my addr 172.21.15.146:0/630211665 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb839000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6f9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:59756/0 (socket says 172.21.15.146:59756) 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f6f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 -- 172.21.15.146:0/630211665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 msgr2=0x55db2f6f9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6f9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 -- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 msgr2=0x55db2f6f9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f6f9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 -- 172.21.15.146:0/630211665 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db2f491f70 con 0x55db2f6f1090 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb839000640 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6f9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb83a400640 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f6f9050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009f00 tx=0x7fb82c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48dbe2bf94ce73c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.648+0000 7fb81f400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:24:39.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.649+0000 7fb81f400640 1 -- 172.21.15.146:0/630211665 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c02b060 con 0x55db2f6f1090 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.649+0000 7fb81f400640 1 -- 172.21.15.146:0/630211665 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb82c0199f0 con 0x55db2f6f1090 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.649+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb82c00fb20 con 0x55db2f6f1090 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.649+0000 7fb81f400640 1 -- 172.21.15.146:0/630211665 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb82c01e070 con 0x55db2f6f1090 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.649+0000 7fb83f167740 10 client.15957 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.649+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55db2f329170 con 0x55db2f6f1090 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.649+0000 7fb81f400640 1 -- 172.21.15.146:0/630211665 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb82c00fb20 con 0x55db2f6f1090 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb81f400640 1 -- 172.21.15.146:0/630211665 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb82c004c10 con 0x55db2f6f1090 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb83f167740 10 client.15957 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:39.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb83f167740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb83f167740 20 client.15957 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb83f167740 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f70e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb83f167740 4 client.15957 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db2f664890 con 0x55db2f70c3f0 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.650+0000 7fb839000640 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f70e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.651+0000 7fb839000640 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f70e7e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b7dc44fadebf04 server_cookie=6a716eacd06df8ca in_seq=0 out_seq=0 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.651+0000 7fb81f400640 10 client.15957 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb81f400640 1 -- 172.21.15.146:0/630211665 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db2f664890 con 0x55db2f70c3f0 2024-04-24T08:24:39.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb81f400640 10 client.15957 handle_command_reply: tid=0 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 client.15957 shutdown 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 msgr2=0x55db2f6f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f6f9050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009f00 tx=0x7fb82c014000 comp rx=0 tx=0).stop 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 shutdown_connections 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f70e7e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6f9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f6f9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb81f400640 0 client.15957 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:39.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 --2- 172.21.15.146:0/630211665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f6f9050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 >> 172.21.15.146:0/630211665 conn(0x55db2f6ed3b0 msgr2=0x55db2f6f31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 shutdown_connections 2024-04-24T08:24:39.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 1 -- 172.21.15.146:0/630211665 wait complete. 2024-04-24T08:24:39.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.652+0000 7fb83f167740 20 client.15957 trim_cache size 0 max 16384 2024-04-24T08:24:39.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 Processor -- start 2024-04-24T08:24:39.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 -- start start 2024-04-24T08:24:39.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f707d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6ec390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f713f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341b28b0 con 0x55db2f6f1090 2024-04-24T08:24:39.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341b0180 con 0x55db2f6fc080 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.672+0000 7fb83f167740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341adfd0 con 0x55db2f6f19b0 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f707d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f707d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54880/0 (socket says 172.21.15.146:54880) 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 -- 172.21.15.146:0/3590985007 learned_addr learned my addr 172.21.15.146:0/3590985007 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb839000640 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6ec390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f713f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 -- 172.21.15.146:0/3590985007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 msgr2=0x55db2f6ec390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6ec390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 -- 172.21.15.146:0/3590985007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 msgr2=0x55db2f713f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f713f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 -- 172.21.15.146:0/3590985007 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db2f6fe470 con 0x55db2f6f1090 2024-04-24T08:24:39.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb839000640 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6ec390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:39.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb83a400640 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f707d30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009cb0 tx=0x7fb82c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8262b3d1328e3eb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.673+0000 7fb82a000640 1 -- 172.21.15.146:0/3590985007 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c02b040 con 0x55db2f6f1090 2024-04-24T08:24:39.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb82a000640 1 -- 172.21.15.146:0/3590985007 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb82c01f040 con 0x55db2f6f1090 2024-04-24T08:24:39.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb82a000640 1 -- 172.21.15.146:0/3590985007 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c023d70 con 0x55db2f6f1090 2024-04-24T08:24:39.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb83f167740 1 -- 172.21.15.146:0/3590985007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 msgr2=0x55db2f707d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb83f167740 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f707d30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb82c009cb0 tx=0x7fb82c001e70 comp rx=0 tx=0).stop 2024-04-24T08:24:39.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb83f167740 1 -- 172.21.15.146:0/3590985007 shutdown_connections 2024-04-24T08:24:39.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb83f167740 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f6ec390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb83f167740 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6fc080 0x55db2f713f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb83f167740 1 --2- 172.21.15.146:0/3590985007 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6f1090 0x55db2f707d30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.674+0000 7fb83f167740 1 -- 172.21.15.146:0/3590985007 >> 172.21.15.146:0/3590985007 conn(0x55db2f6ed3b0 msgr2=0x55db2f6ebc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.675+0000 7fb83f167740 1 -- 172.21.15.146:0/3590985007 shutdown_connections 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.675+0000 7fb83f167740 1 -- 172.21.15.146:0/3590985007 wait complete. 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.675+0000 7fb83f167740 1 -- start start 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.675+0000 7fb83f167740 1 Processor -- start 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83f167740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f1090 0x55db2f6ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83f167740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f700290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83f167740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f7007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83f167740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341ae640 con 0x55db2f6fc080 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83f167740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341afab0 con 0x55db2f6f1090 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83f167740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8341aed10 con 0x55db2f6f19b0 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83f167740 10 client.0 _ll_register_callbacks cb 0x7fb82c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f7007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f7007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54888/0 (socket says 172.21.15.146:54888) 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 -- 172.21.15.146:0/2029270446 learned_addr learned my addr 172.21.15.146:0/2029270446 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb839000640 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f700290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f1090 0x55db2f6ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 -- 172.21.15.146:0/2029270446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 msgr2=0x55db2f700290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f700290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 -- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f1090 msgr2=0x55db2f6ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f1090 0x55db2f6ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.676+0000 7fb83a400640 1 -- 172.21.15.146:0/2029270446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db2f771870 con 0x55db2f6fc080 2024-04-24T08:24:39.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb83a400640 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f7007d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0059f0 tx=0x7fb82c023950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed209c6fbb7e0c19 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb81f400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb81f400640 1 -- 172.21.15.146:0/2029270446 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c02b030 con 0x55db2f6fc080 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb81f400640 1 -- 172.21.15.146:0/2029270446 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb82c023bf0 con 0x55db2f6fc080 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb82c01f040 con 0x55db2f6fc080 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb81f400640 1 -- 172.21.15.146:0/2029270446 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb82c00fb00 con 0x55db2f6fc080 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb83f167740 10 client.15963 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55db2f772b90 con 0x55db2f6fc080 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.677+0000 7fb81f400640 1 -- 172.21.15.146:0/2029270446 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb82c01f040 con 0x55db2f6fc080 2024-04-24T08:24:39.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb81f400640 1 -- 172.21.15.146:0/2029270446 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb82c038450 con 0x55db2f6fc080 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb83f167740 10 client.15963 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb83f167740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb83f167740 20 client.15963 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb83f167740 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f7751e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb83f167740 4 client.15963 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55db2f65e700 con 0x55db2f70c3f0 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.678+0000 7fb839000640 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f7751e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.679+0000 7fb839000640 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f7751e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60141ccb108dde7a server_cookie=c00b373dc085c5c1 in_seq=0 out_seq=0 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.679+0000 7fb81f400640 10 client.15963 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb81f400640 1 -- 172.21.15.146:0/2029270446 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55db2f65e700 con 0x55db2f70c3f0 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb81f400640 10 client.15963 handle_command_reply: tid=0 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 client.15963 shutdown 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 msgr2=0x55db2f7007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f7007d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0059f0 tx=0x7fb82c023950 comp rx=0 tx=0).stop 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 shutdown_connections 2024-04-24T08:24:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55db2f70c3f0 0x55db2f7751e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db2f6f19b0 0x55db2f700290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55db2f6fc080 0x55db2f7007d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 --2- 172.21.15.146:0/2029270446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55db2f6f1090 0x55db2f6ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 >> 172.21.15.146:0/2029270446 conn(0x55db2f6ed3b0 msgr2=0x55db2f6ec510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.680+0000 7fb81f400640 0 client.15963 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.681+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 shutdown_connections 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.681+0000 7fb83f167740 1 -- 172.21.15.146:0/2029270446 wait complete. 2024-04-24T08:24:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.681+0000 7fb83f167740 20 client.15963 trim_cache size 0 max 16384 2024-04-24T08:24:39.691 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:24:39.691 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:24:39.691 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "de2c1ea7-a1a2-4942-9608-289083aff439", 2024-04-24T08:24:39.691 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:24:39.691 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:24:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.681+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb8180a3da0 msgr2=0x7fb8180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.681+0000 7fb83c400640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb8180a3da0 0x7fb8180a6260 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55db2f6f2650 tx=0x7fb8240d4be0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.681+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8341044b0 msgr2=0x7fb8341a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.681+0000 7fb83c400640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8341044b0 0x7fb8341a07f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb82400eb00 tx=0x7fb82400c2d0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 shutdown_connections 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb8180a3da0 0x7fb8180a6260 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb83419a9a0 0x7fb83419ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb834109580 0x7fb8341a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 --2- 172.21.15.146:0/615701722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8341044b0 0x7fb8341a07f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 >> 172.21.15.146:0/615701722 conn(0x7fb8340ffd90 msgr2=0x7fb8341020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 shutdown_connections 2024-04-24T08:24:39.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.682+0000 7fb83c400640 1 -- 172.21.15.146:0/615701722 wait complete. 2024-04-24T08:24:39.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:24:39.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:24:39.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 -- 172.21.15.146:0/4251273380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c1060d0 msgr2=0x7f2d2c10dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 --2- 172.21.15.146:0/4251273380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c1060d0 0x7f2d2c10dca0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009ca0 tx=0x7f2d2402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 -- 172.21.15.146:0/4251273380 shutdown_connections 2024-04-24T08:24:39.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 --2- 172.21.15.146:0/4251273380 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c1060d0 0x7f2d2c10dca0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 --2- 172.21.15.146:0/4251273380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d2c1019b0 0x7f2d2c1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 --2- 172.21.15.146:0/4251273380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d2c101000 0x7f2d2c1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 -- 172.21.15.146:0/4251273380 >> 172.21.15.146:0/4251273380 conn(0x7f2d2c0fc880 msgr2=0x7f2d2c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 -- 172.21.15.146:0/4251273380 shutdown_connections 2024-04-24T08:24:39.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.959+0000 7f2d32200640 1 -- 172.21.15.146:0/4251273380 wait complete. 2024-04-24T08:24:39.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.961+0000 7f2d32200640 1 Processor -- start 2024-04-24T08:24:39.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.961+0000 7f2d32200640 1 -- start start 2024-04-24T08:24:39.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d32200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c101000 0x7f2d2c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d32200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d2c1060d0 0x7f2d2c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d32200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d2c19a730 0x7f2d2c19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d30e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c101000 0x7f2d2c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d30e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c101000 0x7f2d2c1a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39834/0 (socket says 172.21.15.146:39834) 2024-04-24T08:24:39.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d32200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1100c0 con 0x7f2d2c19a730 2024-04-24T08:24:39.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c10ff40 con 0x7f2d2c101000 2024-04-24T08:24:39.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d30e00640 1 -- 172.21.15.146:0/2124345263 learned_addr learned my addr 172.21.15.146:0/2124345263 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.962+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c110240 con 0x7f2d2c1060d0 2024-04-24T08:24:39.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.963+0000 7f2d2be00640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d2c1060d0 0x7f2d2c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.963+0000 7f2d31800640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d2c19a730 0x7f2d2c19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.963+0000 7f2d30e00640 1 -- 172.21.15.146:0/2124345263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d2c1060d0 msgr2=0x7f2d2c1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.963+0000 7f2d30e00640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d2c1060d0 0x7f2d2c1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.963+0000 7f2d30e00640 1 -- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d2c19a730 msgr2=0x7f2d2c19abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.963+0000 7f2d30e00640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d2c19a730 0x7f2d2c19abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.963+0000 7f2d30e00640 1 -- 172.21.15.146:0/2124345263 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c19b330 con 0x7f2d2c101000 2024-04-24T08:24:39.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.964+0000 7f2d30e00640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c101000 0x7f2d2c1a0670 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2d140097d0 tx=0x7f2d14009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.964+0000 7f2d29600640 1 -- 172.21.15.146:0/2124345263 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1400eab0 con 0x7f2d2c101000 2024-04-24T08:24:39.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.964+0000 7f2d29600640 1 -- 172.21.15.146:0/2124345263 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d14019070 con 0x7f2d2c101000 2024-04-24T08:24:39.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.964+0000 7f2d29600640 1 -- 172.21.15.146:0/2124345263 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14013380 con 0x7f2d2c101000 2024-04-24T08:24:39.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.964+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2c1a7410 con 0x7f2d2c101000 2024-04-24T08:24:39.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.964+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2c1a7870 con 0x7f2d2c101000 2024-04-24T08:24:39.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.970+0000 7f2d3665e740 1 Processor -- start 2024-04-24T08:24:39.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d3665e740 1 -- start start 2024-04-24T08:24:39.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599eff8de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff984e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff9ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d3665e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5599efef31f0 con 0x5599eff8da30 2024-04-24T08:24:39.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d3665e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5599efe99350 con 0x5599eff8e350 2024-04-24T08:24:39.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d3665e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599efeef1b0 con 0x5599eff98a20 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.971+0000 7f2d31800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599eff8de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d31800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599eff8de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54904/0 (socket says 172.21.15.146:54904) 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d31800640 1 -- 172.21.15.146:0/2479856460 learned_addr learned my addr 172.21.15.146:0/2479856460 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d2be00640 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff9ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d31800640 1 -- 172.21.15.146:0/2479856460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 msgr2=0x5599eff9ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d31800640 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff9ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d31800640 1 -- 172.21.15.146:0/2479856460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 msgr2=0x5599eff984e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:24:39.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d31800640 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff984e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d31800640 1 -- 172.21.15.146:0/2479856460 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599efeedf50 con 0x5599eff8da30 2024-04-24T08:24:39.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.972+0000 7f2d29600640 1 -- 172.21.15.146:0/2124345263 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f2d14014420 con 0x7f2d2c101000 2024-04-24T08:24:39.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.973+0000 7f2d2be00640 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff9ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:24:39.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.973+0000 7f2d31800640 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599eff8de10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2d24004af0 tx=0x7f2d24005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5648665efd519390 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.974+0000 7f2d22000640 1 -- 172.21.15.146:0/2479856460 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24031db0 con 0x5599eff8da30 2024-04-24T08:24:39.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.974+0000 7f2d22000640 1 -- 172.21.15.146:0/2479856460 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d24036070 con 0x5599eff8da30 2024-04-24T08:24:39.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.974+0000 7f2d22000640 1 -- 172.21.15.146:0/2479856460 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240042a0 con 0x5599eff8da30 2024-04-24T08:24:39.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.975+0000 7f2d29600640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d100a3e20 0x7f2d100a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.975+0000 7f2d29600640 1 -- 172.21.15.146:0/2124345263 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2d140cfbe0 con 0x7f2d2c101000 2024-04-24T08:24:39.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d3665e740 1 -- 172.21.15.146:0/2479856460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 msgr2=0x5599eff8de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d3665e740 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599eff8de10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2d24004af0 tx=0x7f2d24005f10 comp rx=0 tx=0).stop 2024-04-24T08:24:39.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d3665e740 1 -- 172.21.15.146:0/2479856460 shutdown_connections 2024-04-24T08:24:39.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d3665e740 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff9ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d3665e740 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d3665e740 1 --2- 172.21.15.146:0/2479856460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599eff8de10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d3665e740 1 -- 172.21.15.146:0/2479856460 >> 172.21.15.146:0/2479856460 conn(0x5599eff89d50 msgr2=0x5599eff8a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.976+0000 7f2d30e00640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d100a3e20 0x7f2d100a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.977+0000 7f2d3665e740 1 -- 172.21.15.146:0/2479856460 shutdown_connections 2024-04-24T08:24:39.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.977+0000 7f2d3665e740 1 -- 172.21.15.146:0/2479856460 wait complete. 2024-04-24T08:24:39.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.977+0000 7f2d30e00640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d100a3e20 0x7f2d100a62e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5599eff8eff0 tx=0x7f2d140149a0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.977+0000 7f2d3665e740 1 -- start start 2024-04-24T08:24:39.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.978+0000 7f2d3665e740 1 Processor -- start 2024-04-24T08:24:39.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8da30 0x5599eff95a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8e350 0x5599eff95f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff964c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d3665e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5599eff9b7b0 con 0x5599eff8e350 2024-04-24T08:24:39.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d3665e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5599efeac690 con 0x5599eff8da30 2024-04-24T08:24:39.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d3665e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599eff9b930 con 0x5599eff98a20 2024-04-24T08:24:39.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d3665e740 10 client.0 _ll_register_callbacks cb 0x5599efee8650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.979+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8e350 0x5599eff95f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8e350 0x5599eff95f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:54906/0 (socket says 172.21.15.146:54906) 2024-04-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d2be00640 1 -- 172.21.15.146:0/261251471 learned_addr learned my addr 172.21.15.146:0/261251471 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d31800640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8da30 0x5599eff95a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d31800640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d2be00640 1 -- 172.21.15.146:0/261251471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 msgr2=0x5599eff964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d2be00640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d2be00640 1 -- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8da30 msgr2=0x5599eff95a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d2be00640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8da30 0x5599eff95a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d2be00640 1 -- 172.21.15.146:0/261251471 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599efeab920 con 0x5599eff8e350 2024-04-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.980+0000 7f2d31800640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8da30 0x5599eff95a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.981+0000 7f2d2be00640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8e350 0x5599eff95f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00bdf0 tx=0x7f2d1c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7e3f6274f10ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.981+0000 7f2d1aa00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:24:39.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.981+0000 7f2d1aa00640 1 -- 172.21.15.146:0/261251471 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c012860 con 0x5599eff8e350 2024-04-24T08:24:39.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.981+0000 7f2d1aa00640 1 -- 172.21.15.146:0/261251471 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2d1c012a00 con 0x5599eff8e350 2024-04-24T08:24:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.981+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d24036070 con 0x5599eff8e350 2024-04-24T08:24:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.982+0000 7f2d1aa00640 1 -- 172.21.15.146:0/261251471 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2d1c012c00 con 0x5599eff8e350 2024-04-24T08:24:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.982+0000 7f2d3665e740 10 client.15972 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.982+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5599efccf7d0 con 0x5599eff8e350 2024-04-24T08:24:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.982+0000 7f2d1aa00640 1 -- 172.21.15.146:0/261251471 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d24036070 con 0x5599eff8e350 2024-04-24T08:24:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d1aa00640 1 -- 172.21.15.146:0/261251471 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f2d1c00f3d0 con 0x5599eff8e350 2024-04-24T08:24:39.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d3665e740 10 client.15972 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d3665e740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d3665e740 20 client.15972 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d3665e740 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599effa8dd0 0x5599effab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d3665e740 4 client.15972 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599eff00c10 con 0x5599effa8dd0 2024-04-24T08:24:39.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.983+0000 7f2d31800640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599effa8dd0 0x5599effab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:39.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.984+0000 7f2d31800640 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599effa8dd0 0x5599effab1e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e6e2c79ac9baa22 server_cookie=1448022e35eec366 in_seq=0 out_seq=0 2024-04-24T08:24:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.984+0000 7f2d1aa00640 10 client.15972 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.985+0000 7f2d1aa00640 1 -- 172.21.15.146:0/261251471 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599eff00c10 con 0x5599effa8dd0 2024-04-24T08:24:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.985+0000 7f2d1aa00640 10 client.15972 handle_command_reply: tid=0 2024-04-24T08:24:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.985+0000 7f2d3665e740 1 client.15972 shutdown 2024-04-24T08:24:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8e350 msgr2=0x5599eff95f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8e350 0x5599eff95f80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00bdf0 tx=0x7f2d1c0117b0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 shutdown_connections 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599effa8dd0 0x5599effab1e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8e350 0x5599eff95f80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 --2- 172.21.15.146:0/261251471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8da30 0x5599eff95a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d1aa00640 0 client.15972 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.986+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 >> 172.21.15.146:0/261251471 conn(0x5599eff89d50 msgr2=0x5599eff88fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.987+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 shutdown_connections 2024-04-24T08:24:39.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.987+0000 7f2d3665e740 1 -- 172.21.15.146:0/261251471 wait complete. 2024-04-24T08:24:39.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:39.987+0000 7f2d3665e740 20 client.15972 trim_cache size 0 max 16384 2024-04-24T08:24:40.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 Processor -- start 2024-04-24T08:24:40.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 -- start start 2024-04-24T08:24:40.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff8da30 0x5599eff89750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:40.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599effa46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:40.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff98a20 0x5599eff8f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:40.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1b2710 con 0x5599eff98a20 2024-04-24T08:24:40.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1affe0 con 0x5599eff8e350 2024-04-24T08:24:40.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.009+0000 7f2d3665e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1ade30 con 0x5599eff8da30 2024-04-24T08:24:40.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599effa46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:40.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d31800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff98a20 0x5599eff8f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599effa46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39856/0 (socket says 172.21.15.146:39856) 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 -- 172.21.15.146:0/3025806550 learned_addr learned my addr 172.21.15.146:0/3025806550 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d31800640 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff8da30 0x5599eff89750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 -- 172.21.15.146:0/3025806550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff8da30 msgr2=0x5599eff89750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff8da30 0x5599eff89750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 -- 172.21.15.146:0/3025806550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff98a20 msgr2=0x5599eff8f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff98a20 0x5599eff8f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 -- 172.21.15.146:0/3025806550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599efc053f0 con 0x5599eff8e350 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d31800640 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff8da30 0x5599eff89750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.010+0000 7f2d2be00640 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599effa46a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c006010 tx=0x7f2d1c005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a14fa8aa645871cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.011+0000 7f2d21600640 1 -- 172.21.15.146:0/3025806550 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c033070 con 0x5599eff8e350 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.011+0000 7f2d21600640 1 -- 172.21.15.146:0/3025806550 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c02b040 con 0x5599eff8e350 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.011+0000 7f2d21600640 1 -- 172.21.15.146:0/3025806550 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c02a070 con 0x5599eff8e350 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.011+0000 7f2d3665e740 1 -- 172.21.15.146:0/3025806550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 msgr2=0x5599effa46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.011+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599effa46a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c006010 tx=0x7f2d1c005d80 comp rx=0 tx=0).stop 2024-04-24T08:24:40.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 -- 172.21.15.146:0/3025806550 shutdown_connections 2024-04-24T08:24:40.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff98a20 0x5599eff8f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599effa46a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3025806550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff8da30 0x5599eff89750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 -- 172.21.15.146:0/3025806550 >> 172.21.15.146:0/3025806550 conn(0x5599eff89d50 msgr2=0x5599eff8f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:40.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 -- 172.21.15.146:0/3025806550 shutdown_connections 2024-04-24T08:24:40.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 -- 172.21.15.146:0/3025806550 wait complete. 2024-04-24T08:24:40.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 -- start start 2024-04-24T08:24:40.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.012+0000 7f2d3665e740 1 Processor -- start 2024-04-24T08:24:40.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599effa0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:40.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff92360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:40.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d3665e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff928a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d3665e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1ae4a0 con 0x5599eff8da30 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d3665e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1af910 con 0x5599eff8e350 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d3665e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1aeb70 con 0x5599eff98a20 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d3665e740 10 client.0 _ll_register_callbacks cb 0x7f2d1c00ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff92360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff92360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:39858/0 (socket says 172.21.15.146:39858) 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 -- 172.21.15.146:0/3294492044 learned_addr learned my addr 172.21.15.146:0/3294492044 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:24:40.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d31800640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599effa0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d31800640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff928a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 -- 172.21.15.146:0/3294492044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 msgr2=0x5599eff928a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff928a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 -- 172.21.15.146:0/3294492044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 msgr2=0x5599effa0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599effa0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d2be00640 1 -- 172.21.15.146:0/3294492044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599efbc50f0 con 0x5599eff8e350 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.013+0000 7f2d31800640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff928a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:24:40.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d2be00640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff92360 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00f7f0 tx=0x7f2d1c011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df1fa50e309f970d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d1aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d1aa00640 1 -- 172.21.15.146:0/3294492044 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c02a030 con 0x5599eff8e350 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d1aa00640 1 -- 172.21.15.146:0/3294492044 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2d1c004020 con 0x5599eff8e350 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d1c02b040 con 0x5599eff8e350 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d1aa00640 1 -- 172.21.15.146:0/3294492044 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2d1c045050 con 0x5599eff8e350 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d3665e740 10 client.26120 fetch_fsmap learned FSMap version 25 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.014+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5599efeb8810 con 0x5599eff8e350 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d1aa00640 1 -- 172.21.15.146:0/3294492044 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c02b040 con 0x5599eff8e350 2024-04-24T08:24:40.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d1aa00640 1 -- 172.21.15.146:0/3294492044 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f2d1c040030 con 0x5599eff8e350 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d3665e740 10 client.26120 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d3665e740 10 client.26120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d3665e740 20 client.26120 populate_metadata read hostname 'smithi146' 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599f000f600 0x5599f0011a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d3665e740 4 client.26120 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5599efefaa80 con 0x5599f000f600 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.015+0000 7f2d31800640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599f000f600 0x5599f0011a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:24:40.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d31800640 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599f000f600 0x5599f0011a20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=161d696871714a0b server_cookie=d16d89ca172f99b3 in_seq=0 out_seq=0 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d1aa00640 10 client.26120 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d1aa00640 1 -- 172.21.15.146:0/3294492044 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5599efefaa80 con 0x5599f000f600 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d1aa00640 10 client.26120 handle_command_reply: tid=0 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 client.26120 shutdown 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 msgr2=0x5599eff92360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff92360 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00f7f0 tx=0x7f2d1c011cb0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 shutdown_connections 2024-04-24T08:24:40.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5599f000f600 0x5599f0011a20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599eff98a20 0x5599eff928a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5599eff8e350 0x5599eff92360 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 --2- 172.21.15.146:0/3294492044 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5599eff8da30 0x5599effa0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 >> 172.21.15.146:0/3294492044 conn(0x5599eff89d50 msgr2=0x5599eff96270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:40.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.016+0000 7f2d1aa00640 0 client.26120 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:24:40.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 shutdown_connections 2024-04-24T08:24:40.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d3665e740 1 -- 172.21.15.146:0/3294492044 wait complete. 2024-04-24T08:24:40.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d3665e740 20 client.26120 trim_cache size 0 max 16384 2024-04-24T08:24:40.027 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:24:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:24:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:24:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "de2c1ea7-a1a2-4942-9608-289083aff439": { 2024-04-24T08:24:40.027 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:24:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "de2c1ea7-a1a2-4942-9608-289083aff439", 2024-04-24T08:24:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:24:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:24:40.028 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:24:40.028 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:24:40.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d100a3e20 msgr2=0x7f2d100a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:40.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d100a3e20 0x7f2d100a62e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5599eff8eff0 tx=0x7f2d140149a0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c101000 msgr2=0x7f2d2c1a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:24:40.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c101000 0x7f2d2c1a0670 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2d140097d0 tx=0x7f2d14009c90 comp rx=0 tx=0).stop 2024-04-24T08:24:40.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 shutdown_connections 2024-04-24T08:24:40.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d100a3e20 0x7f2d100a62e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d2c1060d0 0x7f2d2c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d2c19a730 0x7f2d2c19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 --2- 172.21.15.146:0/2124345263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d2c101000 0x7f2d2c1a0670 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:24:40.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.017+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 >> 172.21.15.146:0/2124345263 conn(0x7f2d2c0fc880 msgr2=0x7f2d2c10c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:24:40.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.018+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 shutdown_connections 2024-04-24T08:24:40.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:24:40.018+0000 7f2d32200640 1 -- 172.21.15.146:0/2124345263 wait complete. 2024-04-24T08:24:40.036 INFO:tasks.cephfs.filesystem:scrub status for tag:de2c1ea7-a1a2-4942-9608-289083aff439 - {'path': '/', 'tag': 'de2c1ea7-a1a2-4942-9608-289083aff439', 'options': 'recursive,force'} 2024-04-24T08:24:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: pgmap v107: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 67 KiB/s wr, 8 op/s 2024-04-24T08:24:40.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:24:40.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:24:40.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:24:40.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:24:40.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:40 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: pgmap v107: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 67 KiB/s wr, 8 op/s 2024-04-24T08:24:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:24:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:24:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:24:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:24:40.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:40 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: pgmap v107: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 67 KiB/s wr, 8 op/s 2024-04-24T08:24:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:24:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:24:40.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:24:40.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:24:40.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:40.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:40 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:24:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:42 smithi177 ceph-mon[33046]: pgmap v108: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 67 KiB/s wr, 8 op/s 2024-04-24T08:24:42.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:42 smithi153 ceph-mon[32746]: pgmap v108: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 67 KiB/s wr, 8 op/s 2024-04-24T08:24:42.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:42 smithi146 ceph-mon[27805]: pgmap v108: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 67 KiB/s wr, 8 op/s 2024-04-24T08:24:44.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:44 smithi153 ceph-mon[32746]: pgmap v109: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 68 KiB/s wr, 11 op/s 2024-04-24T08:24:44.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:44 smithi146 ceph-mon[27805]: pgmap v109: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 68 KiB/s wr, 11 op/s 2024-04-24T08:24:45.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:44 smithi177 ceph-mon[33046]: pgmap v109: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 68 KiB/s wr, 11 op/s 2024-04-24T08:24:46.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:46 smithi153 ceph-mon[32746]: pgmap v110: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 61 KiB/s wr, 8 op/s 2024-04-24T08:24:46.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:46 smithi146 ceph-mon[27805]: pgmap v110: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 61 KiB/s wr, 8 op/s 2024-04-24T08:24:47.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:46 smithi177 ceph-mon[33046]: pgmap v110: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 61 KiB/s wr, 8 op/s 2024-04-24T08:24:47.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:47 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:24:47.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:47 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:24:47.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:47 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:24:47.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:47 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:24:48.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:47 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:24:48.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:47 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:24:48.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:48 smithi153 ceph-mon[32746]: pgmap v111: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 62 KiB/s wr, 10 op/s 2024-04-24T08:24:48.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:48 smithi146 ceph-mon[27805]: pgmap v111: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 62 KiB/s wr, 10 op/s 2024-04-24T08:24:49.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:48 smithi177 ceph-mon[33046]: pgmap v111: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 62 KiB/s wr, 10 op/s 2024-04-24T08:24:50.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:50 smithi153 ceph-mon[32746]: pgmap v112: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-24T08:24:50.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:50 smithi146 ceph-mon[27805]: pgmap v112: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-24T08:24:51.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:50 smithi177 ceph-mon[33046]: pgmap v112: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-24T08:24:52.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:52 smithi153 ceph-mon[32746]: pgmap v113: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-24T08:24:52.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:52 smithi146 ceph-mon[27805]: pgmap v113: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-24T08:24:53.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:52 smithi177 ceph-mon[33046]: pgmap v113: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.1 KiB/s wr, 5 op/s 2024-04-24T08:24:54.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:54 smithi153 ceph-mon[32746]: pgmap v114: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-04-24T08:24:54.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:54 smithi146 ceph-mon[27805]: pgmap v114: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-04-24T08:24:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:54 smithi177 ceph-mon[33046]: pgmap v114: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-04-24T08:24:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:56 smithi153 ceph-mon[32746]: pgmap v115: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-24T08:24:56.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:56 smithi146 ceph-mon[27805]: pgmap v115: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-24T08:24:57.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:56 smithi177 ceph-mon[33046]: pgmap v115: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-24T08:24:58.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:24:58 smithi153 ceph-mon[32746]: pgmap v116: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-24T08:24:58.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:24:58 smithi146 ceph-mon[27805]: pgmap v116: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-24T08:24:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:24:58 smithi177 ceph-mon[33046]: pgmap v116: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-24T08:25:00.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:00 smithi153 ceph-mon[32746]: pgmap v117: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:00.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:00 smithi146 ceph-mon[27805]: pgmap v117: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:00 smithi177 ceph-mon[33046]: pgmap v117: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:02.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:02 smithi153 ceph-mon[32746]: pgmap v118: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:02.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:02 smithi146 ceph-mon[27805]: pgmap v118: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:03.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:02 smithi177 ceph-mon[33046]: pgmap v118: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:04 smithi153 ceph-mon[32746]: pgmap v119: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:04.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:04 smithi146 ceph-mon[27805]: pgmap v119: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:05.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:04 smithi177 ceph-mon[33046]: pgmap v119: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:06 smithi153 ceph-mon[32746]: pgmap v120: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:25:06.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:06 smithi146 ceph-mon[27805]: pgmap v120: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:25:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:06 smithi177 ceph-mon[33046]: pgmap v120: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:25:08.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:08 smithi153 ceph-mon[32746]: pgmap v121: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:08.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:08 smithi146 ceph-mon[27805]: pgmap v121: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:09.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:08 smithi177 ceph-mon[33046]: pgmap v121: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:10.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:25:10.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:25:10.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 -- 172.21.15.146:0/2407659448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c108f40 msgr2=0x7fac8c1153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 --2- 172.21.15.146:0/2407659448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c108f40 0x7fac8c1153b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fac84009cd0 tx=0x7fac8402a8f0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 -- 172.21.15.146:0/2407659448 shutdown_connections 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 --2- 172.21.15.146:0/2407659448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c108f40 0x7fac8c1153b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 --2- 172.21.15.146:0/2407659448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 0x7fac8c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 --2- 172.21.15.146:0/2407659448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c10d500 0x7fac8c10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 -- 172.21.15.146:0/2407659448 >> 172.21.15.146:0/2407659448 conn(0x7fac8c077e20 msgr2=0x7fac8c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 -- 172.21.15.146:0/2407659448 shutdown_connections 2024-04-24T08:25:10.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.255+0000 7fac90e00640 1 -- 172.21.15.146:0/2407659448 wait complete. 2024-04-24T08:25:10.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.256+0000 7fac90e00640 1 Processor -- start 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.256+0000 7fac90e00640 1 -- start start 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac90e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 0x7fac8c1a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac90e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c108f40 0x7fac8c1a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac90e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c10d500 0x7fac8c19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac90e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c1188d0 con 0x7fac8c10d500 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac90e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c118750 con 0x7fac8c108580 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac90e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c118a50 con 0x7fac8c108f40 2024-04-24T08:25:10.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c10d500 0x7fac8c19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c108f40 0x7fac8c1a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac8b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 0x7fac8c1a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c108f40 0x7fac8c1a4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:55876/0 (socket says 172.21.15.146:55876) 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c10d500 0x7fac8c19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36318/0 (socket says 172.21.15.146:36318) 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac8b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 0x7fac8c1a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37202/0 (socket says 172.21.15.146:37202) 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.257+0000 7fac83e00640 1 -- 172.21.15.146:0/1167511422 learned_addr learned my addr 172.21.15.146:0/1167511422 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac83e00640 1 -- 172.21.15.146:0/1167511422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 msgr2=0x7fac8c1a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac83e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 0x7fac8c1a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac83e00640 1 -- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c10d500 msgr2=0x7fac8c19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac83e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c10d500 0x7fac8c19eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac83e00640 1 -- 172.21.15.146:0/1167511422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8c19f2d0 con 0x7fac8c108f40 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac8b400640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 0x7fac8c1a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac83e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c108f40 0x7fac8c1a4e90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fac7400ea60 tx=0x7fac7400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac88c00640 1 -- 172.21.15.146:0/1167511422 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7400cde0 con 0x7fac8c108f40 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8c19f5c0 con 0x7fac8c108f40 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac88c00640 1 -- 172.21.15.146:0/1167511422 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac740035f0 con 0x7fac8c108f40 2024-04-24T08:25:10.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.258+0000 7fac88c00640 1 -- 172.21.15.146:0/1167511422 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac74010670 con 0x7fac8c108f40 2024-04-24T08:25:10.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.259+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac8c1ab810 con 0x7fac8c108f40 2024-04-24T08:25:10.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 Processor -- start 2024-04-24T08:25:10.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 -- start start 2024-04-24T08:25:10.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac88c00640 1 -- 172.21.15.146:0/1167511422 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fac740037a0 con 0x7fac8c108f40 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42e4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42efa20 0x5631c42f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5631c424a1f0 con 0x5631c42e4a30 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5631c41f0350 con 0x5631c42e5350 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac95208740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5631c42461b0 con 0x5631c42efa20 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42e4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42e4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36332/0 (socket says 172.21.15.146:36332) 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.262+0000 7fac8be00640 1 -- 172.21.15.146:0/380794501 learned_addr learned my addr 172.21.15.146:0/380794501 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8b400640 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac83e00640 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42efa20 0x5631c42f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8be00640 1 -- 172.21.15.146:0/380794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42efa20 msgr2=0x5631c42f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8be00640 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42efa20 0x5631c42f1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8be00640 1 -- 172.21.15.146:0/380794501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 msgr2=0x5631c42ef4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8be00640 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42ef4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8be00640 1 -- 172.21.15.146:0/380794501 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631c4244f50 con 0x5631c42e4a30 2024-04-24T08:25:10.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8b400640 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.263+0000 7fac8be00640 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42e4e10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fac84009380 tx=0x7fac840093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec59e7fd53b78c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac80c00640 1 -- 172.21.15.146:0/380794501 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac84030690 con 0x5631c42e4a30 2024-04-24T08:25:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac80c00640 1 -- 172.21.15.146:0/380794501 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac84036070 con 0x5631c42e4a30 2024-04-24T08:25:10.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac80c00640 1 -- 172.21.15.146:0/380794501 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac84004420 con 0x5631c42e4a30 2024-04-24T08:25:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac88c00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fac700a3d20 0x7fac700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac88c00640 1 -- 172.21.15.146:0/1167511422 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fac740d0b10 con 0x7fac8c108f40 2024-04-24T08:25:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac8b400640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fac700a3d20 0x7fac700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac95208740 1 -- 172.21.15.146:0/380794501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 msgr2=0x5631c42e4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac95208740 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42e4e10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fac84009380 tx=0x7fac840093b0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac95208740 1 -- 172.21.15.146:0/380794501 shutdown_connections 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac95208740 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42efa20 0x5631c42f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac95208740 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac95208740 1 --2- 172.21.15.146:0/380794501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42e4e10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.264+0000 7fac95208740 1 -- 172.21.15.146:0/380794501 >> 172.21.15.146:0/380794501 conn(0x5631c42e0d50 msgr2=0x5631c42e1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.265+0000 7fac95208740 1 -- 172.21.15.146:0/380794501 shutdown_connections 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.265+0000 7fac95208740 1 -- 172.21.15.146:0/380794501 wait complete. 2024-04-24T08:25:10.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.265+0000 7fac8b400640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fac700a3d20 0x7fac700a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009fd0 tx=0x7fac7c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.265+0000 7fac95208740 1 -- start start 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.265+0000 7fac95208740 1 Processor -- start 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac95208740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e4a30 0x5631c42ec9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac95208740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42ecee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac95208740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c42ed420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac95208740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5631c42f27b0 con 0x5631c42efa20 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac95208740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5631c4203690 con 0x5631c42e4a30 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac95208740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5631c42f2930 con 0x5631c42e5350 2024-04-24T08:25:10.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac95208740 10 client.0 _ll_register_callbacks cb 0x5631c423f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac8b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e4a30 0x5631c42ec9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42ecee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac8b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e4a30 0x5631c42ec9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37228/0 (socket says 172.21.15.146:37228) 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac8b400640 1 -- 172.21.15.146:0/229761978 learned_addr learned my addr 172.21.15.146:0/229761978 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac8be00640 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c42ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac83e00640 1 -- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e4a30 msgr2=0x5631c42ec9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac83e00640 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e4a30 0x5631c42ec9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac83e00640 1 -- 172.21.15.146:0/229761978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 msgr2=0x5631c42ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac83e00640 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c42ed420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac83e00640 1 -- 172.21.15.146:0/229761978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631c4202920 con 0x5631c42e5350 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.266+0000 7fac8b400640 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e4a30 0x5631c42ec9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac83e00640 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42ecee0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fac740ce210 tx=0x7fac7409c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6acdbd8227ad330 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac79600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac79600640 1 -- 172.21.15.146:0/229761978 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7409d160 con 0x5631c42e5350 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac79600640 1 -- 172.21.15.146:0/229761978 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fac740990d0 con 0x5631c42e5350 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac84036070 con 0x5631c42e5350 2024-04-24T08:25:10.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac79600640 1 -- 172.21.15.146:0/229761978 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fac7409ca60 con 0x5631c42e5350 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac95208740 10 client.26113 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5631c40267d0 con 0x5631c42e5350 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac79600640 1 -- 172.21.15.146:0/229761978 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac84036070 con 0x5631c42e5350 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.267+0000 7fac79600640 1 -- 172.21.15.146:0/229761978 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fac7409beb0 con 0x5631c42e5350 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.268+0000 7fac95208740 10 client.26113 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.268+0000 7fac95208740 10 client.26113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.268+0000 7fac95208740 20 client.26113 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:10.276 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.268+0000 7fac95208740 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c42ffe00 0x5631c4302230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.268+0000 7fac95208740 4 client.26113 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.268+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631c4257c10 con 0x5631c42ffe00 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.268+0000 7fac8b400640 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c42ffe00 0x5631c4302230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.269+0000 7fac8b400640 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c42ffe00 0x5631c4302230 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e429cceed7f7049d server_cookie=cbc2531556b36636 in_seq=0 out_seq=0 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.269+0000 7fac79600640 10 client.26113 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.269+0000 7fac79600640 1 -- 172.21.15.146:0/229761978 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631c4257c10 con 0x5631c42ffe00 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.269+0000 7fac79600640 10 client.26113 handle_command_reply: tid=0 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.269+0000 7fac95208740 1 client.26113 shutdown 2024-04-24T08:25:10.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 msgr2=0x5631c42ecee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42ecee0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fac740ce210 tx=0x7fac7409c830 comp rx=0 tx=0).stop 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 shutdown_connections 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c42ffe00 0x5631c4302230 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42ecee0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c42ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 --2- 172.21.15.146:0/229761978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e4a30 0x5631c42ec9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 >> 172.21.15.146:0/229761978 conn(0x5631c42e0d50 msgr2=0x5631c42dffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac79600640 0 client.26113 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 shutdown_connections 2024-04-24T08:25:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 1 -- 172.21.15.146:0/229761978 wait complete. 2024-04-24T08:25:10.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.270+0000 7fac95208740 20 client.26113 trim_cache size 0 max 16384 2024-04-24T08:25:10.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.289+0000 7fac95208740 1 Processor -- start 2024-04-24T08:25:10.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.289+0000 7fac95208740 1 -- start start 2024-04-24T08:25:10.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.289+0000 7fac95208740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e4a30 0x5631c42fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.289+0000 7fac95208740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42e71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.289+0000 7fac95208740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c4307b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.289+0000 7fac95208740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c1b6930 con 0x5631c42efa20 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac95208740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c1b4200 con 0x5631c42e5350 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac95208740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c1b2050 con 0x5631c42e4a30 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c4307b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e4a30 0x5631c42fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e4a30 0x5631c42fb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:55912/0 (socket says 172.21.15.146:55912) 2024-04-24T08:25:10.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 -- 172.21.15.146:0/3755193769 learned_addr learned my addr 172.21.15.146:0/3755193769 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c4307b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36350/0 (socket says 172.21.15.146:36350) 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac83e00640 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42e71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 -- 172.21.15.146:0/3755193769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 msgr2=0x5631c42e71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42e71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 -- 172.21.15.146:0/3755193769 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 msgr2=0x5631c4307b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c4307b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.290+0000 7fac8b400640 1 -- 172.21.15.146:0/3755193769 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631c3f5c3f0 con 0x5631c42e4a30 2024-04-24T08:25:10.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.291+0000 7fac8b400640 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e4a30 0x5631c42fb6a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x5631c4302a40 tx=0x7fac7c01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1559af29108ef467 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.291+0000 7fac7be00640 1 -- 172.21.15.146:0/3755193769 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c00fa10 con 0x5631c42e4a30 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.291+0000 7fac7be00640 1 -- 172.21.15.146:0/3755193769 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac7c030070 con 0x5631c42e4a30 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.291+0000 7fac7be00640 1 -- 172.21.15.146:0/3755193769 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c01f8c0 con 0x5631c42e4a30 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 -- 172.21.15.146:0/3755193769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e4a30 msgr2=0x5631c42fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e4a30 0x5631c42fb6a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x5631c4302a40 tx=0x7fac7c01b000 comp rx=0 tx=0).stop 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 -- 172.21.15.146:0/3755193769 shutdown_connections 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42efa20 0x5631c4307b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42e5350 0x5631c42e71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 --2- 172.21.15.146:0/3755193769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e4a30 0x5631c42fb6a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 -- 172.21.15.146:0/3755193769 >> 172.21.15.146:0/3755193769 conn(0x5631c42e0d50 msgr2=0x5631c42ed1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 -- 172.21.15.146:0/3755193769 shutdown_connections 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 -- 172.21.15.146:0/3755193769 wait complete. 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.292+0000 7fac95208740 1 -- start start 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 1 Processor -- start 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42efa20 0x5631c42f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c1b26c0 con 0x5631c42e4a30 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c1b3b30 con 0x5631c42efa20 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac8c1b2d90 con 0x5631c42e5350 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac95208740 10 client.0 _ll_register_callbacks cb 0x7fac74003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac8b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac8b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42f7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36358/0 (socket says 172.21.15.146:36358) 2024-04-24T08:25:10.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac8b400640 1 -- 172.21.15.146:0/994736355 learned_addr learned my addr 172.21.15.146:0/994736355 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac8be00640 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42efa20 0x5631c42f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.293+0000 7fac83e00640 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac8b400640 1 -- 172.21.15.146:0/994736355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 msgr2=0x5631c42f80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac8b400640 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42f80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac8b400640 1 -- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42efa20 msgr2=0x5631c42f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac8b400640 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42efa20 0x5631c42f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac8b400640 1 -- 172.21.15.146:0/994736355 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631c3f1c0f0 con 0x5631c42e4a30 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac8b400640 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42f7bb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fac7c003780 tx=0x7fac7c01f400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36dff26df5cb1962 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac79600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac79600640 1 -- 172.21.15.146:0/994736355 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c03a070 con 0x5631c42e4a30 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac7c030070 con 0x5631c42e4a30 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.294+0000 7fac79600640 1 -- 172.21.15.146:0/994736355 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fac7c00fb50 con 0x5631c42e4a30 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac79600640 1 -- 172.21.15.146:0/994736355 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fac7c046050 con 0x5631c42e4a30 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 10 client.15996 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5631c420f810 con 0x5631c42e4a30 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac79600640 1 -- 172.21.15.146:0/994736355 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac7c030070 con 0x5631c42e4a30 2024-04-24T08:25:10.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac79600640 1 -- 172.21.15.146:0/994736355 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fac7c01faf0 con 0x5631c42e4a30 2024-04-24T08:25:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 10 client.15996 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 20 client.15996 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c4366f70 0x5631c4369360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 4 client.15996 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:25:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.295+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5631c4251a80 con 0x5631c4366f70 2024-04-24T08:25:10.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac83e00640 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c4366f70 0x5631c4369360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac83e00640 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c4366f70 0x5631c4369360 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=948e1a83a8a7586f server_cookie=7975e0c16e17c144 in_seq=0 out_seq=0 2024-04-24T08:25:10.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac79600640 10 client.15996 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac79600640 1 -- 172.21.15.146:0/994736355 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5631c4251a80 con 0x5631c4366f70 2024-04-24T08:25:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac79600640 10 client.15996 handle_command_reply: tid=0 2024-04-24T08:25:10.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac95208740 1 client.15996 shutdown 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 msgr2=0x5631c42f7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.296+0000 7fac95208740 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42f7bb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fac7c003780 tx=0x7fac7c01f400 comp rx=0 tx=0).stop 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 shutdown_connections 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5631c4366f70 0x5631c4369360 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5631c42e5350 0x5631c42f80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5631c42efa20 0x5631c42f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac79600640 0 client.15996 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 --2- 172.21.15.146:0/994736355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5631c42e4a30 0x5631c42f7bb0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 >> 172.21.15.146:0/994736355 conn(0x5631c42e0d50 msgr2=0x5631c42ee1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 shutdown_connections 2024-04-24T08:25:10.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 1 -- 172.21.15.146:0/994736355 wait complete. 2024-04-24T08:25:10.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.297+0000 7fac95208740 20 client.15996 trim_cache size 0 max 16384 2024-04-24T08:25:10.307 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:25:10.307 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:25:10.308 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:25:10.308 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:25:10.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fac700a3d20 msgr2=0x7fac700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fac700a3d20 0x7fac700a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009fd0 tx=0x7fac7c005f90 comp rx=0 tx=0).stop 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c108f40 msgr2=0x7fac8c1a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c108f40 0x7fac8c1a4e90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fac7400ea60 tx=0x7fac7400ef20 comp rx=0 tx=0).stop 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 shutdown_connections 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fac700a3d20 0x7fac700a61e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac8c108f40 0x7fac8c1a4e90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac8c10d500 0x7fac8c19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 --2- 172.21.15.146:0/1167511422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac8c108580 0x7fac8c1a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 >> 172.21.15.146:0/1167511422 conn(0x7fac8c077e20 msgr2=0x7fac8c078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 shutdown_connections 2024-04-24T08:25:10.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.298+0000 7fac90e00640 1 -- 172.21.15.146:0/1167511422 wait complete. 2024-04-24T08:25:10.316 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:25:10.317 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:25:10.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 -- 172.21.15.146:0/2398685386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f14109340 msgr2=0x7f5f14110f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 --2- 172.21.15.146:0/2398685386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f14109340 0x7f5f14110f10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009ca0 tx=0x7f5f0c02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 -- 172.21.15.146:0/2398685386 shutdown_connections 2024-04-24T08:25:10.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 --2- 172.21.15.146:0/2398685386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f14104c20 0x7f5f14108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 --2- 172.21.15.146:0/2398685386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f14109340 0x7f5f14110f10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 --2- 172.21.15.146:0/2398685386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14104270 0x7f5f14104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 -- 172.21.15.146:0/2398685386 >> 172.21.15.146:0/2398685386 conn(0x7f5f140ffad0 msgr2=0x7f5f14101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 -- 172.21.15.146:0/2398685386 shutdown_connections 2024-04-24T08:25:10.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.522+0000 7f5f1b000640 1 -- 172.21.15.146:0/2398685386 wait complete. 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 Processor -- start 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 -- start start 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f14104270 0x7f5f141a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14109340 0x7f5f141a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f1419a760 0x7f5f1419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5f14114420 con 0x7f5f14109340 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5f141142a0 con 0x7f5f1419a760 2024-04-24T08:25:10.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.523+0000 7f5f1b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5f141145a0 con 0x7f5f14104270 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14109340 0x7f5f141a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14109340 0x7f5f141a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36376/0 (socket says 172.21.15.146:36376) 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 -- 172.21.15.146:0/131901556 learned_addr learned my addr 172.21.15.146:0/131901556 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19c00640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f14104270 0x7f5f141a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f1a600640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f1419a760 0x7f5f1419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 -- 172.21.15.146:0/131901556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f14104270 msgr2=0x7f5f141a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f14104270 0x7f5f141a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 -- 172.21.15.146:0/131901556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f1419a760 msgr2=0x7f5f1419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f1419a760 0x7f5f1419abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 -- 172.21.15.146:0/131901556 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f1419b360 con 0x7f5f14109340 2024-04-24T08:25:10.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f1a600640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f1419a760 0x7f5f1419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:10.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.524+0000 7f5f19200640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14109340 0x7f5f141a0b90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000e9a0 tx=0x7f5f0000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.525+0000 7f5f12000640 1 -- 172.21.15.146:0/131901556 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0000cd80 con 0x7f5f14109340 2024-04-24T08:25:10.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.525+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f141a73f0 con 0x7f5f14109340 2024-04-24T08:25:10.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.525+0000 7f5f12000640 1 -- 172.21.15.146:0/131901556 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f000035f0 con 0x7f5f14109340 2024-04-24T08:25:10.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.525+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f141a7850 con 0x7f5f14109340 2024-04-24T08:25:10.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.525+0000 7f5f12000640 1 -- 172.21.15.146:0/131901556 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00018630 con 0x7f5f14109340 2024-04-24T08:25:10.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.526+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f14105740 con 0x7f5f14109340 2024-04-24T08:25:10.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.528+0000 7f5f12000640 1 -- 172.21.15.146:0/131901556 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f5f000037a0 con 0x7f5f14109340 2024-04-24T08:25:10.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.530+0000 7f5f12000640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5efc0a3d20 0x7f5efc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:10.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.530+0000 7f5f12000640 1 -- 172.21.15.146:0/131901556 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5f000d0a40 con 0x7f5f14109340 2024-04-24T08:25:10.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.530+0000 7f5f19c00640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5efc0a3d20 0x7f5efc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:10.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.531+0000 7f5f19c00640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5efc0a3d20 0x7f5efc0a61e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5f08009bd0 tx=0x7f5f08005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:10.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.535+0000 7f5f12000640 1 -- 172.21.15.146:0/131901556 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f0009d090 con 0x7f5f14109340 2024-04-24T08:25:10.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.780+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f1410c570 con 0x7f5f14109340 2024-04-24T08:25:10.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.782+0000 7f5f12000640 1 -- 172.21.15.146:0/131901556 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11892 (secure 0 0 0) 0x7f5f0009c990 con 0x7f5f14109340 2024-04-24T08:25:10.783 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:25:10.783 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:23:07.211437+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24568,24569,24553]},"id":1}]} 2024-04-24T08:25:10.783 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 25 2024-04-24T08:25:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.784+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5efc0a3d20 msgr2=0x7f5efc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.784+0000 7f5f1b000640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5efc0a3d20 0x7f5efc0a61e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5f08009bd0 tx=0x7f5f08005f90 comp rx=0 tx=0).stop 2024-04-24T08:25:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14109340 msgr2=0x7f5f141a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:10.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14109340 0x7f5f141a0b90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000e9a0 tx=0x7f5f0000ee60 comp rx=0 tx=0).stop 2024-04-24T08:25:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 shutdown_connections 2024-04-24T08:25:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5efc0a3d20 0x7f5efc0a61e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f1419a760 0x7f5f1419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f14109340 0x7f5f141a0b90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 --2- 172.21.15.146:0/131901556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f14104270 0x7f5f141a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 >> 172.21.15.146:0/131901556 conn(0x7f5f140ffad0 msgr2=0x7f5f141006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 shutdown_connections 2024-04-24T08:25:10.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:10.785+0000 7f5f1b000640 1 -- 172.21.15.146:0/131901556 wait complete. 2024-04-24T08:25:10.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:25:10.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:10 smithi153 ceph-mon[32746]: pgmap v122: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:25:10.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:10 smithi146 ceph-mon[27805]: pgmap v122: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.013+0000 7f1483e00640 1 -- 172.21.15.146:0/2262068308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14841019b0 msgr2=0x7f14841059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.013+0000 7f1483e00640 1 --2- 172.21.15.146:0/2262068308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14841019b0 0x7f14841059f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1470009900 tx=0x7f147002a410 comp rx=0 tx=0).stop 2024-04-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.014+0000 7f1483e00640 1 -- 172.21.15.146:0/2262068308 shutdown_connections 2024-04-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.014+0000 7f1483e00640 1 --2- 172.21.15.146:0/2262068308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14841060d0 0x7f148410dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.014+0000 7f1483e00640 1 --2- 172.21.15.146:0/2262068308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14841019b0 0x7f14841059f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.014+0000 7f1483e00640 1 --2- 172.21.15.146:0/2262068308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1484101000 0x7f14841013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.014+0000 7f1483e00640 1 -- 172.21.15.146:0/2262068308 >> 172.21.15.146:0/2262068308 conn(0x7f14840fc860 msgr2=0x7f14840fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.014+0000 7f1483e00640 1 -- 172.21.15.146:0/2262068308 shutdown_connections 2024-04-24T08:25:11.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.014+0000 7f1483e00640 1 -- 172.21.15.146:0/2262068308 wait complete. 2024-04-24T08:25:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 Processor -- start 2024-04-24T08:25:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 -- start start 2024-04-24T08:25:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1484101000 0x7f14841a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841060d0 0x7f14841a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f148419a7c0 0x7f148419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f14841122d0 con 0x7f1484101000 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1484112150 con 0x7f14841060d0 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1483e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1484112450 con 0x7f148419a7c0 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1482000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841060d0 0x7f14841a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1482a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1484101000 0x7f14841a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.015+0000 7f1482000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841060d0 0x7f14841a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59458/0 (socket says 172.21.15.146:59458) 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482000640 1 -- 172.21.15.146:0/2349726150 learned_addr learned my addr 172.21.15.146:0/2349726150 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1483400640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f148419a7c0 0x7f148419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482000640 1 -- 172.21.15.146:0/2349726150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f148419a7c0 msgr2=0x7f148419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482000640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f148419a7c0 0x7f148419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482000640 1 -- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1484101000 msgr2=0x7f14841a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482000640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1484101000 0x7f14841a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482000640 1 -- 172.21.15.146:0/2349726150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148419b350 con 0x7f14841060d0 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482a00640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1484101000 0x7f14841a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.016+0000 7f1482000640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841060d0 0x7f14841a0b90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f147002a930 tx=0x7f1470003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.017+0000 7f147b400640 1 -- 172.21.15.146:0/2349726150 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147003f070 con 0x7f14841060d0 2024-04-24T08:25:11.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.017+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f148419b5e0 con 0x7f14841060d0 2024-04-24T08:25:11.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.017+0000 7f147b400640 1 -- 172.21.15.146:0/2349726150 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1470031c90 con 0x7f14841060d0 2024-04-24T08:25:11.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.017+0000 7f147b400640 1 -- 172.21.15.146:0/2349726150 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470035030 con 0x7f14841060d0 2024-04-24T08:25:11.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.017+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14841a7790 con 0x7f14841060d0 2024-04-24T08:25:11.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 Processor -- start 2024-04-24T08:25:11.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 -- start start 2024-04-24T08:25:11.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f147b400640 1 -- 172.21.15.146:0/2349726150 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f1470042070 con 0x7f14841060d0 2024-04-24T08:25:11.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff5460 0x56499efff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499f001f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56499ef5a1d0 con 0x56499eff4b40 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56499ef00450 con 0x56499eff5460 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.020+0000 7f148c5a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56499ef56190 con 0x56499efffb30 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1483400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1483400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55602/0 (socket says 172.21.15.146:55602) 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1483400640 1 -- 172.21.15.146:0/2858758064 learned_addr learned my addr 172.21.15.146:0/2858758064 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482a00640 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff5460 0x56499efff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482000640 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499f001f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482a00640 1 -- 172.21.15.146:0/2858758064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 msgr2=0x56499f001f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482a00640 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499f001f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482a00640 1 -- 172.21.15.146:0/2858758064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 msgr2=0x56499eff4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482a00640 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482a00640 1 -- 172.21.15.146:0/2858758064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56499efbd310 con 0x56499eff5460 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1483400640 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.021+0000 7f1482a00640 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff5460 0x56499efff5f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1474009fd0 tx=0x7f14740118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84cac0ee2357e28d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.022+0000 7f146fe00640 1 -- 172.21.15.146:0/2858758064 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1474004020 con 0x56499eff5460 2024-04-24T08:25:11.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.022+0000 7f146fe00640 1 -- 172.21.15.146:0/2858758064 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1474005500 con 0x56499eff5460 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.022+0000 7f146fe00640 1 -- 172.21.15.146:0/2858758064 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1474004ce0 con 0x56499eff5460 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.022+0000 7f147b400640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14680a3d20 0x7f14680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 -- 172.21.15.146:0/2858758064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff5460 msgr2=0x56499efff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f1483400640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14680a3d20 0x7f14680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff5460 0x56499efff5f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1474009fd0 tx=0x7f14740118d0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f147b400640 1 -- 172.21.15.146:0/2349726150 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f14700f0e00 con 0x7f14841060d0 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 -- 172.21.15.146:0/2858758064 shutdown_connections 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499f001f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff5460 0x56499efff5f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2858758064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 -- 172.21.15.146:0/2858758064 >> 172.21.15.146:0/2858758064 conn(0x56499eff0e60 msgr2=0x56499eff1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 -- 172.21.15.146:0/2858758064 shutdown_connections 2024-04-24T08:25:11.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f148c5a0740 1 -- 172.21.15.146:0/2858758064 wait complete. 2024-04-24T08:25:11.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.023+0000 7f1483400640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14680a3d20 0x7f14680a61e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f147c009fd0 tx=0x7f147c009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 -- start start 2024-04-24T08:25:11.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 Processor -- start 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff4b40 0x56499efff4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff5460 0x56499effc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499effcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56499f0029c0 con 0x56499eff5460 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56499ef13810 con 0x56499eff4b40 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56499f002b40 con 0x56499efffb30 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.024+0000 7f148c5a0740 10 client.0 _ll_register_callbacks cb 0x56499eee1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:11.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff5460 0x56499effc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1483400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499effcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff5460 0x56499effc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55606/0 (socket says 172.21.15.146:55606) 2024-04-24T08:25:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 -- 172.21.15.146:0/1106295059 learned_addr learned my addr 172.21.15.146:0/1106295059 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482a00640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff4b40 0x56499efff4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 -- 172.21.15.146:0/1106295059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 msgr2=0x56499effcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499effcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 -- 172.21.15.146:0/1106295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff4b40 msgr2=0x56499efff4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff4b40 0x56499efff4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 -- 172.21.15.146:0/1106295059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499ef243f0 con 0x56499eff5460 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482a00640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff4b40 0x56499efff4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1483400640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499effcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f1482000640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff5460 0x56499effc7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f14700bcdf0 tx=0x7f147002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1290917958659cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.025+0000 7f146cc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f146cc00640 1 -- 172.21.15.146:0/1106295059 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470049070 con 0x56499eff5460 2024-04-24T08:25:11.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f146cc00640 1 -- 172.21.15.146:0/1106295059 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f14700b94d0 con 0x56499eff5460 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1474005500 con 0x56499eff5460 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f146cc00640 1 -- 172.21.15.146:0/1106295059 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f14700b9100 con 0x56499eff5460 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 10 client.16014 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56499ec6c3f0 con 0x56499eff5460 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f146cc00640 1 -- 172.21.15.146:0/1106295059 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14700c0080 con 0x56499eff5460 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f146cc00640 1 -- 172.21.15.146:0/1106295059 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f14700bd150 con 0x56499eff5460 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 10 client.16014 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:11.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 10 client.16014 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 20 client.16014 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f00fb90 0x56499f011fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 4 client.16014 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.026+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56499ef67bf0 con 0x56499f00fb90 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.027+0000 7f1482a00640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f00fb90 0x56499f011fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.027+0000 7f1482a00640 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f00fb90 0x56499f011fa0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b554763d22ee2bd2 server_cookie=5d5749ab25ef746b in_seq=0 out_seq=0 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.027+0000 7f146cc00640 10 client.16014 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:11.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.028+0000 7f146cc00640 1 -- 172.21.15.146:0/1106295059 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56499ef67bf0 con 0x56499f00fb90 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.028+0000 7f146cc00640 10 client.16014 handle_command_reply: tid=0 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.028+0000 7f148c5a0740 1 client.16014 shutdown 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.028+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff5460 msgr2=0x56499effc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.028+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff5460 0x56499effc7d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f14700bcdf0 tx=0x7f147002a410 comp rx=0 tx=0).stop 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 shutdown_connections 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f00fb90 0x56499f011fa0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499efffb30 0x56499effcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff5460 0x56499effc7d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1106295059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499eff4b40 0x56499efff4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f146cc00640 0 client.16014 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 >> 172.21.15.146:0/1106295059 conn(0x56499eff0e60 msgr2=0x56499eff6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 shutdown_connections 2024-04-24T08:25:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 1 -- 172.21.15.146:0/1106295059 wait complete. 2024-04-24T08:25:11.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.029+0000 7f148c5a0740 20 client.16014 trim_cache size 0 max 16384 2024-04-24T08:25:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.043+0000 7f148c5a0740 1 Processor -- start 2024-04-24T08:25:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.043+0000 7f148c5a0740 1 -- start start 2024-04-24T08:25:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499efff010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499f017870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f148c5a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f14841b26a0 con 0x56499eff4b40 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f148c5a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14841aff70 con 0x56499efffb30 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f148c5a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14841addc0 con 0x56499eff5460 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55618/0 (socket says 172.21.15.146:55618) 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 -- 172.21.15.146:0/2500505185 learned_addr learned my addr 172.21.15.146:0/2500505185 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1483400640 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499f017870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482000640 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499efff010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 -- 172.21.15.146:0/2500505185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 msgr2=0x56499efff010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499efff010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 -- 172.21.15.146:0/2500505185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 msgr2=0x56499f017870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499f017870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.044+0000 7f1482a00640 1 -- 172.21.15.146:0/2500505185 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56499ed367d0 con 0x56499eff4b40 2024-04-24T08:25:11.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.045+0000 7f1482a00640 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff06d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1474004950 tx=0x7f1474004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6175cc677475e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.045+0000 7f146f400640 1 -- 172.21.15.146:0/2500505185 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147401d040 con 0x56499eff4b40 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.045+0000 7f146f400640 1 -- 172.21.15.146:0/2500505185 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f147400f5a0 con 0x56499eff4b40 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.045+0000 7f146f400640 1 -- 172.21.15.146:0/2500505185 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14740266a0 con 0x56499eff4b40 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.045+0000 7f148c5a0740 1 -- 172.21.15.146:0/2500505185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 msgr2=0x56499eff06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.045+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff06d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1474004950 tx=0x7f1474004980 comp rx=0 tx=0).stop 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 -- 172.21.15.146:0/2500505185 shutdown_connections 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499efff010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499f017870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 --2- 172.21.15.146:0/2500505185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499eff06d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 -- 172.21.15.146:0/2500505185 >> 172.21.15.146:0/2500505185 conn(0x56499eff0e60 msgr2=0x56499effb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 -- 172.21.15.146:0/2500505185 shutdown_connections 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 -- 172.21.15.146:0/2500505185 wait complete. 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 -- start start 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.046+0000 7f148c5a0740 1 Processor -- start 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499f007d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499f0082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f148c5a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499eff9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f148c5a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f14841ae430 con 0x56499eff4b40 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f148c5a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14841af8a0 con 0x56499efffb30 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f148c5a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14841aeb00 con 0x56499eff5460 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f148c5a0740 10 client.0 _ll_register_callbacks cb 0x7f1470005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499f0082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499f007d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1483400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499eff9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499f0082d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50708/0 (socket says 172.21.15.146:50708) 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 -- 172.21.15.146:0/1756393501 learned_addr learned my addr 172.21.15.146:0/1756393501 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 -- 172.21.15.146:0/1756393501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 msgr2=0x56499eff9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499eff9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 -- 172.21.15.146:0/1756393501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 msgr2=0x56499f007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499f007d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.047+0000 7f1482000640 1 -- 172.21.15.146:0/1756393501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499effa420 con 0x56499eff5460 2024-04-24T08:25:11.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f1482000640 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499f0082d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f14700c5700 tx=0x7f1470031490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5fdb0340c82e69c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f146cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f146cc00640 1 -- 172.21.15.146:0/1756393501 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470049070 con 0x56499eff5460 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f147400f5a0 con 0x56499eff5460 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f146cc00640 1 -- 172.21.15.146:0/1756393501 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f14700d6280 con 0x56499eff5460 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f146cc00640 1 -- 172.21.15.146:0/1756393501 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f14700bc7c0 con 0x56499eff5460 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f148c5a0740 10 client.26143 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56499f000f00 con 0x56499eff5460 2024-04-24T08:25:11.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.048+0000 7f146cc00640 1 -- 172.21.15.146:0/1756393501 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f147400f5a0 con 0x56499eff5460 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f146cc00640 1 -- 172.21.15.146:0/1756393501 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f14700f4b00 con 0x56499eff5460 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f148c5a0740 10 client.26143 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f148c5a0740 10 client.26143 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f148c5a0740 20 client.26143 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f076360 0x56499f078780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f148c5a0740 4 client.26143 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56499ef61a60 con 0x56499f076360 2024-04-24T08:25:11.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.049+0000 7f1482a00640 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f076360 0x56499f078780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f1482a00640 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f076360 0x56499f078780 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=84eeb07d915190ad server_cookie=860bee26da52dfab in_seq=0 out_seq=0 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f146cc00640 10 client.26143 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f146cc00640 1 -- 172.21.15.146:0/1756393501 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56499ef61a60 con 0x56499f076360 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f146cc00640 10 client.26143 handle_command_reply: tid=0 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 client.26143 shutdown 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 msgr2=0x56499f0082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499f0082d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f14700c5700 tx=0x7f1470031490 comp rx=0 tx=0).stop 2024-04-24T08:25:11.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 shutdown_connections 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x56499f076360 0x56499f078780 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56499eff5460 0x56499f0082d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56499efffb30 0x56499eff9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 --2- 172.21.15.146:0/1756393501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56499eff4b40 0x56499f007d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f146cc00640 0 client.26143 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 >> 172.21.15.146:0/1756393501 conn(0x56499eff0e60 msgr2=0x56499eff6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 shutdown_connections 2024-04-24T08:25:11.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.050+0000 7f148c5a0740 1 -- 172.21.15.146:0/1756393501 wait complete. 2024-04-24T08:25:11.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f148c5a0740 20 client.26143 trim_cache size 0 max 16384 2024-04-24T08:25:11.060 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:25:11.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14680a3d20 msgr2=0x7f14680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14680a3d20 0x7f14680a61e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f147c009fd0 tx=0x7f147c009500 comp rx=0 tx=0).stop 2024-04-24T08:25:11.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841060d0 msgr2=0x7f14841a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841060d0 0x7f14841a0b90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f147002a930 tx=0x7f1470003b20 comp rx=0 tx=0).stop 2024-04-24T08:25:11.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 shutdown_connections 2024-04-24T08:25:11.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f14680a3d20 0x7f14680a61e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f148419a7c0 0x7f148419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14841060d0 0x7f14841a0b90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 --2- 172.21.15.146:0/2349726150 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1484101000 0x7f14841a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 >> 172.21.15.146:0/2349726150 conn(0x7f14840fc860 msgr2=0x7f14840fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 shutdown_connections 2024-04-24T08:25:11.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.051+0000 7f1483e00640 1 -- 172.21.15.146:0/2349726150 wait complete. 2024-04-24T08:25:11.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:10 smithi177 ceph-mon[33046]: pgmap v122: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:25:11.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:25:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 -- 172.21.15.146:0/1575887109 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908102240 msgr2=0x7f1908102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 --2- 172.21.15.146:0/1575887109 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908102240 0x7f1908102620 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009900 tx=0x7f18fc02a400 comp rx=0 tx=0).stop 2024-04-24T08:25:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 -- 172.21.15.146:0/1575887109 shutdown_connections 2024-04-24T08:25:11.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 --2- 172.21.15.146:0/1575887109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1908102bf0 0x7f1908106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 --2- 172.21.15.146:0/1575887109 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908107310 0x7f190810eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 --2- 172.21.15.146:0/1575887109 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908102240 0x7f1908102620 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 -- 172.21.15.146:0/1575887109 >> 172.21.15.146:0/1575887109 conn(0x7f19080fda40 msgr2=0x7f19080ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 -- 172.21.15.146:0/1575887109 shutdown_connections 2024-04-24T08:25:11.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.265+0000 7f1910400640 1 -- 172.21.15.146:0/1575887109 wait complete. 2024-04-24T08:25:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.266+0000 7f1910400640 1 Processor -- start 2024-04-24T08:25:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.266+0000 7f1910400640 1 -- start start 2024-04-24T08:25:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.266+0000 7f1910400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1908102240 0x7f1908074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.266+0000 7f1910400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908107310 0x7f1908074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f1910400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908074d80 0x7f19081a6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f1910400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1908112410 con 0x7f1908107310 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f1910400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1908112290 con 0x7f1908074d80 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f1910400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1908112590 con 0x7f1908102240 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908074d80 0x7f19081a6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908074d80 0x7f19081a6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59520/0 (socket says 172.21.15.146:59520) 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 -- 172.21.15.146:0/164580298 learned_addr learned my addr 172.21.15.146:0/164580298 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190d000640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908107310 0x7f1908074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190da00640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1908102240 0x7f1908074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 -- 172.21.15.146:0/164580298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1908102240 msgr2=0x7f1908074300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1908102240 0x7f1908074300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 -- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908107310 msgr2=0x7f1908074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908107310 0x7f1908074840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.267+0000 7f190e400640 1 -- 172.21.15.146:0/164580298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19081a7240 con 0x7f1908074d80 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.268+0000 7f190e400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908074d80 0x7f19081a6b40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f190000eb10 tx=0x7f190000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.268+0000 7f1906000640 1 -- 172.21.15.146:0/164580298 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900004b20 con 0x7f1908074d80 2024-04-24T08:25:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.268+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19081a7530 con 0x7f1908074d80 2024-04-24T08:25:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.268+0000 7f1906000640 1 -- 172.21.15.146:0/164580298 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1900004cc0 con 0x7f1908074d80 2024-04-24T08:25:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.268+0000 7f1906000640 1 -- 172.21.15.146:0/164580298 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900010760 con 0x7f1908074d80 2024-04-24T08:25:11.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.268+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19081a7a80 con 0x7f1908074d80 2024-04-24T08:25:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.271+0000 7f19131b2740 1 Processor -- start 2024-04-24T08:25:11.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f1906000640 1 -- 172.21.15.146:0/164580298 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f1900005020 con 0x7f1908074d80 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f19131b2740 1 -- start start 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f19131b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715d950 0x55fd2715dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f19131b2740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd27168400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f19131b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd2716ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f19131b2740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fd270c31d0 con 0x55fd2715e270 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f19131b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd27069450 con 0x55fd2715d950 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.272+0000 7f19131b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd270bf190 con 0x55fd27168940 2024-04-24T08:25:11.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715d950 0x55fd2715dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715d950 0x55fd2715dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59526/0 (socket says 172.21.15.146:59526) 2024-04-24T08:25:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 -- 172.21.15.146:0/3299654572 learned_addr learned my addr 172.21.15.146:0/3299654572 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd2716ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190da00640 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd27168400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 -- 172.21.15.146:0/3299654572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 msgr2=0x55fd2716ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd2716ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 -- 172.21.15.146:0/3299654572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 msgr2=0x55fd27168400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd27168400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 -- 172.21.15.146:0/3299654572 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd27126310 con 0x55fd2715d950 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190da00640 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd27168400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f1906000640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f18f00a3d50 0x7f18f00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.273+0000 7f190d000640 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715d950 0x55fd2715dd30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f18f40058c0 tx=0x7f18f4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298274d3417e6bde server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.274+0000 7f1906000640 1 -- 172.21.15.146:0/164580298 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f19000d0d20 con 0x7f1908074d80 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.274+0000 7f18faa00640 1 -- 172.21.15.146:0/3299654572 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4004030 con 0x55fd2715d950 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.274+0000 7f18faa00640 1 -- 172.21.15.146:0/3299654572 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18f4005500 con 0x55fd2715d950 2024-04-24T08:25:11.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.274+0000 7f18faa00640 1 -- 172.21.15.146:0/3299654572 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4004d60 con 0x55fd2715d950 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.274+0000 7f190da00640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f18f00a3d50 0x7f18f00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.274+0000 7f19131b2740 1 -- 172.21.15.146:0/3299654572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715d950 msgr2=0x55fd2715dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.274+0000 7f19131b2740 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715d950 0x55fd2715dd30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f18f40058c0 tx=0x7f18f4005850 comp rx=0 tx=0).stop 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 -- 172.21.15.146:0/3299654572 shutdown_connections 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd2716ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd27168400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 --2- 172.21.15.146:0/3299654572 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715d950 0x55fd2715dd30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 -- 172.21.15.146:0/3299654572 >> 172.21.15.146:0/3299654572 conn(0x55fd27159c70 msgr2=0x55fd2715a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 -- 172.21.15.146:0/3299654572 shutdown_connections 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 -- 172.21.15.146:0/3299654572 wait complete. 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f190da00640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f18f00a3d50 0x7f18f00a6210 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18fc000c00 tx=0x7f18fc005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 -- start start 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.275+0000 7f19131b2740 1 Processor -- start 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f19131b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd271683b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f19131b2740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd271656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f19131b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd27168940 0x55fd27165de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f19131b2740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fd2716b7d0 con 0x55fd2715e270 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f19131b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd2707c810 con 0x55fd27168940 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f19131b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd2716b950 con 0x55fd2715d950 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f19131b2740 10 client.0 _ll_register_callbacks cb 0x55fd2704a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f190da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd271683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f190d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd271656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f190da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd271683b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50746/0 (socket says 172.21.15.146:50746) 2024-04-24T08:25:11.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f190da00640 1 -- 172.21.15.146:0/4216652619 learned_addr learned my addr 172.21.15.146:0/4216652619 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.276+0000 7f190e400640 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd27168940 0x55fd27165de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f190da00640 1 -- 172.21.15.146:0/4216652619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd27168940 msgr2=0x55fd27165de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f190da00640 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd27168940 0x55fd27165de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f190da00640 1 -- 172.21.15.146:0/4216652619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 msgr2=0x55fd271656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f190da00640 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd271656a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f190da00640 1 -- 172.21.15.146:0/4216652619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd2708d3f0 con 0x55fd2715d950 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f190e400640 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd27168940 0x55fd27165de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:11.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f190da00640 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd271683b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009900 tx=0x7f18fc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=961c5d5967fa2de4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f18ef400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f18ef400640 1 -- 172.21.15.146:0/4216652619 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc04d070 con 0x55fd2715d950 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f18ef400640 1 -- 172.21.15.146:0/4216652619 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f18fc004bf0 con 0x55fd2715d950 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.277+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18f4005500 con 0x55fd2715d950 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f18ef400640 1 -- 172.21.15.146:0/4216652619 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f18fc059050 con 0x55fd2715d950 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 10 client.26155 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fd26dd53f0 con 0x55fd2715d950 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f18ef400640 1 -- 172.21.15.146:0/4216652619 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18f4005500 con 0x55fd2715d950 2024-04-24T08:25:11.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f18ef400640 1 -- 172.21.15.146:0/4216652619 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f18fc003710 con 0x55fd2715d950 2024-04-24T08:25:11.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 10 client.26155 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:11.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 10 client.26155 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:25:11.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 20 client.26155 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:11.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd27178aa0 0x55fd2717af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 4 client.26155 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:11.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.278+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd270d0bf0 con 0x55fd27178aa0 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.279+0000 7f190d000640 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd27178aa0 0x55fd2717af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.280+0000 7f18ef400640 10 client.26155 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.280+0000 7f190d000640 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd27178aa0 0x55fd2717af40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b76903b5c15ea96 server_cookie=1351865d4d5304ff in_seq=0 out_seq=0 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.280+0000 7f18ef400640 1 -- 172.21.15.146:0/4216652619 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd270d0bf0 con 0x55fd27178aa0 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.280+0000 7f18ef400640 10 client.26155 handle_command_reply: tid=0 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.280+0000 7f19131b2740 1 client.26155 shutdown 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 msgr2=0x55fd271683b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd271683b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009900 tx=0x7f18fc009260 comp rx=0 tx=0).stop 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 shutdown_connections 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd27178aa0 0x55fd2717af40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd27168940 0x55fd27165de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715e270 0x55fd271656a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f18ef400640 0 client.26155 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 --2- 172.21.15.146:0/4216652619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd271683b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 >> 172.21.15.146:0/4216652619 conn(0x55fd27159c70 msgr2=0x55fd2715fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 shutdown_connections 2024-04-24T08:25:11.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 1 -- 172.21.15.146:0/4216652619 wait complete. 2024-04-24T08:25:11.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.281+0000 7f19131b2740 20 client.26155 trim_cache size 0 max 16384 2024-04-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 Processor -- start 2024-04-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 -- start start 2024-04-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd27174690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd27164820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd27168940 0x55fd27167ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f19081b2870 con 0x55fd27168940 2024-04-24T08:25:11.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f19081b0140 con 0x55fd2715e270 2024-04-24T08:25:11.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f19131b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19081adfb0 con 0x55fd2715d950 2024-04-24T08:25:11.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f190e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd27168940 0x55fd27167ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f190e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd27168940 0x55fd27167ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55668/0 (socket says 172.21.15.146:55668) 2024-04-24T08:25:11.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.298+0000 7f190e400640 1 -- 172.21.15.146:0/1067749747 learned_addr learned my addr 172.21.15.146:0/1067749747 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190da00640 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd27174690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190d000640 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd27164820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190e400640 1 -- 172.21.15.146:0/1067749747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 msgr2=0x55fd27174690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190e400640 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd27174690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190e400640 1 -- 172.21.15.146:0/1067749747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 msgr2=0x55fd27164820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190e400640 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd27164820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190e400640 1 -- 172.21.15.146:0/1067749747 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd2715e0a0 con 0x55fd27168940 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f190e400640 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd27168940 0x55fd27167ee0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f19000049d0 tx=0x7f190009c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e175136694c8c887 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f18fa000640 1 -- 172.21.15.146:0/1067749747 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190009cc70 con 0x55fd27168940 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f18fa000640 1 -- 172.21.15.146:0/1067749747 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19000b6070 con 0x55fd27168940 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.299+0000 7f18fa000640 1 -- 172.21.15.146:0/1067749747 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19000ce800 con 0x55fd27168940 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 -- 172.21.15.146:0/1067749747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd27168940 msgr2=0x55fd27167ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd27168940 0x55fd27167ee0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f19000049d0 tx=0x7f190009c7b0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 -- 172.21.15.146:0/1067749747 shutdown_connections 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd27168940 0x55fd27167ee0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd27164820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 --2- 172.21.15.146:0/1067749747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd2715d950 0x55fd27174690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 -- 172.21.15.146:0/1067749747 >> 172.21.15.146:0/1067749747 conn(0x55fd27159c70 msgr2=0x55fd2715f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 -- 172.21.15.146:0/1067749747 shutdown_connections 2024-04-24T08:25:11.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.300+0000 7f19131b2740 1 -- 172.21.15.146:0/1067749747 wait complete. 2024-04-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.301+0000 7f19131b2740 1 -- start start 2024-04-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.301+0000 7f19131b2740 1 Processor -- start 2024-04-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f19131b2740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715d950 0x55fd27171080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f19131b2740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd271629b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f19131b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd27162f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f19131b2740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f19081ae600 con 0x55fd2715d950 2024-04-24T08:25:11.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f19131b2740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f19081afa70 con 0x55fd2715e270 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f19131b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19081aecd0 con 0x55fd27168940 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f19131b2740 10 client.0 _ll_register_callbacks cb 0x7f18f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd271629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd271629b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59546/0 (socket says 172.21.15.146:59546) 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 -- 172.21.15.146:0/1777200791 learned_addr learned my addr 172.21.15.146:0/1777200791 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190da00640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715d950 0x55fd27171080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190e400640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd27162f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 -- 172.21.15.146:0/1777200791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 msgr2=0x55fd27162f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd27162f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 -- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715d950 msgr2=0x55fd27171080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715d950 0x55fd27171080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190d000640 1 -- 172.21.15.146:0/1777200791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd26e9f7d0 con 0x55fd2715e270 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.302+0000 7f190da00640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715d950 0x55fd27171080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f190d000640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd271629b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f18f40058c0 tx=0x7f18f4009d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73b5bcb37242268 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f18ef400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:11.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f18ef400640 1 -- 172.21.15.146:0/1777200791 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f402f070 con 0x55fd2715e270 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19000b6070 con 0x55fd2715e270 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f18ef400640 1 -- 172.21.15.146:0/1777200791 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f18f402b3b0 con 0x55fd2715e270 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f18ef400640 1 -- 172.21.15.146:0/1777200791 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f18f403b050 con 0x55fd2715e270 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f19131b2740 10 client.26180 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fd271df080 con 0x55fd2715e270 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.303+0000 7f18ef400640 1 -- 172.21.15.146:0/1777200791 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19000b6070 con 0x55fd2715e270 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f18ef400640 1 -- 172.21.15.146:0/1777200791 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f18f401f440 con 0x55fd2715e270 2024-04-24T08:25:11.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f19131b2740 10 client.26180 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f19131b2740 10 client.26180 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f19131b2740 20 client.26180 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f19131b2740 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd271df5e0 0x55fd271e1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f19131b2740 4 client.26180 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd270caa60 con 0x55fd271df5e0 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.304+0000 7f190da00640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd271df5e0 0x55fd271e1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f190da00640 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd271df5e0 0x55fd271e1a00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aab660f6ac58e70d server_cookie=822ceb149d49c27f in_seq=0 out_seq=0 2024-04-24T08:25:11.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f18ef400640 10 client.26180 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f18ef400640 1 -- 172.21.15.146:0/1777200791 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd270caa60 con 0x55fd271df5e0 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f18ef400640 10 client.26180 handle_command_reply: tid=0 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 client.26180 shutdown 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 msgr2=0x55fd271629b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd271629b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f18f40058c0 tx=0x7f18f4009d00 comp rx=0 tx=0).stop 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 shutdown_connections 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fd271df5e0 0x55fd271e1a00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd27168940 0x55fd27162f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd2715e270 0x55fd271629b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 --2- 172.21.15.146:0/1777200791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd2715d950 0x55fd27171080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f18ef400640 0 client.26180 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 >> 172.21.15.146:0/1777200791 conn(0x55fd27159c70 msgr2=0x55fd271646a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 shutdown_connections 2024-04-24T08:25:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.305+0000 7f19131b2740 1 -- 172.21.15.146:0/1777200791 wait complete. 2024-04-24T08:25:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f19131b2740 20 client.26180 trim_cache size 0 max 16384 2024-04-24T08:25:11.315 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:25:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f18f00a3d50 msgr2=0x7f18f00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f18f00a3d50 0x7f18f00a6210 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18fc000c00 tx=0x7f18fc005b50 comp rx=0 tx=0).stop 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908074d80 msgr2=0x7f19081a6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908074d80 0x7f19081a6b40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f190000eb10 tx=0x7f190000c2d0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 shutdown_connections 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f18f00a3d50 0x7f18f00a6210 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1908074d80 0x7f19081a6b40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1908107310 0x7f1908074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 --2- 172.21.15.146:0/164580298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1908102240 0x7f1908074300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 >> 172.21.15.146:0/164580298 conn(0x7f19080fda40 msgr2=0x7f19080fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.306+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 shutdown_connections 2024-04-24T08:25:11.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.307+0000 7f1910400640 1 -- 172.21.15.146:0/164580298 wait complete. 2024-04-24T08:25:11.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:25:11.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 -- 172.21.15.146:0/1496644350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108610 msgr2=0x7fb5d4108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 --2- 172.21.15.146:0/1496644350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108610 0x7fb5d4108a90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc009900 tx=0x7fb5bc02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 -- 172.21.15.146:0/1496644350 shutdown_connections 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 --2- 172.21.15.146:0/1496644350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108fd0 0x7fb5d4115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 --2- 172.21.15.146:0/1496644350 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108610 0x7fb5d4108a90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 --2- 172.21.15.146:0/1496644350 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d410e660 0x7fb5d410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 -- 172.21.15.146:0/1496644350 >> 172.21.15.146:0/1496644350 conn(0x7fb5d4076540 msgr2=0x7fb5d4078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 -- 172.21.15.146:0/1496644350 shutdown_connections 2024-04-24T08:25:11.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.519+0000 7fb5da800640 1 -- 172.21.15.146:0/1496644350 wait complete. 2024-04-24T08:25:11.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 Processor -- start 2024-04-24T08:25:11.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 -- start start 2024-04-24T08:25:11.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108610 0x7fb5d419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108fd0 0x7fb5d419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d410e660 0x7fb5d41a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d4118a30 con 0x7fb5d410e660 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41188b0 con 0x7fb5d4108fd0 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.521+0000 7fb5da800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d4118bb0 con 0x7fb5d4108610 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108610 0x7fb5d419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d410e660 0x7fb5d41a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108610 0x7fb5d419eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50788/0 (socket says 172.21.15.146:50788) 2024-04-24T08:25:11.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 -- 172.21.15.146:0/1679150620 learned_addr learned my addr 172.21.15.146:0/1679150620 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108fd0 0x7fb5d419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 -- 172.21.15.146:0/1679150620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108fd0 msgr2=0x7fb5d419f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108fd0 0x7fb5d419f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 -- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d410e660 msgr2=0x7fb5d41a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d410e660 0x7fb5d41a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 -- 172.21.15.146:0/1679150620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d41a3e70 con 0x7fb5d4108610 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.522+0000 7fb5d3400640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108610 0x7fb5d419eed0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400e9c0 tx=0x7fb5c400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.523+0000 7fb5d0c00640 1 -- 172.21.15.146:0/1679150620 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c400cd80 con 0x7fb5d4108610 2024-04-24T08:25:11.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.523+0000 7fb5d0c00640 1 -- 172.21.15.146:0/1679150620 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c40035f0 con 0x7fb5d4108610 2024-04-24T08:25:11.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.523+0000 7fb5d0c00640 1 -- 172.21.15.146:0/1679150620 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4010690 con 0x7fb5d4108610 2024-04-24T08:25:11.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.523+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d41a4160 con 0x7fb5d4108610 2024-04-24T08:25:11.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.523+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d41ab9f0 con 0x7fb5d4108610 2024-04-24T08:25:11.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.527+0000 7fb5dd591740 1 Processor -- start 2024-04-24T08:25:11.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.527+0000 7fb5dd591740 1 -- start start 2024-04-24T08:25:11.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b4dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b58400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b5ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5dd591740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dae2ab31d0 con 0x55dae2b4d950 2024-04-24T08:25:11.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5dd591740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dae2a59450 con 0x55dae2b4e270 2024-04-24T08:25:11.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5dd591740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dae2aaf190 con 0x55dae2b58940 2024-04-24T08:25:11.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d0c00640 1 -- 172.21.15.146:0/1679150620 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fb5c40037a0 con 0x7fb5d4108610 2024-04-24T08:25:11.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b4dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b4dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55722/0 (socket says 172.21.15.146:55722) 2024-04-24T08:25:11.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 -- 172.21.15.146:0/3069236653 learned_addr learned my addr 172.21.15.146:0/3069236653 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3400640 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b58400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b5ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 -- 172.21.15.146:0/3069236653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 msgr2=0x55dae2b5ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b5ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 -- 172.21.15.146:0/3069236653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 msgr2=0x55dae2b58400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b58400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.528+0000 7fb5d3e00640 1 -- 172.21.15.146:0/3069236653 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae2b16310 con 0x55dae2b4d950 2024-04-24T08:25:11.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.529+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b4dd30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009fd0 tx=0x7fb5cc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c8a8511225e48f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.529+0000 7fb5c8c00640 1 -- 172.21.15.146:0/3069236653 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc004d20 con 0x55dae2b4d950 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.529+0000 7fb5c8c00640 1 -- 172.21.15.146:0/3069236653 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5cc004050 con 0x55dae2b4d950 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.529+0000 7fb5c8c00640 1 -- 172.21.15.146:0/3069236653 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc005260 con 0x55dae2b4d950 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5d0c00640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb5b80a3d20 0x7fb5b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 -- 172.21.15.146:0/3069236653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 msgr2=0x55dae2b4dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b4dd30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009fd0 tx=0x7fb5cc0118d0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5d0c00640 1 -- 172.21.15.146:0/1679150620 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb5c40d0c40 con 0x7fb5d4108610 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5d3400640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb5b80a3d20 0x7fb5b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 -- 172.21.15.146:0/3069236653 shutdown_connections 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b5ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b58400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3069236653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b4dd30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 -- 172.21.15.146:0/3069236653 >> 172.21.15.146:0/3069236653 conn(0x55dae2b49c70 msgr2=0x55dae2b4a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 -- 172.21.15.146:0/3069236653 shutdown_connections 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.530+0000 7fb5dd591740 1 -- 172.21.15.146:0/3069236653 wait complete. 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.531+0000 7fb5d3400640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb5b80a3d20 0x7fb5b80a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55dae2b4ef10 tx=0x7fb5c40d46f0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.531+0000 7fb5dd591740 1 -- start start 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.531+0000 7fb5dd591740 1 Processor -- start 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b58340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b55630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b55c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5dd591740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dae2b5b7d0 con 0x55dae2b4d950 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5dd591740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dae2a6c810 con 0x55dae2b4e270 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5dd591740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dae2b5b950 con 0x55dae2b58940 2024-04-24T08:25:11.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5dd591740 10 client.0 _ll_register_callbacks cb 0x55dae2a3a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b58340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5cbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b55630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b58340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55730/0 (socket says 172.21.15.146:55730) 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5d3e00640 1 -- 172.21.15.146:0/3417664566 learned_addr learned my addr 172.21.15.146:0/3417664566 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b55c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5cbe00640 1 -- 172.21.15.146:0/3417664566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 msgr2=0x55dae2b55c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b55c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5cbe00640 1 -- 172.21.15.146:0/3417664566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 msgr2=0x55dae2b58340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b58340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.532+0000 7fb5cbe00640 1 -- 172.21.15.146:0/3417664566 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae2a7d3f0 con 0x55dae2b4e270 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b55630 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc02a9e0 tx=0x7fb5bc0043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c901942fe94cc48c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5c1600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5c1600640 1 -- 172.21.15.146:0/3417664566 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc035040 con 0x55dae2b4e270 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5c1600640 1 -- 172.21.15.146:0/3417664566 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb5bc005450 con 0x55dae2b4e270 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5cc004050 con 0x55dae2b4e270 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5c1600640 1 -- 172.21.15.146:0/3417664566 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb5bc005650 con 0x55dae2b4e270 2024-04-24T08:25:11.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5dd591740 10 client.26192 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.533+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dae27c53f0 con 0x55dae2b4e270 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5c1600640 1 -- 172.21.15.146:0/3417664566 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5cc004050 con 0x55dae2b4e270 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5c1600640 1 -- 172.21.15.146:0/3417664566 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb5bc003a00 con 0x55dae2b4e270 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5dd591740 10 client.26192 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5dd591740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5dd591740 20 client.26192 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2b68ab0 0x55dae2b6aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5dd591740 4 client.26192 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.534+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dae2ac0bf0 con 0x55dae2b68ab0 2024-04-24T08:25:11.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.535+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2b68ab0 0x55dae2b6aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.535+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2b68ab0 0x55dae2b6aec0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e115040bd128c6d6 server_cookie=2fd1e9423d0ae76d in_seq=0 out_seq=0 2024-04-24T08:25:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.535+0000 7fb5c1600640 10 client.26192 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5c1600640 1 -- 172.21.15.146:0/3417664566 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dae2ac0bf0 con 0x55dae2b68ab0 2024-04-24T08:25:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5c1600640 10 client.26192 handle_command_reply: tid=0 2024-04-24T08:25:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 client.26192 shutdown 2024-04-24T08:25:11.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 msgr2=0x55dae2b55630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b55630 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc02a9e0 tx=0x7fb5bc0043b0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 shutdown_connections 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2b68ab0 0x55dae2b6aec0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b55c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5c1600640 0 client.26192 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b55630 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 --2- 172.21.15.146:0/3417664566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b58340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.536+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 >> 172.21.15.146:0/3417664566 conn(0x55dae2b49c70 msgr2=0x55dae2b4f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.537+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 shutdown_connections 2024-04-24T08:25:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.537+0000 7fb5dd591740 1 -- 172.21.15.146:0/3417664566 wait complete. 2024-04-24T08:25:11.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.537+0000 7fb5dd591740 20 client.26192 trim_cache size 0 max 16384 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 Processor -- start 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 -- start start 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b4d950 0x55dae2b500b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b57e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b58940 0x55dae2b70790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41b6bb0 con 0x55dae2b58940 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41b4480 con 0x55dae2b4e270 2024-04-24T08:25:11.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.552+0000 7fb5dd591740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41b22d0 con 0x55dae2b4d950 2024-04-24T08:25:11.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b4d950 0x55dae2b500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5cbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b57e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b4d950 0x55dae2b500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50818/0 (socket says 172.21.15.146:50818) 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5d3e00640 1 -- 172.21.15.146:0/553494907 learned_addr learned my addr 172.21.15.146:0/553494907 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b58940 0x55dae2b70790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5cbe00640 1 -- 172.21.15.146:0/553494907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b4d950 msgr2=0x55dae2b500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b4d950 0x55dae2b500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5cbe00640 1 -- 172.21.15.146:0/553494907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b58940 msgr2=0x55dae2b70790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b58940 0x55dae2b70790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5cbe00640 1 -- 172.21.15.146:0/553494907 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae288f7d0 con 0x55dae2b4e270 2024-04-24T08:25:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.553+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b58940 0x55dae2b70790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:11.556 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.554+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b57e70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc003dd0 tx=0x7fb5bc031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52b207750148623e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.554+0000 7fb5c3e00640 1 -- 172.21.15.146:0/553494907 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc035030 con 0x55dae2b4e270 2024-04-24T08:25:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.554+0000 7fb5c3e00640 1 -- 172.21.15.146:0/553494907 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5bc041070 con 0x55dae2b4e270 2024-04-24T08:25:11.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.554+0000 7fb5c3e00640 1 -- 172.21.15.146:0/553494907 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc053070 con 0x55dae2b4e270 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 -- 172.21.15.146:0/553494907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 msgr2=0x55dae2b57e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b57e70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc003dd0 tx=0x7fb5bc031400 comp rx=0 tx=0).stop 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 -- 172.21.15.146:0/553494907 shutdown_connections 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b58940 0x55dae2b70790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b57e70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 --2- 172.21.15.146:0/553494907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b4d950 0x55dae2b500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 -- 172.21.15.146:0/553494907 >> 172.21.15.146:0/553494907 conn(0x55dae2b49c70 msgr2=0x55dae2b63680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 -- 172.21.15.146:0/553494907 shutdown_connections 2024-04-24T08:25:11.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 -- 172.21.15.146:0/553494907 wait complete. 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.555+0000 7fb5dd591740 1 -- start start 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 1 Processor -- start 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b60ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b610e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b59c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41b2940 con 0x55dae2b4d950 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41b3db0 con 0x55dae2b4e270 2024-04-24T08:25:11.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d41b3010 con 0x55dae2b58940 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5dd591740 10 client.0 _ll_register_callbacks cb 0x7fb5bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b60ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b60ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55744/0 (socket says 172.21.15.146:55744) 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5cbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.556+0000 7fb5d3e00640 1 -- 172.21.15.146:0/2106663704 learned_addr learned my addr 172.21.15.146:0/2106663704 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b59c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5cbe00640 1 -- 172.21.15.146:0/2106663704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 msgr2=0x55dae2b59c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b59c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5cbe00640 1 -- 172.21.15.146:0/2106663704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 msgr2=0x55dae2b60ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b60ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5cbe00640 1 -- 172.21.15.146:0/2106663704 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae2b5a190 con 0x55dae2b4e270 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5cbe00640 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b610e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55dae2b55200 tx=0x7fb5bc0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e31949879615afb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5c1600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5c1600640 1 -- 172.21.15.146:0/2106663704 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc035070 con 0x55dae2b4e270 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5c1600640 1 -- 172.21.15.146:0/2106663704 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb5bc031d70 con 0x55dae2b4e270 2024-04-24T08:25:11.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.557+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5bc041070 con 0x55dae2b4e270 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5c1600640 1 -- 172.21.15.146:0/2106663704 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb5bc053070 con 0x55dae2b4e270 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 10 client.26198 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dae2bcf820 con 0x55dae2b4e270 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5c1600640 1 -- 172.21.15.146:0/2106663704 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5bc041070 con 0x55dae2b4e270 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5c1600640 1 -- 172.21.15.146:0/2106663704 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb5bc004850 con 0x55dae2b4e270 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 10 client.26198 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 10 client.26198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:11.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 20 client.26198 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2bcfdd0 0x55dae2bd21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 4 client.26198 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:25:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.558+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dae2abaa60 con 0x55dae2bcfdd0 2024-04-24T08:25:11.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2bcfdd0 0x55dae2bd21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:11.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5d3e00640 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2bcfdd0 0x55dae2bd21f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71fc368202ce04be server_cookie=14aecb6b6c9f2671 in_seq=0 out_seq=0 2024-04-24T08:25:11.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5c1600640 10 client.26198 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5c1600640 1 -- 172.21.15.146:0/2106663704 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dae2abaa60 con 0x55dae2bcfdd0 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5c1600640 10 client.26198 handle_command_reply: tid=0 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 client.26198 shutdown 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 msgr2=0x55dae2b610e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b610e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55dae2b55200 tx=0x7fb5bc0095f0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 shutdown_connections 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55dae2bcfdd0 0x55dae2bd21f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dae2b58940 0x55dae2b59c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dae2b4e270 0x55dae2b610e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5c1600640 0 client.26198 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 --2- 172.21.15.146:0/2106663704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dae2b4d950 0x55dae2b60ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.559+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 >> 172.21.15.146:0/2106663704 conn(0x55dae2b49c70 msgr2=0x55dae2b4f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 shutdown_connections 2024-04-24T08:25:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5dd591740 1 -- 172.21.15.146:0/2106663704 wait complete. 2024-04-24T08:25:11.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5dd591740 20 client.26198 trim_cache size 0 max 16384 2024-04-24T08:25:11.569 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:25:11.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb5b80a3d20 msgr2=0x7fb5b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb5b80a3d20 0x7fb5b80a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55dae2b4ef10 tx=0x7fb5c40d46f0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108610 msgr2=0x7fb5d419eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108610 0x7fb5d419eed0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb5c400e9c0 tx=0x7fb5c400ee80 comp rx=0 tx=0).stop 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 shutdown_connections 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb5b80a3d20 0x7fb5b80a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d410e660 0x7fb5d41a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d4108fd0 0x7fb5d419f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 --2- 172.21.15.146:0/1679150620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5d4108610 0x7fb5d419eed0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.560+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 >> 172.21.15.146:0/1679150620 conn(0x7fb5d4076540 msgr2=0x7fb5d4077b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.561+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 shutdown_connections 2024-04-24T08:25:11.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:11.561+0000 7fb5da800640 1 -- 172.21.15.146:0/1679150620 wait complete. 2024-04-24T08:25:11.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:11 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/131901556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:25:11.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:11 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/131901556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:25:12.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:11 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/131901556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:25:12.579 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:25:12.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:25:12.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:25:12.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 -- 172.21.15.146:0/2161268032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8498104cc0 msgr2=0x7f8498108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 --2- 172.21.15.146:0/2161268032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8498104cc0 0x7f8498108d00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8484009860 tx=0x7f848402a340 comp rx=0 tx=0).stop 2024-04-24T08:25:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 -- 172.21.15.146:0/2161268032 shutdown_connections 2024-04-24T08:25:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 --2- 172.21.15.146:0/2161268032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84981093e0 0x7f8498110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 --2- 172.21.15.146:0/2161268032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8498104cc0 0x7f8498108d00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 --2- 172.21.15.146:0/2161268032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8498104310 0x7f84981046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 -- 172.21.15.146:0/2161268032 >> 172.21.15.146:0/2161268032 conn(0x7f84980ffb10 msgr2=0x7f8498101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 -- 172.21.15.146:0/2161268032 shutdown_connections 2024-04-24T08:25:12.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.797+0000 7f8497200640 1 -- 172.21.15.146:0/2161268032 wait complete. 2024-04-24T08:25:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.798+0000 7f8497200640 1 Processor -- start 2024-04-24T08:25:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.798+0000 7f8497200640 1 -- start start 2024-04-24T08:25:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8497200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498104310 0x7f84981a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8497200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84981093e0 0x7f84981a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8497200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f849819a7b0 0x7f849819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8497200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8498114450 con 0x7f84981093e0 2024-04-24T08:25:12.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8497200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84981142d0 con 0x7f849819a7b0 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8497200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84981145d0 con 0x7f8498104310 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8495400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84981093e0 0x7f84981a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8496800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f849819a7b0 0x7f849819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8495400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84981093e0 0x7f84981a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55768/0 (socket says 172.21.15.146:55768) 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8495e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498104310 0x7f84981a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.799+0000 7f8495400640 1 -- 172.21.15.146:0/767783042 learned_addr learned my addr 172.21.15.146:0/767783042 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8496800640 1 -- 172.21.15.146:0/767783042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498104310 msgr2=0x7f84981a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8496800640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498104310 0x7f84981a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8496800640 1 -- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84981093e0 msgr2=0x7f84981a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8496800640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84981093e0 0x7f84981a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8496800640 1 -- 172.21.15.146:0/767783042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849819b3b0 con 0x7f849819a7b0 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8495400640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84981093e0 0x7f84981a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8496800640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f849819a7b0 0x7f849819ac30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8490009fd0 tx=0x7f849000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f848ea00640 1 -- 172.21.15.146:0/767783042 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849000cd60 con 0x7f849819a7b0 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f848ea00640 1 -- 172.21.15.146:0/767783042 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8490005020 con 0x7f849819a7b0 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84981a7440 con 0x7f849819a7b0 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f848ea00640 1 -- 172.21.15.146:0/767783042 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490010690 con 0x7f849819a7b0 2024-04-24T08:25:12.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.800+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84981a7840 con 0x7f849819a7b0 2024-04-24T08:25:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.803+0000 7f84a002b740 1 Processor -- start 2024-04-24T08:25:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.803+0000 7f848ea00640 1 -- 172.21.15.146:0/767783042 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f8490005540 con 0x7f849819a7b0 2024-04-24T08:25:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f84a002b740 1 -- start start 2024-04-24T08:25:12.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f84a002b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528270 0x5633e9528650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f84a002b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f84a002b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9533260 0x5633e9535650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f84a002b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5633e948de70 con 0x5633e9528270 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f84a002b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5633e9434380 con 0x5633e9533260 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f84a002b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5633e9489e30 con 0x5633e9528b90 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f8495400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528270 0x5633e9528650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.804+0000 7f8495e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50844/0 (socket says 172.21.15.146:50844) 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 -- 172.21.15.146:0/4012194023 learned_addr learned my addr 172.21.15.146:0/4012194023 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495400640 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9533260 0x5633e9535650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 -- 172.21.15.146:0/4012194023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9533260 msgr2=0x5633e9535650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9533260 0x5633e9535650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 -- 172.21.15.146:0/4012194023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528270 msgr2=0x5633e9528650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528270 0x5633e9528650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 -- 172.21.15.146:0/4012194023 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633e92687e0 con 0x5633e9528b90 2024-04-24T08:25:12.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495400640 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528270 0x5633e9528650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8495e00640 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532d20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8488009cd0 tx=0x7f8488011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc5b878fe8d741bc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f848ea00640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f847c0a3da0 0x7f847c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.805+0000 7f8483400640 1 -- 172.21.15.146:0/4012194023 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848801e070 con 0x5633e9528b90 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f8483400640 1 -- 172.21.15.146:0/4012194023 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8488019b10 con 0x5633e9528b90 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f848ea00640 1 -- 172.21.15.146:0/767783042 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f84900d0d60 con 0x7f849819a7b0 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f8483400640 1 -- 172.21.15.146:0/4012194023 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848800eb50 con 0x5633e9528b90 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f8495400640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f847c0a3da0 0x7f847c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f84a002b740 1 -- 172.21.15.146:0/4012194023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 msgr2=0x5633e9532d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f84a002b740 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532d20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8488009cd0 tx=0x7f8488011810 comp rx=0 tx=0).stop 2024-04-24T08:25:12.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f8495400640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f847c0a3da0 0x7f847c0a6260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8484005d90 tx=0x7f8484005d20 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f84a002b740 1 -- 172.21.15.146:0/4012194023 shutdown_connections 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f84a002b740 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532d20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f84a002b740 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9533260 0x5633e9535650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f84a002b740 1 --2- 172.21.15.146:0/4012194023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528270 0x5633e9528650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.806+0000 7f84a002b740 1 -- 172.21.15.146:0/4012194023 >> 172.21.15.146:0/4012194023 conn(0x5633e9524590 msgr2=0x5633e9524970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.807+0000 7f84a002b740 1 -- 172.21.15.146:0/4012194023 shutdown_connections 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.807+0000 7f84a002b740 1 -- 172.21.15.146:0/4012194023 wait complete. 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.807+0000 7f84a002b740 1 -- start start 2024-04-24T08:25:12.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.807+0000 7f84a002b740 1 Processor -- start 2024-04-24T08:25:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f84a002b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e95301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f84a002b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9530730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f84a002b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e9530c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f84a002b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5633e9536050 con 0x5633e9533260 2024-04-24T08:25:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f84a002b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5633e9447660 con 0x5633e9528270 2024-04-24T08:25:12.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f84a002b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5633e95361d0 con 0x5633e9528b90 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f84a002b740 10 client.0 _ll_register_callbacks cb 0x5633e9435aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e95301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e95301f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59640/0 (socket says 172.21.15.146:59640) 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495e00640 1 -- 172.21.15.146:0/935186755 learned_addr learned my addr 172.21.15.146:0/935186755 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495400640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9530730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8496800640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e9530c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495e00640 1 -- 172.21.15.146:0/935186755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 msgr2=0x5633e9530730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495e00640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9530730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495e00640 1 -- 172.21.15.146:0/935186755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 msgr2=0x5633e9530c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.808+0000 7f8495e00640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e9530c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f8495e00640 1 -- 172.21.15.146:0/935186755 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633e92c8f70 con 0x5633e9528270 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f8495400640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9530730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f8495e00640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e95301f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8488005890 tx=0x7f84880094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff33c59ec31b54df server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f8477e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f8477e00640 1 -- 172.21.15.146:0/935186755 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488029020 con 0x5633e9528270 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8488019b10 con 0x5633e9528270 2024-04-24T08:25:12.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f8477e00640 1 -- 172.21.15.146:0/935186755 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8488005040 con 0x5633e9528270 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f8477e00640 1 -- 172.21.15.146:0/935186755 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8488034050 con 0x5633e9528270 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f84a002b740 10 client.26210 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.809+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5633e9160170 con 0x5633e9528270 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f8477e00640 1 -- 172.21.15.146:0/935186755 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8488019b10 con 0x5633e9528270 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f8477e00640 1 -- 172.21.15.146:0/935186755 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f84880046c0 con 0x5633e9528270 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f84a002b740 10 client.26210 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f84a002b740 10 client.26210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:12.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f84a002b740 20 client.26210 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f84a002b740 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e9543580 0x5633e9545970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f84a002b740 4 client.26210 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633e949b890 con 0x5633e9543580 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.810+0000 7f8495400640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e9543580 0x5633e9545970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.811+0000 7f8495400640 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e9543580 0x5633e9545970 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c94c3587af6f2ea3 server_cookie=b2abc4ec8b6eb653 in_seq=0 out_seq=0 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.811+0000 7f8477e00640 10 client.26210 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f8477e00640 1 -- 172.21.15.146:0/935186755 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5633e949b890 con 0x5633e9543580 2024-04-24T08:25:12.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f8477e00640 10 client.26210 handle_command_reply: tid=0 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 client.26210 shutdown 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 msgr2=0x5633e95301f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e95301f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8488005890 tx=0x7f84880094a0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 shutdown_connections 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e9543580 0x5633e9545970 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9530730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e9530c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 --2- 172.21.15.146:0/935186755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e95301f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 >> 172.21.15.146:0/935186755 conn(0x5633e9524590 msgr2=0x5633e952a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:12.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f8477e00640 0 client.26210 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:12.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 shutdown_connections 2024-04-24T08:25:12.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 1 -- 172.21.15.146:0/935186755 wait complete. 2024-04-24T08:25:12.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.812+0000 7f84a002b740 20 client.26210 trim_cache size 0 max 16384 2024-04-24T08:25:12.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 Processor -- start 2024-04-24T08:25:12.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 -- start start 2024-04-24T08:25:12.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e953ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e954b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f84981b2590 con 0x5633e9533260 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84981afe60 con 0x5633e9528270 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f84a002b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84981adcb0 con 0x5633e9528b90 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f8495e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e953ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f8495e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e953ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59642/0 (socket says 172.21.15.146:59642) 2024-04-24T08:25:12.834 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f8495e00640 1 -- 172.21.15.146:0/1697952339 learned_addr learned my addr 172.21.15.146:0/1697952339 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:12.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f8496800640 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e954b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.832+0000 7f8495400640 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8495e00640 1 -- 172.21.15.146:0/1697952339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 msgr2=0x5633e9532130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8495e00640 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8495e00640 1 -- 172.21.15.146:0/1697952339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 msgr2=0x5633e954b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8495e00640 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e954b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8495e00640 1 -- 172.21.15.146:0/1697952339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633e9545ed0 con 0x5633e9528270 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8495e00640 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e953ef10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f848800e5e0 tx=0x7f8488001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce7c7b59e31e1a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8482a00640 1 -- 172.21.15.146:0/1697952339 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488029070 con 0x5633e9528270 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8482a00640 1 -- 172.21.15.146:0/1697952339 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f848801f040 con 0x5633e9528270 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.833+0000 7f8482a00640 1 -- 172.21.15.146:0/1697952339 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848801e070 con 0x5633e9528270 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 -- 172.21.15.146:0/1697952339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 msgr2=0x5633e953ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e953ef10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f848800e5e0 tx=0x7f8488001e70 comp rx=0 tx=0).stop 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 -- 172.21.15.146:0/1697952339 shutdown_connections 2024-04-24T08:25:12.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9528b90 0x5633e9532130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9533260 0x5633e954b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 --2- 172.21.15.146:0/1697952339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e953ef10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 -- 172.21.15.146:0/1697952339 >> 172.21.15.146:0/1697952339 conn(0x5633e9524590 msgr2=0x5633e9531a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 -- 172.21.15.146:0/1697952339 shutdown_connections 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 -- 172.21.15.146:0/1697952339 wait complete. 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.834+0000 7f84a002b740 1 -- start start 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 1 Processor -- start 2024-04-24T08:25:12.840 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e9536d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528b90 0x5633e95372a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9533260 0x5633e95377e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f84981ae320 con 0x5633e9528b90 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84981af790 con 0x5633e9528270 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84981ae9f0 con 0x5633e9533260 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f84a002b740 10 client.0 _ll_register_callbacks cb 0x7f848801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f8495e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e9536d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f8495e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e9536d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59650/0 (socket says 172.21.15.146:59650) 2024-04-24T08:25:12.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f8495e00640 1 -- 172.21.15.146:0/3491490723 learned_addr learned my addr 172.21.15.146:0/3491490723 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f8495400640 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528b90 0x5633e95372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.835+0000 7f8496800640 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9533260 0x5633e95377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8495e00640 1 -- 172.21.15.146:0/3491490723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9533260 msgr2=0x5633e95377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8495e00640 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9533260 0x5633e95377e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8495e00640 1 -- 172.21.15.146:0/3491490723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528b90 msgr2=0x5633e95372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8495e00640 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528b90 0x5633e95372a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8495e00640 1 -- 172.21.15.146:0/3491490723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633e95b1180 con 0x5633e9528270 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8495e00640 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e9536d60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8488012fd0 tx=0x7f8488009a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87bf91f54512e69b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:12.842 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8477e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8477e00640 1 -- 172.21.15.146:0/3491490723 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488029070 con 0x5633e9528270 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f848801f040 con 0x5633e9528270 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8477e00640 1 -- 172.21.15.146:0/3491490723 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f8488005190 con 0x5633e9528270 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f8477e00640 1 -- 172.21.15.146:0/3491490723 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f848801e040 con 0x5633e9528270 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f84a002b740 10 client.26216 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.836+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5633e95b23d0 con 0x5633e9528270 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f8477e00640 1 -- 172.21.15.146:0/3491490723 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f848801f040 con 0x5633e9528270 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f8477e00640 1 -- 172.21.15.146:0/3491490723 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f84880035f0 con 0x5633e9528270 2024-04-24T08:25:12.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f84a002b740 10 client.26216 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f84a002b740 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f84a002b740 20 client.26216 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f84a002b740 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e95b2980 0x5633e95b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f84a002b740 4 client.26216 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.837+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5633e9495700 con 0x5633e95b2980 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.838+0000 7f8495400640 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e95b2980 0x5633e95b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.838+0000 7f8495400640 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e95b2980 0x5633e95b4da0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f481ee16aa6822f1 server_cookie=9ad83c66696c5381 in_seq=0 out_seq=0 2024-04-24T08:25:12.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.838+0000 7f8477e00640 10 client.26216 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:12.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.839+0000 7f8477e00640 1 -- 172.21.15.146:0/3491490723 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5633e9495700 con 0x5633e95b2980 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.839+0000 7f8477e00640 10 client.26216 handle_command_reply: tid=0 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.839+0000 7f84a002b740 1 client.26216 shutdown 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.839+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 msgr2=0x5633e9536d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.839+0000 7f84a002b740 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e9536d60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8488012fd0 tx=0x7f8488009a10 comp rx=0 tx=0).stop 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 shutdown_connections 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5633e95b2980 0x5633e95b4da0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5633e9533260 0x5633e95377e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5633e9528b90 0x5633e95372a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f8477e00640 0 client.26216 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:12.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 --2- 172.21.15.146:0/3491490723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5633e9528270 0x5633e9536d60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 >> 172.21.15.146:0/3491490723 conn(0x5633e9524590 msgr2=0x5633e95322b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:12.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 shutdown_connections 2024-04-24T08:25:12.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 1 -- 172.21.15.146:0/3491490723 wait complete. 2024-04-24T08:25:12.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f84a002b740 20 client.26216 trim_cache size 0 max 16384 2024-04-24T08:25:12.847 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:25:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:25:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "78e4f0eb-1288-4f2f-bee4-f4648caea4fc", 2024-04-24T08:25:12.848 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:25:12.848 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:25:12.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f847c0a3da0 msgr2=0x7f847c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f8497200640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f847c0a3da0 0x7f847c0a6260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8484005d90 tx=0x7f8484005d20 comp rx=0 tx=0).stop 2024-04-24T08:25:12.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f849819a7b0 msgr2=0x7f849819ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:12.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.840+0000 7f8497200640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f849819a7b0 0x7f849819ac30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8490009fd0 tx=0x7f849000ee50 comp rx=0 tx=0).stop 2024-04-24T08:25:12.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 shutdown_connections 2024-04-24T08:25:12.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f847c0a3da0 0x7f847c0a6260 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f849819a7b0 0x7f849819ac30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84981093e0 0x7f84981a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 --2- 172.21.15.146:0/767783042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498104310 0x7f84981a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 >> 172.21.15.146:0/767783042 conn(0x7f84980ffb10 msgr2=0x7f8498101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 shutdown_connections 2024-04-24T08:25:12.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:12.841+0000 7f8497200640 1 -- 172.21.15.146:0/767783042 wait complete. 2024-04-24T08:25:12.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:25:12.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:25:12.945 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:12 smithi146 ceph-mon[27805]: pgmap v123: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:25:12.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:12 smithi153 ceph-mon[32746]: pgmap v123: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:25:13.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:12 smithi177 ceph-mon[33046]: pgmap v123: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:25:13.137 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.136+0000 7fbc5a000640 1 -- 172.21.15.146:0/1311819189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc541019b0 msgr2=0x7fbc541059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.136+0000 7fbc5a000640 1 --2- 172.21.15.146:0/1311819189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc541019b0 0x7fbc541059f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009900 tx=0x7fbc4002a400 comp rx=0 tx=0).stop 2024-04-24T08:25:13.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.137+0000 7fbc5a000640 1 -- 172.21.15.146:0/1311819189 shutdown_connections 2024-04-24T08:25:13.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.137+0000 7fbc5a000640 1 --2- 172.21.15.146:0/1311819189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc541019b0 0x7fbc541059f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.137+0000 7fbc5a000640 1 --2- 172.21.15.146:0/1311819189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc541060d0 0x7fbc5410dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.137+0000 7fbc5a000640 1 --2- 172.21.15.146:0/1311819189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 0x7fbc541013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.138 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.137+0000 7fbc5a000640 1 -- 172.21.15.146:0/1311819189 >> 172.21.15.146:0/1311819189 conn(0x7fbc540fc880 msgr2=0x7fbc540feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:13.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.137+0000 7fbc5a000640 1 -- 172.21.15.146:0/1311819189 shutdown_connections 2024-04-24T08:25:13.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.137+0000 7fbc5a000640 1 -- 172.21.15.146:0/1311819189 wait complete. 2024-04-24T08:25:13.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 Processor -- start 2024-04-24T08:25:13.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 -- start start 2024-04-24T08:25:13.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 0x7fbc5410feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc541060d0 0x7fbc5410e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc5410e840 0x7fbc5410ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbc54112260 con 0x7fbc54101000 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc541120e0 con 0x7fbc541060d0 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.138+0000 7fbc5a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbc541123e0 con 0x7fbc5410e840 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 0x7fbc5410feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 0x7fbc5410feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55824/0 (socket says 172.21.15.146:55824) 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 -- 172.21.15.146:0/880294285 learned_addr learned my addr 172.21.15.146:0/880294285 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:13.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc53e00640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc541060d0 0x7fbc5410e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc59600640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc5410e840 0x7fbc5410ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 -- 172.21.15.146:0/880294285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc5410e840 msgr2=0x7fbc5410ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc5410e840 0x7fbc5410ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 -- 172.21.15.146:0/880294285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc541060d0 msgr2=0x7fbc5410e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc541060d0 0x7fbc5410e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.139+0000 7fbc58c00640 1 -- 172.21.15.146:0/880294285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc541ab420 con 0x7fbc54101000 2024-04-24T08:25:13.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.140+0000 7fbc58c00640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 0x7fbc5410feb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800e9f0 tx=0x7fbc4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:13.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.140+0000 7fbc51600640 1 -- 172.21.15.146:0/880294285 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4800cd80 con 0x7fbc54101000 2024-04-24T08:25:13.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.140+0000 7fbc51600640 1 -- 172.21.15.146:0/880294285 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc48010910 con 0x7fbc54101000 2024-04-24T08:25:13.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.140+0000 7fbc51600640 1 -- 172.21.15.146:0/880294285 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48018990 con 0x7fbc54101000 2024-04-24T08:25:13.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.141+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc541ab710 con 0x7fbc54101000 2024-04-24T08:25:13.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.141+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc541abbe0 con 0x7fbc54101000 2024-04-24T08:25:13.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 Processor -- start 2024-04-24T08:25:13.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 -- start start 2024-04-24T08:25:13.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b32e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b3d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b3fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e083a981f0 con 0x55e083b33350 2024-04-24T08:25:13.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e083a3e350 con 0x55e083b32a30 2024-04-24T08:25:13.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.145+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e083a941b0 con 0x55e083b3da20 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc59600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b32e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc53e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b3fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc59600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b32e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59676/0 (socket says 172.21.15.146:59676) 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc59600640 1 -- 172.21.15.146:0/1132348795 learned_addr learned my addr 172.21.15.146:0/1132348795 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc51600640 1 -- 172.21.15.146:0/880294285 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fbc48010ac0 con 0x7fbc54101000 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc53e00640 1 -- 172.21.15.146:0/1132348795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 msgr2=0x55e083b32e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b32e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc53e00640 1 -- 172.21.15.146:0/1132348795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 msgr2=0x55e083b3d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b3d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc53e00640 1 -- 172.21.15.146:0/1132348795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e083a92f50 con 0x55e083b3da20 2024-04-24T08:25:13.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.146+0000 7fbc59600640 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b32e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.147+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b3fe10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbc40006010 tx=0x7fbc40005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d199b5f03eaf326 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.147+0000 7fbc45c00640 1 -- 172.21.15.146:0/1132348795 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40040070 con 0x55e083b3da20 2024-04-24T08:25:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.147+0000 7fbc45c00640 1 -- 172.21.15.146:0/1132348795 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc40005500 con 0x55e083b3da20 2024-04-24T08:25:13.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.147+0000 7fbc45c00640 1 -- 172.21.15.146:0/1132348795 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40035040 con 0x55e083b3da20 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1132348795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 msgr2=0x55e083b3fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b3fe10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbc40006010 tx=0x7fbc40005be0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc51600640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbc3c0a3e20 0x7fbc3c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc51600640 1 -- 172.21.15.146:0/880294285 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbc480d0db0 con 0x7fbc54101000 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1132348795 shutdown_connections 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b3fe10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b3d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.148+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1132348795 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b32e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.149+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1132348795 >> 172.21.15.146:0/1132348795 conn(0x55e083b2ed50 msgr2=0x55e083b2f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:13.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.149+0000 7fbc59600640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbc3c0a3e20 0x7fbc3c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.149+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1132348795 shutdown_connections 2024-04-24T08:25:13.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.149+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1132348795 wait complete. 2024-04-24T08:25:13.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.149+0000 7fbc5e44c740 1 -- start start 2024-04-24T08:25:13.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.149+0000 7fbc59600640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbc3c0a3e20 0x7fbc3c0a62e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c005fd0 tx=0x7fbc4c009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:13.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 1 Processor -- start 2024-04-24T08:25:13.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b32a30 0x55e083b3aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b3afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b3da20 0x55e083b3b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55e083b407b0 con 0x55e083b33350 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55e083a51690 con 0x55e083b3da20 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e083b40930 con 0x55e083b32a30 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc5e44c740 10 client.0 _ll_register_callbacks cb 0x55e083a8d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc58c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b32a30 0x55e083b3aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc58c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b32a30 0x55e083b3aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50902/0 (socket says 172.21.15.146:50902) 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc58c00640 1 -- 172.21.15.146:0/2754544493 learned_addr learned my addr 172.21.15.146:0/2754544493 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:13.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc53e00640 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b3afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.150+0000 7fbc59600640 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b3da20 0x55e083b3b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc58c00640 1 -- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b3da20 msgr2=0x55e083b3b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc58c00640 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b3da20 0x55e083b3b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc58c00640 1 -- 172.21.15.146:0/2754544493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 msgr2=0x55e083b3afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc58c00640 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b3afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc58c00640 1 -- 172.21.15.146:0/2754544493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e083a50920 con 0x55e083b32a30 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc58c00640 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b32a30 0x55e083b3aa80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800c6a0 tx=0x7fbc4809d400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc3c8ec886954171 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:13.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc3a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc3a000640 1 -- 172.21.15.146:0/2754544493 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc480ce930 con 0x55e083b32a30 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc3a000640 1 -- 172.21.15.146:0/2754544493 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbc480bd070 con 0x55e083b32a30 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc40005500 con 0x55e083b32a30 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc3a000640 1 -- 172.21.15.146:0/2754544493 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbc480bb050 con 0x55e083b32a30 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc5e44c740 10 client.26209 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.151+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e0838747d0 con 0x55e083b32a30 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc3a000640 1 -- 172.21.15.146:0/2754544493 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc40005500 con 0x55e083b32a30 2024-04-24T08:25:13.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc3a000640 1 -- 172.21.15.146:0/2754544493 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fbc4809c150 con 0x55e083b32a30 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc5e44c740 10 client.26209 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc5e44c740 10 client.26209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc5e44c740 20 client.26209 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083b5ee30 0x55e083b61220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc5e44c740 4 client.26209 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.152+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e083aa5c10 con 0x55e083b5ee30 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.153+0000 7fbc53e00640 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083b5ee30 0x55e083b61220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.153+0000 7fbc53e00640 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083b5ee30 0x55e083b61220 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=570849a7661760ca server_cookie=3fc778b0826c3ae3 in_seq=0 out_seq=0 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.153+0000 7fbc3a000640 10 client.26209 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc3a000640 1 -- 172.21.15.146:0/2754544493 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e083aa5c10 con 0x55e083b5ee30 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc3a000640 10 client.26209 handle_command_reply: tid=0 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 client.26209 shutdown 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b32a30 msgr2=0x55e083b3aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b32a30 0x55e083b3aa80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800c6a0 tx=0x7fbc4809d400 comp rx=0 tx=0).stop 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 shutdown_connections 2024-04-24T08:25:13.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083b5ee30 0x55e083b61220 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b3da20 0x55e083b3b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b3afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/2754544493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b32a30 0x55e083b3aa80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 >> 172.21.15.146:0/2754544493 conn(0x55e083b2ed50 msgr2=0x55e083b2dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc3a000640 0 client.26209 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 shutdown_connections 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.154+0000 7fbc5e44c740 1 -- 172.21.15.146:0/2754544493 wait complete. 2024-04-24T08:25:13.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.155+0000 7fbc5e44c740 20 client.26209 trim_cache size 0 max 16384 2024-04-24T08:25:13.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.173+0000 7fbc5e44c740 1 Processor -- start 2024-04-24T08:25:13.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.173+0000 7fbc5e44c740 1 -- start start 2024-04-24T08:25:13.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b49670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b33350 0x55e083b2de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b3da20 0x55e083b66ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbc541ae200 con 0x55e083b3da20 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc54073e30 con 0x55e083b32a30 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbc54071c80 con 0x55e083b33350 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b33350 0x55e083b2de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc58c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b49670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b33350 0x55e083b2de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:50916/0 (socket says 172.21.15.146:50916) 2024-04-24T08:25:13.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 -- 172.21.15.146:0/1765577111 learned_addr learned my addr 172.21.15.146:0/1765577111 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:13.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc59600640 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b3da20 0x55e083b66ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 -- 172.21.15.146:0/1765577111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 msgr2=0x55e083b49670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b49670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 -- 172.21.15.146:0/1765577111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b3da20 msgr2=0x55e083b66ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b3da20 0x55e083b66ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.174+0000 7fbc53e00640 1 -- 172.21.15.146:0/1765577111 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0837aa3f0 con 0x55e083b33350 2024-04-24T08:25:13.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.175+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b33350 0x55e083b2de50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009900 tx=0x7fbc40004a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e43b5ca7d9963b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:13.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.175+0000 7fbc45200640 1 -- 172.21.15.146:0/1765577111 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40040040 con 0x55e083b33350 2024-04-24T08:25:13.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.175+0000 7fbc45200640 1 -- 172.21.15.146:0/1765577111 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc40031870 con 0x55e083b33350 2024-04-24T08:25:13.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.175+0000 7fbc45200640 1 -- 172.21.15.146:0/1765577111 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40035040 con 0x55e083b33350 2024-04-24T08:25:13.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1765577111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b33350 msgr2=0x55e083b2de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b33350 0x55e083b2de50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009900 tx=0x7fbc40004a90 comp rx=0 tx=0).stop 2024-04-24T08:25:13.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1765577111 shutdown_connections 2024-04-24T08:25:13.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b33350 0x55e083b2de50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b3da20 0x55e083b66ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1765577111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b49670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1765577111 >> 172.21.15.146:0/1765577111 conn(0x55e083b2ed50 msgr2=0x55e083b2d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1765577111 shutdown_connections 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1765577111 wait complete. 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 -- start start 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.176+0000 7fbc5e44c740 1 Processor -- start 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b37860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b37da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc5e44c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b382e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbc540722f0 con 0x55e083b33350 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbc54073760 con 0x55e083b32a30 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc5e44c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbc540729c0 con 0x55e083b3da20 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc5e44c740 10 client.0 _ll_register_callbacks cb 0x7fbc4809c840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc58c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b37860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc58c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b37860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:59692/0 (socket says 172.21.15.146:59692) 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc58c00640 1 -- 172.21.15.146:0/1037773871 learned_addr learned my addr 172.21.15.146:0/1037773871 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:13.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b37da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.177+0000 7fbc59600640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc58c00640 1 -- 172.21.15.146:0/1037773871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 msgr2=0x55e083b382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc58c00640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b382e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc58c00640 1 -- 172.21.15.146:0/1037773871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 msgr2=0x55e083b37da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc58c00640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b37da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc58c00640 1 -- 172.21.15.146:0/1037773871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e08376a0f0 con 0x55e083b32a30 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b37da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc58c00640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b37860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbc4809c560 tx=0x7fbc48018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72de5098778313a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:13.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc3a000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc3a000640 1 -- 172.21.15.146:0/1037773871 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc480a9070 con 0x55e083b32a30 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc3a000640 1 -- 172.21.15.146:0/1037773871 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fbc480a5bc0 con 0x55e083b32a30 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc40031870 con 0x55e083b32a30 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.178+0000 7fbc3a000640 1 -- 172.21.15.146:0/1037773871 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbc4809cad0 con 0x55e083b32a30 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 10 client.26234 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e083a5d810 con 0x55e083b32a30 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc3a000640 1 -- 172.21.15.146:0/1037773871 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc40031870 con 0x55e083b32a30 2024-04-24T08:25:13.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc3a000640 1 -- 172.21.15.146:0/1037773871 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fbc48098c40 con 0x55e083b32a30 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 10 client.26234 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 10 client.26234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 20 client.26234 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083bc59d0 0x55e083bc7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 4 client.26234 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e083a9fa80 con 0x55e083bc59d0 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.179+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083bc59d0 0x55e083bc7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc53e00640 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083bc59d0 0x55e083bc7df0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d94ee90235cb55 server_cookie=c9c06b5fc896132 in_seq=0 out_seq=0 2024-04-24T08:25:13.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc3a000640 10 client.26234 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc3a000640 1 -- 172.21.15.146:0/1037773871 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e083a9fa80 con 0x55e083bc59d0 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc3a000640 10 client.26234 handle_command_reply: tid=0 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 client.26234 shutdown 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 msgr2=0x55e083b37860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b37860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbc4809c560 tx=0x7fbc48018420 comp rx=0 tx=0).stop 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 shutdown_connections 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55e083bc59d0 0x55e083bc7df0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e083b3da20 0x55e083b382e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55e083b33350 0x55e083b37da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc3a000640 0 client.26234 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 --2- 172.21.15.146:0/1037773871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55e083b32a30 0x55e083b37860 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 >> 172.21.15.146:0/1037773871 conn(0x55e083b2ed50 msgr2=0x55e083b3b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 shutdown_connections 2024-04-24T08:25:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.180+0000 7fbc5e44c740 1 -- 172.21.15.146:0/1037773871 wait complete. 2024-04-24T08:25:13.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5e44c740 20 client.26234 trim_cache size 0 max 16384 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "78e4f0eb-1288-4f2f-bee4-f4648caea4fc": { 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "78e4f0eb-1288-4f2f-bee4-f4648caea4fc", 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:25:13.190 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:25:13.191 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:25:13.191 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbc3c0a3e20 msgr2=0x7fbc3c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbc3c0a3e20 0x7fbc3c0a62e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c005fd0 tx=0x7fbc4c009500 comp rx=0 tx=0).stop 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 msgr2=0x7fbc5410feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 0x7fbc5410feb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800e9f0 tx=0x7fbc4800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 shutdown_connections 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbc3c0a3e20 0x7fbc3c0a62e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc5410e840 0x7fbc5410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbc541060d0 0x7fbc5410e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 --2- 172.21.15.146:0/880294285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc54101000 0x7fbc5410feb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 >> 172.21.15.146:0/880294285 conn(0x7fbc540fc880 msgr2=0x7fbc540fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 shutdown_connections 2024-04-24T08:25:13.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:13.181+0000 7fbc5a000640 1 -- 172.21.15.146:0/880294285 wait complete. 2024-04-24T08:25:13.199 INFO:tasks.cephfs.filesystem:scrub status for tag:78e4f0eb-1288-4f2f-bee4-f4648caea4fc - {'path': '/', 'tag': '78e4f0eb-1288-4f2f-bee4-f4648caea4fc', 'options': 'recursive,force'} 2024-04-24T08:25:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:25:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:25:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:25:13.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:13.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:25:13.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:13.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:13 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:13.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:25:13.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:13.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:25:13.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:25:13.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:13.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:25:13.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:13.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:13 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:14.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:25:14.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:14.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:25:14.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:25:14.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:14.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:25:14.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:14.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:13 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:14 smithi153 ceph-mon[32746]: pgmap v124: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:14.966 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:14 smithi177 ceph-mon[33046]: pgmap v124: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:14.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:14 smithi146 ceph-mon[27805]: pgmap v124: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:25:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:16 smithi153 ceph-mon[32746]: pgmap v125: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:25:16.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:16 smithi146 ceph-mon[27805]: pgmap v125: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:25:17.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:16 smithi177 ceph-mon[33046]: pgmap v125: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:25:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:18 smithi153 ceph-mon[32746]: pgmap v126: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:25:18.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:18 smithi146 ceph-mon[27805]: pgmap v126: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:25:19.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:18 smithi177 ceph-mon[33046]: pgmap v126: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:25:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:20 smithi153 ceph-mon[32746]: pgmap v127: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:25:20.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:20 smithi146 ceph-mon[27805]: pgmap v127: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:25:21.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:20 smithi177 ceph-mon[33046]: pgmap v127: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:25:22.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:22 smithi153 ceph-mon[32746]: pgmap v128: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:25:22.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:22 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:25:22.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:22 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:25:22.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:22 smithi146 ceph-mon[27805]: pgmap v128: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:25:22.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:22 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:25:22.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:22 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:25:23.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:22 smithi177 ceph-mon[33046]: pgmap v128: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:25:23.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:22 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:25:23.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:22 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:25:24.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:24 smithi153 ceph-mon[32746]: pgmap v129: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.8 KiB/s wr, 8 op/s 2024-04-24T08:25:24.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:24 smithi146 ceph-mon[27805]: pgmap v129: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.8 KiB/s wr, 8 op/s 2024-04-24T08:25:25.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:24 smithi177 ceph-mon[33046]: pgmap v129: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.8 KiB/s wr, 8 op/s 2024-04-24T08:25:25.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:25 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:25:26.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:25 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:25:26.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:25 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:25:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:26 smithi146 ceph-mon[27805]: pgmap v130: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-04-24T08:25:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:26 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:25:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:26 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:25:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:26 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:25:27.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:26 smithi177 ceph-mon[33046]: pgmap v130: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-04-24T08:25:27.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:26 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:25:27.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:26 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:25:27.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:26 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:25:27.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:26 smithi153 ceph-mon[32746]: pgmap v130: 97 pgs: 97 active+clean; 74 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-04-24T08:25:27.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:26 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:25:27.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:26 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:25:27.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:26 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:25:28.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:28 smithi146 ceph-mon[27805]: pgmap v131: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:25:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:28 smithi177 ceph-mon[33046]: pgmap v131: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:25:29.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:28 smithi153 ceph-mon[32746]: pgmap v131: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:25:30.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:30 smithi146 ceph-mon[27805]: pgmap v132: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:25:31.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:30 smithi177 ceph-mon[33046]: pgmap v132: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:25:31.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:30 smithi153 ceph-mon[32746]: pgmap v132: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:25:32.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:32 smithi146 ceph-mon[27805]: pgmap v133: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:25:33.013 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:32 smithi153 ceph-mon[32746]: pgmap v133: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:25:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:32 smithi177 ceph-mon[33046]: pgmap v133: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:25:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:34 smithi146 ceph-mon[27805]: pgmap v134: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 9 op/s 2024-04-24T08:25:35.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:34 smithi177 ceph-mon[33046]: pgmap v134: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 9 op/s 2024-04-24T08:25:35.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:34 smithi153 ceph-mon[32746]: pgmap v134: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 9 op/s 2024-04-24T08:25:37.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:36 smithi177 ceph-mon[33046]: pgmap v135: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 5 op/s 2024-04-24T08:25:37.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:36 smithi153 ceph-mon[32746]: pgmap v135: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 5 op/s 2024-04-24T08:25:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:36 smithi146 ceph-mon[27805]: pgmap v135: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 5 op/s 2024-04-24T08:25:39.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:38 smithi177 ceph-mon[33046]: pgmap v136: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2024-04-24T08:25:39.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:38 smithi153 ceph-mon[32746]: pgmap v136: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2024-04-24T08:25:39.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:38 smithi146 ceph-mon[27805]: pgmap v136: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2024-04-24T08:25:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:40 smithi177 ceph-mon[33046]: pgmap v137: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-04-24T08:25:41.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:40 smithi153 ceph-mon[32746]: pgmap v137: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-04-24T08:25:41.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:40 smithi146 ceph-mon[27805]: pgmap v137: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-04-24T08:25:43.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:42 smithi177 ceph-mon[33046]: pgmap v138: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-04-24T08:25:43.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:25:43.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:25:43.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:42 smithi153 ceph-mon[32746]: pgmap v138: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-04-24T08:25:43.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:42 smithi146 ceph-mon[27805]: pgmap v138: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-04-24T08:25:43.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 -- 172.21.15.146:0/3811745805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee181071f0 msgr2=0x7fee1810edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 --2- 172.21.15.146:0/3811745805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee181071f0 0x7fee1810edc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fee10009ca0 tx=0x7fee1002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 -- 172.21.15.146:0/3811745805 shutdown_connections 2024-04-24T08:25:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 --2- 172.21.15.146:0/3811745805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee181071f0 0x7fee1810edc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 --2- 172.21.15.146:0/3811745805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee18102ad0 0x7fee18106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 --2- 172.21.15.146:0/3811745805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee18102120 0x7fee18102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 -- 172.21.15.146:0/3811745805 >> 172.21.15.146:0/3811745805 conn(0x7fee180fd9a0 msgr2=0x7fee180ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:43.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 -- 172.21.15.146:0/3811745805 shutdown_connections 2024-04-24T08:25:43.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.424+0000 7fee1ee00640 1 -- 172.21.15.146:0/3811745805 wait complete. 2024-04-24T08:25:43.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.425+0000 7fee1ee00640 1 Processor -- start 2024-04-24T08:25:43.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.425+0000 7fee1ee00640 1 -- start start 2024-04-24T08:25:43.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1ee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee18102120 0x7fee181a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1ee00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee181071f0 0x7fee181a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee1819a7d0 0x7fee1819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fee18112290 con 0x7fee1819a7d0 2024-04-24T08:25:43.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1ee00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee18112110 con 0x7fee181071f0 2024-04-24T08:25:43.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1ee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee18112410 con 0x7fee18102120 2024-04-24T08:25:43.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee1819a7d0 0x7fee1819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee18102120 0x7fee181a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee18102120 0x7fee181a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48646/0 (socket says 172.21.15.146:48646) 2024-04-24T08:25:43.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1da00640 1 -- 172.21.15.146:0/1112191288 learned_addr learned my addr 172.21.15.146:0/1112191288 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:43.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.426+0000 7fee1d000640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee181071f0 0x7fee181a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee1da00640 1 -- 172.21.15.146:0/1112191288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee181071f0 msgr2=0x7fee181a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee1da00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee181071f0 0x7fee181a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee1da00640 1 -- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee1819a7d0 msgr2=0x7fee1819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee1da00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee1819a7d0 0x7fee1819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee1da00640 1 -- 172.21.15.146:0/1112191288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1819b360 con 0x7fee18102120 2024-04-24T08:25:43.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee1da00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee18102120 0x7fee181a0610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e9f0 tx=0x7fee0c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee16000640 1 -- 172.21.15.146:0/1112191288 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c00cd80 con 0x7fee18102120 2024-04-24T08:25:43.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee16000640 1 -- 172.21.15.146:0/1112191288 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0c010910 con 0x7fee18102120 2024-04-24T08:25:43.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee16000640 1 -- 172.21.15.146:0/1112191288 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c018990 con 0x7fee18102120 2024-04-24T08:25:43.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.427+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee1819b650 con 0x7fee18102120 2024-04-24T08:25:43.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.428+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee181a7790 con 0x7fee18102120 2024-04-24T08:25:43.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 Processor -- start 2024-04-24T08:25:43.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 -- start start 2024-04-24T08:25:43.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a8378ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a837974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a83797a20 0x561a83799e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561a836f21f0 con 0x561a8378d350 2024-04-24T08:25:43.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a83698350 con 0x561a83797a20 2024-04-24T08:25:43.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.431+0000 7fee231be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561a836ee1b0 con 0x561a8378ca30 2024-04-24T08:25:43.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee16000640 1 -- 172.21.15.146:0/1112191288 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fee0c010ac0 con 0x7fee18102120 2024-04-24T08:25:43.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a8378ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a8378ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48652/0 (socket says 172.21.15.146:48652) 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 -- 172.21.15.146:0/1104119692 learned_addr learned my addr 172.21.15.146:0/1104119692 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1d000640 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a83797a20 0x561a83799e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 -- 172.21.15.146:0/1104119692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a83797a20 msgr2=0x561a83799e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a83797a20 0x561a83799e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 -- 172.21.15.146:0/1104119692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 msgr2=0x561a837974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a837974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 -- 172.21.15.146:0/1104119692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a836ecf50 con 0x561a8378ca30 2024-04-24T08:25:43.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.432+0000 7fee1e400640 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a8378ce10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fee10009ca0 tx=0x7fee10009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb5afddb183835db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.433+0000 7fee0aa00640 1 -- 172.21.15.146:0/1104119692 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10031db0 con 0x561a8378ca30 2024-04-24T08:25:43.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.433+0000 7fee0aa00640 1 -- 172.21.15.146:0/1104119692 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee10036040 con 0x561a8378ca30 2024-04-24T08:25:43.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.433+0000 7fee0aa00640 1 -- 172.21.15.146:0/1104119692 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee100043c0 con 0x561a8378ca30 2024-04-24T08:25:43.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.433+0000 7fee16000640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fee000a3d20 0x7fee000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 -- 172.21.15.146:0/1104119692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 msgr2=0x561a8378ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a8378ce10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fee10009ca0 tx=0x7fee10009800 comp rx=0 tx=0).stop 2024-04-24T08:25:43.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee16000640 1 -- 172.21.15.146:0/1112191288 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fee0c0d0b20 con 0x7fee18102120 2024-04-24T08:25:43.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee1da00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fee000a3d20 0x7fee000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 -- 172.21.15.146:0/1104119692 shutdown_connections 2024-04-24T08:25:43.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a83797a20 0x561a83799e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a837974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 --2- 172.21.15.146:0/1104119692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a8378ce10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 -- 172.21.15.146:0/1104119692 >> 172.21.15.146:0/1104119692 conn(0x561a83788d50 msgr2=0x561a83789130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:43.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 -- 172.21.15.146:0/1104119692 shutdown_connections 2024-04-24T08:25:43.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 -- 172.21.15.146:0/1104119692 wait complete. 2024-04-24T08:25:43.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee231be740 1 -- start start 2024-04-24T08:25:43.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.434+0000 7fee1da00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fee000a3d20 0x7fee000a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x561a8378dff0 tx=0x7fee0c0d0df0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 1 Processor -- start 2024-04-24T08:25:43.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a83794940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378d350 0x561a83794e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a837953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561a8379a630 con 0x561a83797a20 2024-04-24T08:25:43.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561a836ab690 con 0x561a8378ca30 2024-04-24T08:25:43.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561a8379a7b0 con 0x561a8378d350 2024-04-24T08:25:43.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee231be740 10 client.0 _ll_register_callbacks cb 0x561a836e7650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:43.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee1d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378d350 0x561a83794e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.435+0000 7fee1e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a83794940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378d350 0x561a83794e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48660/0 (socket says 172.21.15.146:48660) 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 -- 172.21.15.146:0/2371531061 learned_addr learned my addr 172.21.15.146:0/2371531061 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1e400640 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a837953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 -- 172.21.15.146:0/2371531061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 msgr2=0x561a83794940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a83794940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 -- 172.21.15.146:0/2371531061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 msgr2=0x561a837953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a837953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 -- 172.21.15.146:0/2371531061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a836aa920 con 0x561a8378d350 2024-04-24T08:25:43.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fee1d000640 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378d350 0x561a83794e80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fee0400b9f0 tx=0x7fee0400beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c1c4d709d268 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fedff400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:43.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.436+0000 7fedff400640 1 -- 172.21.15.146:0/2371531061 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04003dc0 con 0x561a8378d350 2024-04-24T08:25:43.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee10036040 con 0x561a8378d350 2024-04-24T08:25:43.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fedff400640 1 -- 172.21.15.146:0/2371531061 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fee04005020 con 0x561a8378d350 2024-04-24T08:25:43.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fedff400640 1 -- 172.21.15.146:0/2371531061 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fee04005220 con 0x561a8378d350 2024-04-24T08:25:43.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 10 client.26227 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:43.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561a834ce7d0 con 0x561a8378d350 2024-04-24T08:25:43.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fedff400640 1 -- 172.21.15.146:0/2371531061 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee10036040 con 0x561a8378d350 2024-04-24T08:25:43.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fedff400640 1 -- 172.21.15.146:0/2371531061 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fee0401b590 con 0x561a8378d350 2024-04-24T08:25:43.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 10 client.26227 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:43.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 10 client.26227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:43.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 20 client.26227 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:43.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a837a7c90 0x561a837aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 4 client.26227 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:43.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.437+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a836ffc10 con 0x561a837a7c90 2024-04-24T08:25:43.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.438+0000 7fee1e400640 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a837a7c90 0x561a837aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.438+0000 7fee1e400640 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a837a7c90 0x561a837aa0b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9466717551ae503b server_cookie=588741744d98ea95 in_seq=0 out_seq=0 2024-04-24T08:25:43.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.438+0000 7fedff400640 10 client.26227 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:43.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.439+0000 7fedff400640 1 -- 172.21.15.146:0/2371531061 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a836ffc10 con 0x561a837a7c90 2024-04-24T08:25:43.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.439+0000 7fedff400640 10 client.26227 handle_command_reply: tid=0 2024-04-24T08:25:43.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.439+0000 7fee231be740 1 client.26227 shutdown 2024-04-24T08:25:43.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378d350 msgr2=0x561a83794e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378d350 0x561a83794e80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fee0400b9f0 tx=0x7fee0400beb0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 shutdown_connections 2024-04-24T08:25:43.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a837a7c90 0x561a837aa0b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378d350 0x561a83794e80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a837953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fedff400640 0 client.26227 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:43.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 --2- 172.21.15.146:0/2371531061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a83794940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 >> 172.21.15.146:0/2371531061 conn(0x561a83788d50 msgr2=0x561a83787fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:43.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 shutdown_connections 2024-04-24T08:25:43.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 1 -- 172.21.15.146:0/2371531061 wait complete. 2024-04-24T08:25:43.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.440+0000 7fee231be740 20 client.26227 trim_cache size 0 max 16384 2024-04-24T08:25:43.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 Processor -- start 2024-04-24T08:25:43.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 -- start start 2024-04-24T08:25:43.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a837a3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378d350 0x561a83793b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a83795010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fee181b25d0 con 0x561a83797a20 2024-04-24T08:25:43.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee181afea0 con 0x561a8378d350 2024-04-24T08:25:43.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.459+0000 7fee231be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee181adcf0 con 0x561a8378ca30 2024-04-24T08:25:43.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a83795010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a837a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a837a3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48668/0 (socket says 172.21.15.146:48668) 2024-04-24T08:25:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 -- 172.21.15.146:0/1999786931 learned_addr learned my addr 172.21.15.146:0/1999786931 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1d000640 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378d350 0x561a83793b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 -- 172.21.15.146:0/1999786931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378d350 msgr2=0x561a83793b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378d350 0x561a83793b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 -- 172.21.15.146:0/1999786931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 msgr2=0x561a83795010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a83795010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 -- 172.21.15.146:0/1999786931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a8379a180 con 0x561a8378ca30 2024-04-24T08:25:43.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.460+0000 7fee1e400640 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a837a3520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fee10000f80 tx=0x7fee10031ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64eaf2ad6641990 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee0a000640 1 -- 172.21.15.146:0/1999786931 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10031bb0 con 0x561a8378ca30 2024-04-24T08:25:43.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee0a000640 1 -- 172.21.15.146:0/1999786931 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee10036040 con 0x561a8378ca30 2024-04-24T08:25:43.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee0a000640 1 -- 172.21.15.146:0/1999786931 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10004660 con 0x561a8378ca30 2024-04-24T08:25:43.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 -- 172.21.15.146:0/1999786931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 msgr2=0x561a837a3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a837a3520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fee10000f80 tx=0x7fee10031ad0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 -- 172.21.15.146:0/1999786931 shutdown_connections 2024-04-24T08:25:43.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a83797a20 0x561a83795010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378d350 0x561a83793b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 --2- 172.21.15.146:0/1999786931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a8378ca30 0x561a837a3520 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 -- 172.21.15.146:0/1999786931 >> 172.21.15.146:0/1999786931 conn(0x561a83788d50 msgr2=0x561a83794900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:43.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 -- 172.21.15.146:0/1999786931 shutdown_connections 2024-04-24T08:25:43.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.461+0000 7fee231be740 1 -- 172.21.15.146:0/1999786931 wait complete. 2024-04-24T08:25:43.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 -- start start 2024-04-24T08:25:43.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 Processor -- start 2024-04-24T08:25:43.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a8379fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a83791350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a83797a20 0x561a83791890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fee181ae360 con 0x561a8378d350 2024-04-24T08:25:43.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee181af7d0 con 0x561a8378ca30 2024-04-24T08:25:43.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.462+0000 7fee231be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee181aea30 con 0x561a83797a20 2024-04-24T08:25:43.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee231be740 10 client.0 _ll_register_callbacks cb 0x7fee04004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a83791350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a8379fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a83791350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35592/0 (socket says 172.21.15.146:35592) 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 -- 172.21.15.146:0/2212841089 learned_addr learned my addr 172.21.15.146:0/2212841089 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1e400640 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a83797a20 0x561a83791890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 -- 172.21.15.146:0/2212841089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a83797a20 msgr2=0x561a83791890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a83797a20 0x561a83791890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 -- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 msgr2=0x561a8379fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a8379fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 -- 172.21.15.146:0/2212841089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a834043f0 con 0x561a8378d350 2024-04-24T08:25:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1e400640 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a8379fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fee1d000640 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a83791350 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x561a83795f10 tx=0x7fee0401cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab1a5353ee69c99 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.463+0000 7fedff400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:25:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fedff400640 1 -- 172.21.15.146:0/2212841089 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04033020 con 0x561a8378d350 2024-04-24T08:25:43.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fedff400640 1 -- 172.21.15.146:0/2212841089 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fee040055d0 con 0x561a8378d350 2024-04-24T08:25:43.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee10036040 con 0x561a8378d350 2024-04-24T08:25:43.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fedff400640 1 -- 172.21.15.146:0/2212841089 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fee0403e050 con 0x561a8378d350 2024-04-24T08:25:43.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fee231be740 10 client.16104 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:43.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561a833c40f0 con 0x561a8378d350 2024-04-24T08:25:43.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fedff400640 1 -- 172.21.15.146:0/2212841089 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee10036040 con 0x561a8378d350 2024-04-24T08:25:43.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.464+0000 7fedff400640 1 -- 172.21.15.146:0/2212841089 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fee0402fbb0 con 0x561a8378d350 2024-04-24T08:25:43.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee231be740 10 client.16104 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee231be740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee231be740 20 client.16104 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee231be740 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a8380e5e0 0x561a83810a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee231be740 4 client.16104 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561a836f9a80 con 0x561a8380e5e0 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee1e400640 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a8380e5e0 0x561a83810a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fedff400640 10 client.16104 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:43.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.465+0000 7fee1e400640 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a8380e5e0 0x561a83810a00 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4279391f3f7b7e6 server_cookie=e93c15702a5a43c0 in_seq=0 out_seq=0 2024-04-24T08:25:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fedff400640 1 -- 172.21.15.146:0/2212841089 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561a836f9a80 con 0x561a8380e5e0 2024-04-24T08:25:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fedff400640 10 client.16104 handle_command_reply: tid=0 2024-04-24T08:25:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 client.16104 shutdown 2024-04-24T08:25:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 msgr2=0x561a83791350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a83791350 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x561a83795f10 tx=0x7fee0401cf00 comp rx=0 tx=0).stop 2024-04-24T08:25:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 shutdown_connections 2024-04-24T08:25:43.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561a8380e5e0 0x561a83810a00 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561a83797a20 0x561a83791890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561a8378d350 0x561a83791350 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 --2- 172.21.15.146:0/2212841089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561a8378ca30 0x561a8379fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fedff400640 0 client.16104 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 >> 172.21.15.146:0/2212841089 conn(0x561a83788d50 msgr2=0x561a837939e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 shutdown_connections 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 1 -- 172.21.15.146:0/2212841089 wait complete. 2024-04-24T08:25:43.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.466+0000 7fee231be740 20 client.16104 trim_cache size 0 max 16384 2024-04-24T08:25:43.478 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:25:43.478 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:25:43.478 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:25:43.479 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:25:43.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fee000a3d20 msgr2=0x7fee000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fee000a3d20 0x7fee000a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x561a8378dff0 tx=0x7fee0c0d0df0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee18102120 msgr2=0x7fee181a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee18102120 0x7fee181a0610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e9f0 tx=0x7fee0c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 shutdown_connections 2024-04-24T08:25:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fee000a3d20 0x7fee000a61e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee1819a7d0 0x7fee1819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee181071f0 0x7fee181a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 --2- 172.21.15.146:0/1112191288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee18102120 0x7fee181a0610 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 >> 172.21.15.146:0/1112191288 conn(0x7fee180fd9a0 msgr2=0x7fee1810d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 shutdown_connections 2024-04-24T08:25:43.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.467+0000 7fee1ee00640 1 -- 172.21.15.146:0/1112191288 wait complete. 2024-04-24T08:25:43.487 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:25:43.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:25:43.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 -- 172.21.15.146:0/1448486686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104c10 msgr2=0x7f6a68108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 --2- 172.21.15.146:0/1448486686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104c10 0x7f6a68108c50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009900 tx=0x7f6a5802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 -- 172.21.15.146:0/1448486686 shutdown_connections 2024-04-24T08:25:43.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 --2- 172.21.15.146:0/1448486686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a68109330 0x7f6a68110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 --2- 172.21.15.146:0/1448486686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104c10 0x7f6a68108c50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 --2- 172.21.15.146:0/1448486686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68104260 0x7f6a68104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 -- 172.21.15.146:0/1448486686 >> 172.21.15.146:0/1448486686 conn(0x7f6a680ffaa0 msgr2=0x7f6a68101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:43.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 -- 172.21.15.146:0/1448486686 shutdown_connections 2024-04-24T08:25:43.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.735+0000 7f6a67200640 1 -- 172.21.15.146:0/1448486686 wait complete. 2024-04-24T08:25:43.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.736+0000 7f6a67200640 1 Processor -- start 2024-04-24T08:25:43.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.736+0000 7f6a67200640 1 -- start start 2024-04-24T08:25:43.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a67200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104260 0x7f6a681a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a67200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68109330 0x7f6a681a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a67200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a6819a730 0x7f6a6819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a67200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68114410 con 0x7f6a68104260 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a67200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68114290 con 0x7f6a68109330 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a67200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68114590 con 0x7f6a6819a730 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104260 0x7f6a681a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68109330 0x7f6a681a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104260 0x7f6a681a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35608/0 (socket says 172.21.15.146:35608) 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 -- 172.21.15.146:0/1725352707 learned_addr learned my addr 172.21.15.146:0/1725352707 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a66800640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a6819a730 0x7f6a6819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 -- 172.21.15.146:0/1725352707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a6819a730 msgr2=0x7f6a6819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a6819a730 0x7f6a6819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 -- 172.21.15.146:0/1725352707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68109330 msgr2=0x7f6a681a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68109330 0x7f6a681a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.737+0000 7f6a65e00640 1 -- 172.21.15.146:0/1725352707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6819b330 con 0x7f6a68104260 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.738+0000 7f6a65e00640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104260 0x7f6a681a0620 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000ea60 tx=0x7f6a5000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.738+0000 7f6a5ea00640 1 -- 172.21.15.146:0/1725352707 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5000cde0 con 0x7f6a68104260 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.738+0000 7f6a5ea00640 1 -- 172.21.15.146:0/1725352707 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a500035f0 con 0x7f6a68104260 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.738+0000 7f6a5ea00640 1 -- 172.21.15.146:0/1725352707 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50010670 con 0x7f6a68104260 2024-04-24T08:25:43.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.738+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a681a73c0 con 0x7f6a68104260 2024-04-24T08:25:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.738+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a681a7770 con 0x7f6a68104260 2024-04-24T08:25:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.740+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a68105730 con 0x7f6a68104260 2024-04-24T08:25:43.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.741+0000 7f6a5ea00640 1 -- 172.21.15.146:0/1725352707 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f6a500037a0 con 0x7f6a68104260 2024-04-24T08:25:43.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.743+0000 7f6a5ea00640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a4c0a3d20 0x7f6a4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:43.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.743+0000 7f6a5ea00640 1 -- 172.21.15.146:0/1725352707 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6a500d0b10 con 0x7f6a68104260 2024-04-24T08:25:43.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.743+0000 7f6a65400640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a4c0a3d20 0x7f6a4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:43.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.744+0000 7f6a65400640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a4c0a3d20 0x7f6a4c0a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a58006010 tx=0x7f6a58000bc0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:43.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.747+0000 7f6a5ea00640 1 -- 172.21.15.146:0/1725352707 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a5009d160 con 0x7f6a68104260 2024-04-24T08:25:43.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.993+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a6810c5c0 con 0x7f6a68104260 2024-04-24T08:25:43.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.995+0000 7f6a5ea00640 1 -- 172.21.15.146:0/1725352707 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11892 (secure 0 0 0) 0x7f6a5009ca60 con 0x7f6a68104260 2024-04-24T08:25:43.997 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:25:43.998 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:23:07.211437+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24568,24569,24553]},"id":1}]} 2024-04-24T08:25:43.998 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 25 2024-04-24T08:25:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a4c0a3d20 msgr2=0x7f6a4c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a4c0a3d20 0x7f6a4c0a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a58006010 tx=0x7f6a58000bc0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104260 msgr2=0x7f6a681a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104260 0x7f6a681a0620 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6a5000ea60 tx=0x7f6a5000ef20 comp rx=0 tx=0).stop 2024-04-24T08:25:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 shutdown_connections 2024-04-24T08:25:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a4c0a3d20 0x7f6a4c0a61e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a6819a730 0x7f6a6819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a68109330 0x7f6a681a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 --2- 172.21.15.146:0/1725352707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a68104260 0x7f6a681a0620 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 >> 172.21.15.146:0/1725352707 conn(0x7f6a680ffaa0 msgr2=0x7f6a68101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 shutdown_connections 2024-04-24T08:25:44.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:43.998+0000 7f6a67200640 1 -- 172.21.15.146:0/1725352707 wait complete. 2024-04-24T08:25:44.019 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:25:44.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 -- 172.21.15.146:0/960225984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 msgr2=0x7fb39c112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 --2- 172.21.15.146:0/960225984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c112800 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb38c009900 tx=0x7fb38c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 -- 172.21.15.146:0/960225984 shutdown_connections 2024-04-24T08:25:44.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 --2- 172.21.15.146:0/960225984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c112d40 0x7fb39c115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 --2- 172.21.15.146:0/960225984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c112800 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 --2- 172.21.15.146:0/960225984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39c076530 0x7fb39c076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 -- 172.21.15.146:0/960225984 >> 172.21.15.146:0/960225984 conn(0x7fb39c0ffb10 msgr2=0x7fb39c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 -- 172.21.15.146:0/960225984 shutdown_connections 2024-04-24T08:25:44.223 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.220+0000 7fb3a3200640 1 -- 172.21.15.146:0/960225984 wait complete. 2024-04-24T08:25:44.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 Processor -- start 2024-04-24T08:25:44.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 -- start start 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39c076530 0x7fb39c19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c19f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c112d40 0x7fb39c1a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1188b0 con 0x7fb39c076e50 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c118730 con 0x7fb39c112d40 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.221+0000 7fb3a3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c118a30 con 0x7fb39c076530 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c19f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c112d40 0x7fb39c1a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c19f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35624/0 (socket says 172.21.15.146:35624) 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a1400640 1 -- 172.21.15.146:0/2716046059 learned_addr learned my addr 172.21.15.146:0/2716046059 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39c076530 0x7fb39c19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a2800640 1 -- 172.21.15.146:0/2716046059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39c076530 msgr2=0x7fb39c19ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a2800640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39c076530 0x7fb39c19ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a2800640 1 -- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 msgr2=0x7fb39c19f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a2800640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c19f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a2800640 1 -- 172.21.15.146:0/2716046059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c1a3dd0 con 0x7fb39c112d40 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c19f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.227 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.222+0000 7fb3a2800640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c112d40 0x7fb39c1a36a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb398005fd0 tx=0x7fb39800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.223+0000 7fb396a00640 1 -- 172.21.15.146:0/2716046059 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398003b70 con 0x7fb39c112d40 2024-04-24T08:25:44.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.223+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c1a40c0 con 0x7fb39c112d40 2024-04-24T08:25:44.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.223+0000 7fb396a00640 1 -- 172.21.15.146:0/2716046059 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb398003d10 con 0x7fb39c112d40 2024-04-24T08:25:44.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.223+0000 7fb396a00640 1 -- 172.21.15.146:0/2716046059 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398010830 con 0x7fb39c112d40 2024-04-24T08:25:44.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.223+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c1ab950 con 0x7fb39c112d40 2024-04-24T08:25:44.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.226+0000 7fb3a75f3740 1 Processor -- start 2024-04-24T08:25:44.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.226+0000 7fb396a00640 1 -- 172.21.15.146:0/2716046059 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fb398004020 con 0x7fb39c112d40 2024-04-24T08:25:44.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.226+0000 7fb3a75f3740 1 -- start start 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d14b40 0x5567f0d14f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d1fb30 0x5567f0d21f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5567f0c7a1d0 con 0x5567f0d1fb30 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5567f0c20450 con 0x5567f0d14b40 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5567f0c76190 con 0x5567f0d15460 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d14b40 0x5567f0d14f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d1fb30 0x5567f0d21f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d1fb30 0x5567f0d21f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35636/0 (socket says 172.21.15.146:35636) 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 -- 172.21.15.146:0/1427869734 learned_addr learned my addr 172.21.15.146:0/1427869734 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 -- 172.21.15.146:0/1427869734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 msgr2=0x5567f0d1f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 -- 172.21.15.146:0/1427869734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d14b40 msgr2=0x5567f0d14f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d14b40 0x5567f0d14f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1400640 1 -- 172.21.15.146:0/1427869734 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f0cdd310 con 0x5567f0d1fb30 2024-04-24T08:25:44.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.227+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.228+0000 7fb3a1400640 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d1fb30 0x5567f0d21f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000c00 tx=0x7fb38c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a77405e6c6e53d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.228+0000 7fb396a00640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb3840a3d20 0x7fb3840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.228+0000 7fb38b000640 1 -- 172.21.15.146:0/1427869734 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c035050 con 0x5567f0d1fb30 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.228+0000 7fb38b000640 1 -- 172.21.15.146:0/1427869734 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb38c003bd0 con 0x5567f0d1fb30 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.228+0000 7fb38b000640 1 -- 172.21.15.146:0/1427869734 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c03b6c0 con 0x5567f0d1fb30 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.228+0000 7fb396a00640 1 -- 172.21.15.146:0/2716046059 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb3980d0d60 con 0x7fb39c112d40 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.228+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb3840a3d20 0x7fb3840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1427869734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d1fb30 msgr2=0x5567f0d21f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d1fb30 0x5567f0d21f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000c00 tx=0x7fb38c005a20 comp rx=0 tx=0).stop 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1427869734 shutdown_connections 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d1fb30 0x5567f0d21f20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1427869734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d14b40 0x5567f0d14f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1427869734 >> 172.21.15.146:0/1427869734 conn(0x5567f0d10e60 msgr2=0x5567f0d11240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb3840a3d20 0x7fb3840a61e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5567f0d16100 tx=0x7fb390005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1427869734 shutdown_connections 2024-04-24T08:25:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1427869734 wait complete. 2024-04-24T08:25:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.229+0000 7fb3a75f3740 1 -- start start 2024-04-24T08:25:44.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 1 Processor -- start 2024-04-24T08:25:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d1f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d15460 0x5567f0d1c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d1fb30 0x5567f0d1cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5567f0d22960 con 0x5567f0d14b40 2024-04-24T08:25:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5567f0c33810 con 0x5567f0d15460 2024-04-24T08:25:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5567f0d22ae0 con 0x5567f0d1fb30 2024-04-24T08:25:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a75f3740 10 client.0 _ll_register_callbacks cb 0x5567f0c01140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.230+0000 7fb3a1e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d1f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d1f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35648/0 (socket says 172.21.15.146:35648) 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d1fb30 0x5567f0d1cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2944254824 learned_addr learned my addr 172.21.15.146:0/2944254824 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d15460 0x5567f0d1c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2944254824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d1fb30 msgr2=0x5567f0d1cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d1fb30 0x5567f0d1cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2944254824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d15460 msgr2=0x5567f0d1c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d15460 0x5567f0d1c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2944254824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f0c443f0 con 0x5567f0d14b40 2024-04-24T08:25:44.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d15460 0x5567f0d1c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d1f5c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb39001e9f0 tx=0x7fb390004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f9177f3b0a86c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.231+0000 7fb37f400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb37f400640 1 -- 172.21.15.146:0/2944254824 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390004020 con 0x5567f0d14b40 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb37f400640 1 -- 172.21.15.146:0/2944254824 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb390018030 con 0x5567f0d14b40 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb38c003bd0 con 0x5567f0d14b40 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb37f400640 1 -- 172.21.15.146:0/2944254824 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb3900041c0 con 0x5567f0d14b40 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb3a75f3740 10 client.16122 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:44.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5567f098c3f0 con 0x5567f0d14b40 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb37f400640 1 -- 172.21.15.146:0/2944254824 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb38c003bd0 con 0x5567f0d14b40 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb37f400640 1 -- 172.21.15.146:0/2944254824 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb3900294b0 con 0x5567f0d14b40 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb3a75f3740 10 client.16122 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb3a75f3740 10 client.16122 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.232+0000 7fb3a75f3740 20 client.16122 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.233+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d2fd40 0x5567f0d321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.233+0000 7fb3a75f3740 4 client.16122 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:44.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.233+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567f0c87bf0 con 0x5567f0d2fd40 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.233+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d2fd40 0x5567f0d321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.233+0000 7fb37f400640 10 client.16122 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.233+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d2fd40 0x5567f0d321b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=374386378737c75 server_cookie=c9706645bed67fb4 in_seq=0 out_seq=0 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb37f400640 1 -- 172.21.15.146:0/2944254824 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5567f0c87bf0 con 0x5567f0d2fd40 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb37f400640 10 client.16122 handle_command_reply: tid=0 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb3a75f3740 1 client.16122 shutdown 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 msgr2=0x5567f0d1f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d1f5c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb39001e9f0 tx=0x7fb390004730 comp rx=0 tx=0).stop 2024-04-24T08:25:44.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 shutdown_connections 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d2fd40 0x5567f0d321b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.234+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d1fb30 0x5567f0d1cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.235+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d15460 0x5567f0d1c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.235+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2944254824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d1f5c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.235+0000 7fb37f400640 0 client.16122 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.235+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 >> 172.21.15.146:0/2944254824 conn(0x5567f0d10e60 msgr2=0x5567f0d16c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.235+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 shutdown_connections 2024-04-24T08:25:44.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.235+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2944254824 wait complete. 2024-04-24T08:25:44.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.235+0000 7fb3a75f3740 20 client.16122 trim_cache size 0 max 16384 2024-04-24T08:25:44.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.249+0000 7fb3a75f3740 1 Processor -- start 2024-04-24T08:25:44.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.249+0000 7fb3a75f3740 1 -- start start 2024-04-24T08:25:44.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d2b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d1f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1b6a20 con 0x5567f0d14b40 2024-04-24T08:25:44.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1b42f0 con 0x5567f0d1fb30 2024-04-24T08:25:44.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1b2140 con 0x5567f0d15460 2024-04-24T08:25:44.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d1f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48764/0 (socket says 172.21.15.146:48764) 2024-04-24T08:25:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 -- 172.21.15.146:0/1339092625 learned_addr learned my addr 172.21.15.146:0/1339092625 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d2b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 -- 172.21.15.146:0/1339092625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 msgr2=0x5567f0d1f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d1f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 -- 172.21.15.146:0/1339092625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 msgr2=0x5567f0d2b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d2b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.250+0000 7fb3a1400640 1 -- 172.21.15.146:0/1339092625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f0a567d0 con 0x5567f0d15460 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.251+0000 7fb3a1400640 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1b890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb38c02a430 tx=0x7fb38c005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f17bb4a81b49ac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.251+0000 7fb38a600640 1 -- 172.21.15.146:0/1339092625 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c035040 con 0x5567f0d15460 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.251+0000 7fb38a600640 1 -- 172.21.15.146:0/1339092625 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb38c041070 con 0x5567f0d15460 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.251+0000 7fb38a600640 1 -- 172.21.15.146:0/1339092625 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c031a50 con 0x5567f0d15460 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1339092625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 msgr2=0x5567f0d1b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1b890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb38c02a430 tx=0x7fb38c005e50 comp rx=0 tx=0).stop 2024-04-24T08:25:44.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1339092625 shutdown_connections 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d15460 0x5567f0d1b890 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d1f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/1339092625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d14b40 0x5567f0d2b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1339092625 >> 172.21.15.146:0/1339092625 conn(0x5567f0d10e60 msgr2=0x5567f0d1b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1339092625 shutdown_connections 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 -- 172.21.15.146:0/1339092625 wait complete. 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 -- start start 2024-04-24T08:25:44.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.252+0000 7fb3a75f3740 1 Processor -- start 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d14b40 0x5567f0d200b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d15460 0x5567f0d205f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a75f3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d20b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1b27b0 con 0x5567f0d15460 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1b3c20 con 0x5567f0d1fb30 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a75f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1b2e80 con 0x5567f0d14b40 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a75f3740 10 client.0 _ll_register_callbacks cb 0x7fb390009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d14b40 0x5567f0d200b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d14b40 0x5567f0d200b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48768/0 (socket says 172.21.15.146:48768) 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2303993448 learned_addr learned my addr 172.21.15.146:0/2303993448 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d15460 0x5567f0d205f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a2800640 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d20b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2303993448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 msgr2=0x5567f0d20b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d20b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d15460 msgr2=0x5567f0d205f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d15460 0x5567f0d205f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.253+0000 7fb3a1e00640 1 -- 172.21.15.146:0/2303993448 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f0d27020 con 0x5567f0d14b40 2024-04-24T08:25:44.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb3a1e00640 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d14b40 0x5567f0d200b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb39000a010 tx=0x7fb390004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4f101f181ad64a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb37f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:44.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb37f400640 1 -- 172.21.15.146:0/2303993448 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390029630 con 0x5567f0d14b40 2024-04-24T08:25:44.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb38c041070 con 0x5567f0d14b40 2024-04-24T08:25:44.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb37f400640 1 -- 172.21.15.146:0/2303993448 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb390032450 con 0x5567f0d14b40 2024-04-24T08:25:44.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb37f400640 1 -- 172.21.15.146:0/2303993448 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb390032650 con 0x5567f0d14b40 2024-04-24T08:25:44.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb3a75f3740 10 client.26257 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:44.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.254+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5567f0d28380 con 0x5567f0d14b40 2024-04-24T08:25:44.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb37f400640 1 -- 172.21.15.146:0/2303993448 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb38c041070 con 0x5567f0d14b40 2024-04-24T08:25:44.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb37f400640 1 -- 172.21.15.146:0/2303993448 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb3900039c0 con 0x5567f0d14b40 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb3a75f3740 10 client.26257 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb3a75f3740 10 client.26257 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb3a75f3740 20 client.26257 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d96630 0x5567f0d98a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb3a75f3740 4 client.26257 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567f0c81a60 con 0x5567f0d96630 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.255+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d96630 0x5567f0d98a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a1400640 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d96630 0x5567f0d98a90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f7ff132d369962d server_cookie=ac779e9110bac703 in_seq=0 out_seq=0 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb37f400640 10 client.26257 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb37f400640 1 -- 172.21.15.146:0/2303993448 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567f0c81a60 con 0x5567f0d96630 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb37f400640 10 client.26257 handle_command_reply: tid=0 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 client.26257 shutdown 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d14b40 msgr2=0x5567f0d200b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d14b40 0x5567f0d200b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb39000a010 tx=0x7fb390004610 comp rx=0 tx=0).stop 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 shutdown_connections 2024-04-24T08:25:44.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5567f0d96630 0x5567f0d98a90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5567f0d1fb30 0x5567f0d20b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5567f0d15460 0x5567f0d205f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 --2- 172.21.15.146:0/2303993448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5567f0d14b40 0x5567f0d200b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb37f400640 0 client.26257 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 >> 172.21.15.146:0/2303993448 conn(0x5567f0d10e60 msgr2=0x5567f0d1ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 shutdown_connections 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 1 -- 172.21.15.146:0/2303993448 wait complete. 2024-04-24T08:25:44.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.256+0000 7fb3a75f3740 20 client.26257 trim_cache size 0 max 16384 2024-04-24T08:25:44.267 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:25:44.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb3840a3d20 msgr2=0x7fb3840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb3840a3d20 0x7fb3840a61e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5567f0d16100 tx=0x7fb390005f90 comp rx=0 tx=0).stop 2024-04-24T08:25:44.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c112d40 msgr2=0x7fb39c1a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c112d40 0x7fb39c1a36a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb398005fd0 tx=0x7fb39800c6a0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 shutdown_connections 2024-04-24T08:25:44.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb3840a3d20 0x7fb3840a61e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb39c112d40 0x7fb39c1a36a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb39c076e50 0x7fb39c19f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 --2- 172.21.15.146:0/2716046059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb39c076530 0x7fb39c19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 >> 172.21.15.146:0/2716046059 conn(0x7fb39c0ffb10 msgr2=0x7fb39c114400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 shutdown_connections 2024-04-24T08:25:44.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.257+0000 7fb3a3200640 1 -- 172.21.15.146:0/2716046059 wait complete. 2024-04-24T08:25:44.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:25:44.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 -- 172.21.15.146:0/1991693246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c104cd0 msgr2=0x7fc84c108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 --2- 172.21.15.146:0/1991693246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c104cd0 0x7fc84c108d10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc838009900 tx=0x7fc83802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 -- 172.21.15.146:0/1991693246 shutdown_connections 2024-04-24T08:25:44.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 --2- 172.21.15.146:0/1991693246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c1093f0 0x7fc84c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 --2- 172.21.15.146:0/1991693246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c104cd0 0x7fc84c108d10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 --2- 172.21.15.146:0/1991693246 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c104320 0x7fc84c104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 -- 172.21.15.146:0/1991693246 >> 172.21.15.146:0/1991693246 conn(0x7fc84c0ffb40 msgr2=0x7fc84c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 -- 172.21.15.146:0/1991693246 shutdown_connections 2024-04-24T08:25:44.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.481+0000 7fc851e00640 1 -- 172.21.15.146:0/1991693246 wait complete. 2024-04-24T08:25:44.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.482+0000 7fc851e00640 1 Processor -- start 2024-04-24T08:25:44.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.482+0000 7fc851e00640 1 -- start start 2024-04-24T08:25:44.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c104320 0x7fc84c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c1093f0 0x7fc84c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c19a810 0x7fc84c19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1144b0 con 0x7fc84c1093f0 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c114330 con 0x7fc84c104320 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c114630 con 0x7fc84c19a810 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c19a810 0x7fc84c19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c19a810 0x7fc84c19acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48784/0 (socket says 172.21.15.146:48784) 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc851400640 1 -- 172.21.15.146:0/249588249 learned_addr learned my addr 172.21.15.146:0/249588249 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc850a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c104320 0x7fc84c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.483+0000 7fc84be00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c1093f0 0x7fc84c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851400640 1 -- 172.21.15.146:0/249588249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c104320 msgr2=0x7fc84c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851400640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c104320 0x7fc84c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851400640 1 -- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c1093f0 msgr2=0x7fc84c1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851400640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c1093f0 0x7fc84c1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851400640 1 -- 172.21.15.146:0/249588249 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c19b3a0 con 0x7fc84c19a810 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc84be00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c1093f0 0x7fc84c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc850a00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c104320 0x7fc84c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851400640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c19a810 0x7fc84c19acc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc84400eb10 tx=0x7fc84400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc849600640 1 -- 172.21.15.146:0/249588249 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844004b20 con 0x7fc84c19a810 2024-04-24T08:25:44.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc849600640 1 -- 172.21.15.146:0/249588249 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc844004cc0 con 0x7fc84c19a810 2024-04-24T08:25:44.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c19b690 con 0x7fc84c19a810 2024-04-24T08:25:44.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc849600640 1 -- 172.21.15.146:0/249588249 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844010760 con 0x7fc84c19a810 2024-04-24T08:25:44.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.484+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c1a77f0 con 0x7fc84c19a810 2024-04-24T08:25:44.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.487+0000 7fc856214740 1 Processor -- start 2024-04-24T08:25:44.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc849600640 1 -- 172.21.15.146:0/249588249 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fc844005020 con 0x7fc84c19a810 2024-04-24T08:25:44.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc856214740 1 -- start start 2024-04-24T08:25:44.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc856214740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4ca4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc856214740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc856214740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4cb1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc856214740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5588b4c0a1d0 con 0x5588b4ca4b40 2024-04-24T08:25:44.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc856214740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5588b4bb0450 con 0x5588b4ca5460 2024-04-24T08:25:44.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc856214740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588b4c06190 con 0x5588b4cafb30 2024-04-24T08:25:44.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc84be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4ca4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc84be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4ca4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35692/0 (socket says 172.21.15.146:35692) 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.488+0000 7fc84be00640 1 -- 172.21.15.146:0/3029513450 learned_addr learned my addr 172.21.15.146:0/3029513450 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc850a00640 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc84be00640 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4cb1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc850a00640 1 -- 172.21.15.146:0/3029513450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 msgr2=0x5588b4cb1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc850a00640 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4cb1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc850a00640 1 -- 172.21.15.146:0/3029513450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 msgr2=0x5588b4ca4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc850a00640 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4ca4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc850a00640 1 -- 172.21.15.146:0/3029513450 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588b4c6d310 con 0x5588b4ca5460 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc84be00640 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4ca4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc850a00640 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf5f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc840009fd0 tx=0x7fc8400118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9458cab95169e9d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.489+0000 7fc849600640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc8340a3da0 0x7fc8340a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc83dc00640 1 -- 172.21.15.146:0/3029513450 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840004020 con 0x5588b4ca5460 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc849600640 1 -- 172.21.15.146:0/249588249 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc8440d0db0 con 0x7fc84c19a810 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc83dc00640 1 -- 172.21.15.146:0/3029513450 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc840005500 con 0x5588b4ca5460 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc83dc00640 1 -- 172.21.15.146:0/3029513450 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840004ce0 con 0x5588b4ca5460 2024-04-24T08:25:44.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc84be00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc8340a3da0 0x7fc8340a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc856214740 1 -- 172.21.15.146:0/3029513450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 msgr2=0x5588b4caf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc856214740 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf5f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc840009fd0 tx=0x7fc8400118d0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc856214740 1 -- 172.21.15.146:0/3029513450 shutdown_connections 2024-04-24T08:25:44.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc856214740 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4cb1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc856214740 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf5f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc84be00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc8340a3da0 0x7fc8340a6260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc838005c40 tx=0x7fc838031400 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc856214740 1 --2- 172.21.15.146:0/3029513450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4ca4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.490+0000 7fc856214740 1 -- 172.21.15.146:0/3029513450 >> 172.21.15.146:0/3029513450 conn(0x5588b4ca0e60 msgr2=0x5588b4ca1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.491+0000 7fc856214740 1 -- 172.21.15.146:0/3029513450 shutdown_connections 2024-04-24T08:25:44.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.491+0000 7fc856214740 1 -- 172.21.15.146:0/3029513450 wait complete. 2024-04-24T08:25:44.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.491+0000 7fc856214740 1 -- start start 2024-04-24T08:25:44.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.491+0000 7fc856214740 1 Processor -- start 2024-04-24T08:25:44.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc856214740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4caf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc856214740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4cac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc856214740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4cafb30 0x5588b4cacfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc856214740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5588b4cb29c0 con 0x5588b4cafb30 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc856214740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5588b4bc3810 con 0x5588b4ca5460 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc856214740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588b4cb2b40 con 0x5588b4ca4b40 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc856214740 10 client.0 _ll_register_callbacks cb 0x5588b4b91140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc850a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4caf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc84be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4cac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc851400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4cafb30 0x5588b4cacfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc850a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4caf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48792/0 (socket says 172.21.15.146:48792) 2024-04-24T08:25:44.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc850a00640 1 -- 172.21.15.146:0/3085062363 learned_addr learned my addr 172.21.15.146:0/3085062363 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc850a00640 1 -- 172.21.15.146:0/3085062363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 msgr2=0x5588b4cac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc850a00640 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4cac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc850a00640 1 -- 172.21.15.146:0/3085062363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4cafb30 msgr2=0x5588b4cacfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.492+0000 7fc850a00640 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4cafb30 0x5588b4cacfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.493+0000 7fc850a00640 1 -- 172.21.15.146:0/3085062363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588b4bd43f0 con 0x5588b4ca4b40 2024-04-24T08:25:44.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.493+0000 7fc850a00640 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4caf580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc840009fa0 tx=0x7fc840005170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eecb3b57064311d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.493+0000 7fc832000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:44.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.493+0000 7fc832000640 1 -- 172.21.15.146:0/3085062363 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840028020 con 0x5588b4ca4b40 2024-04-24T08:25:44.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.493+0000 7fc832000640 1 -- 172.21.15.146:0/3085062363 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc84000f5a0 con 0x5588b4ca4b40 2024-04-24T08:25:44.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.493+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc840005500 con 0x5588b4ca4b40 2024-04-24T08:25:44.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.493+0000 7fc832000640 1 -- 172.21.15.146:0/3085062363 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc840033050 con 0x5588b4ca4b40 2024-04-24T08:25:44.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 10 client.26269 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:44.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5588b491c3f0 con 0x5588b4ca4b40 2024-04-24T08:25:44.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc832000640 1 -- 172.21.15.146:0/3085062363 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc840005500 con 0x5588b4ca4b40 2024-04-24T08:25:44.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc832000640 1 -- 172.21.15.146:0/3085062363 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fc840023470 con 0x5588b4ca4b40 2024-04-24T08:25:44.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 10 client.26269 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:44.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 10 client.26269 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:25:44.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 20 client.26269 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:44.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4cbfd00 0x5588b4cc20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 4 client.26269 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:44.507 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.494+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588b4c17bf0 con 0x5588b4cbfd00 2024-04-24T08:25:44.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.495+0000 7fc84be00640 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4cbfd00 0x5588b4cc20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.495+0000 7fc832000640 10 client.26269 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:44.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.495+0000 7fc84be00640 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4cbfd00 0x5588b4cc20f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2e3851962e080a11 server_cookie=768be3a859f0ca0d in_seq=0 out_seq=0 2024-04-24T08:25:44.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc832000640 1 -- 172.21.15.146:0/3085062363 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588b4c17bf0 con 0x5588b4cbfd00 2024-04-24T08:25:44.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc832000640 10 client.26269 handle_command_reply: tid=0 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 client.26269 shutdown 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 msgr2=0x5588b4caf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4caf580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc840009fa0 tx=0x7fc840005170 comp rx=0 tx=0).stop 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 shutdown_connections 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4cbfd00 0x5588b4cc20f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4cafb30 0x5588b4cacfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4cac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 --2- 172.21.15.146:0/3085062363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4caf580 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 >> 172.21.15.146:0/3085062363 conn(0x5588b4ca0e60 msgr2=0x5588b4ca6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc832000640 0 client.26269 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:44.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.496+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 shutdown_connections 2024-04-24T08:25:44.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.497+0000 7fc856214740 1 -- 172.21.15.146:0/3085062363 wait complete. 2024-04-24T08:25:44.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.497+0000 7fc856214740 20 client.26269 trim_cache size 0 max 16384 2024-04-24T08:25:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.511+0000 7fc856214740 1 Processor -- start 2024-04-24T08:25:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.511+0000 7fc856214740 1 -- start start 2024-04-24T08:25:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc856214740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4cbb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc856214740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc856214740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4caf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc856214740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1b2650 con 0x5588b4ca4b40 2024-04-24T08:25:44.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc856214740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1aff20 con 0x5588b4ca5460 2024-04-24T08:25:44.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc856214740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1add90 con 0x5588b4cafb30 2024-04-24T08:25:44.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4cbb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4cbb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35720/0 (socket says 172.21.15.146:35720) 2024-04-24T08:25:44.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 -- 172.21.15.146:0/2642118568 learned_addr learned my addr 172.21.15.146:0/2642118568 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc84be00640 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc851400640 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4caf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 -- 172.21.15.146:0/2642118568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 msgr2=0x5588b4caf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4caf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 -- 172.21.15.146:0/2642118568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 msgr2=0x5588b4caf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc850a00640 1 -- 172.21.15.146:0/2642118568 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588b49e67d0 con 0x5588b4ca4b40 2024-04-24T08:25:44.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc851400640 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4caf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:25:44.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.512+0000 7fc84be00640 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:25:44.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.513+0000 7fc850a00640 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4cbb880 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc840000c00 tx=0x7fc840023fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b996136d71799a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.513+0000 7fc83d200640 1 -- 172.21.15.146:0/2642118568 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840028070 con 0x5588b4ca4b40 2024-04-24T08:25:44.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.513+0000 7fc83d200640 1 -- 172.21.15.146:0/2642118568 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc840016070 con 0x5588b4ca4b40 2024-04-24T08:25:44.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.513+0000 7fc83d200640 1 -- 172.21.15.146:0/2642118568 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84001d040 con 0x5588b4ca4b40 2024-04-24T08:25:44.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.513+0000 7fc856214740 1 -- 172.21.15.146:0/2642118568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 msgr2=0x5588b4cbb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.513+0000 7fc856214740 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4cbb880 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc840000c00 tx=0x7fc840023fa0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 -- 172.21.15.146:0/2642118568 shutdown_connections 2024-04-24T08:25:44.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4cafb30 0x5588b4caf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4ca5460 0x5588b4caf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 --2- 172.21.15.146:0/2642118568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca4b40 0x5588b4cbb880 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 -- 172.21.15.146:0/2642118568 >> 172.21.15.146:0/2642118568 conn(0x5588b4ca0e60 msgr2=0x5588b4cab8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 -- 172.21.15.146:0/2642118568 shutdown_connections 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 -- 172.21.15.146:0/2642118568 wait complete. 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 -- start start 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.514+0000 7fc856214740 1 Processor -- start 2024-04-24T08:25:44.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc856214740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4cb7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc856214740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca5460 0x5588b4cb82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc856214740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4cafb30 0x5588b4ca8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc856214740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ae3e0 con 0x5588b4ca5460 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc856214740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1af850 con 0x5588b4cafb30 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc856214740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1aeab0 con 0x5588b4ca4b40 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc856214740 10 client.0 _ll_register_callbacks cb 0x7fc840005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc84be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca5460 0x5588b4cb82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc851400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4cafb30 0x5588b4ca8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc84be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca5460 0x5588b4cb82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35732/0 (socket says 172.21.15.146:35732) 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc84be00640 1 -- 172.21.15.146:0/3844972525 learned_addr learned my addr 172.21.15.146:0/3844972525 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc850a00640 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4cb7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc851400640 1 -- 172.21.15.146:0/3844972525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 msgr2=0x5588b4cb7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc851400640 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4cb7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc851400640 1 -- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca5460 msgr2=0x5588b4cb82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc851400640 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca5460 0x5588b4cb82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc851400640 1 -- 172.21.15.146:0/3844972525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588b4ca90f0 con 0x5588b4cafb30 2024-04-24T08:25:44.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.515+0000 7fc84be00640 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca5460 0x5588b4cb82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc851400640 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4cafb30 0x5588b4ca8b00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc8440ce4b0 tx=0x7fc84409c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3999405212374c15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc832000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:44.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc832000640 1 -- 172.21.15.146:0/3844972525 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8440bb060 con 0x5588b4cafb30 2024-04-24T08:25:44.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc840016070 con 0x5588b4cafb30 2024-04-24T08:25:44.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc832000640 1 -- 172.21.15.146:0/3844972525 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc8440a56e0 con 0x5588b4cafb30 2024-04-24T08:25:44.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc832000640 1 -- 172.21.15.146:0/3844972525 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc844082050 con 0x5588b4cafb30 2024-04-24T08:25:44.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc856214740 10 client.26294 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5588b4caa3f0 con 0x5588b4cafb30 2024-04-24T08:25:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc832000640 1 -- 172.21.15.146:0/3844972525 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc840016070 con 0x5588b4cafb30 2024-04-24T08:25:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc832000640 1 -- 172.21.15.146:0/3844972525 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fc8440bd1d0 con 0x5588b4cafb30 2024-04-24T08:25:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc856214740 10 client.26294 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.516+0000 7fc856214740 10 client.26294 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:25:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.517+0000 7fc856214740 20 client.26294 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:44.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.517+0000 7fc856214740 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4d27180 0x5588b4d29570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.517+0000 7fc856214740 4 client.26294 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.517+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588b4c11a60 con 0x5588b4d27180 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.517+0000 7fc850a00640 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4d27180 0x5588b4d29570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.517+0000 7fc850a00640 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4d27180 0x5588b4d29570 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e5b2df256de86f17 server_cookie=b25ae6c3125f4181 in_seq=0 out_seq=0 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.517+0000 7fc832000640 10 client.26294 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc832000640 1 -- 172.21.15.146:0/3844972525 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588b4c11a60 con 0x5588b4d27180 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc832000640 10 client.26294 handle_command_reply: tid=0 2024-04-24T08:25:44.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 client.26294 shutdown 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4cafb30 msgr2=0x5588b4ca8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4cafb30 0x5588b4ca8b00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc8440ce4b0 tx=0x7fc84409c660 comp rx=0 tx=0).stop 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 shutdown_connections 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5588b4d27180 0x5588b4d29570 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5588b4cafb30 0x5588b4ca8b00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588b4ca5460 0x5588b4cb82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc832000640 0 client.26294 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:25:44.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 --2- 172.21.15.146:0/3844972525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588b4ca4b40 0x5588b4cb7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 >> 172.21.15.146:0/3844972525 conn(0x5588b4ca0e60 msgr2=0x5588b4c9f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 shutdown_connections 2024-04-24T08:25:44.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 1 -- 172.21.15.146:0/3844972525 wait complete. 2024-04-24T08:25:44.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.518+0000 7fc856214740 20 client.26294 trim_cache size 0 max 16384 2024-04-24T08:25:44.531 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:25:44.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc8340a3da0 msgr2=0x7fc8340a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc8340a3da0 0x7fc8340a6260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc838005c40 tx=0x7fc838031400 comp rx=0 tx=0).stop 2024-04-24T08:25:44.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c19a810 msgr2=0x7fc84c19acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c19a810 0x7fc84c19acc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc84400eb10 tx=0x7fc84400c2d0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 shutdown_connections 2024-04-24T08:25:44.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc8340a3da0 0x7fc8340a6260 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc84c19a810 0x7fc84c19acc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc84c1093f0 0x7fc84c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 --2- 172.21.15.146:0/249588249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc84c104320 0x7fc84c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 >> 172.21.15.146:0/249588249 conn(0x7fc84c0ffb40 msgr2=0x7fc84c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 shutdown_connections 2024-04-24T08:25:44.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.519+0000 7fc851e00640 1 -- 172.21.15.146:0/249588249 wait complete. 2024-04-24T08:25:44.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:25:44.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.745+0000 7f5165600640 1 -- 172.21.15.146:0/3483189142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 msgr2=0x7f5160108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.745+0000 7f5165600640 1 --2- 172.21.15.146:0/3483189142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 0x7f5160108a90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f514c009930 tx=0x7f514c02a420 comp rx=0 tx=0).stop 2024-04-24T08:25:44.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.746+0000 7f5165600640 1 -- 172.21.15.146:0/3483189142 shutdown_connections 2024-04-24T08:25:44.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.746+0000 7f5165600640 1 --2- 172.21.15.146:0/3483189142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 0x7f5160108a90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.746+0000 7f5165600640 1 --2- 172.21.15.146:0/3483189142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 0x7f5160115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.746+0000 7f5165600640 1 --2- 172.21.15.146:0/3483189142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f516010e660 0x7f516010ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.746+0000 7f5165600640 1 -- 172.21.15.146:0/3483189142 >> 172.21.15.146:0/3483189142 conn(0x7f5160076540 msgr2=0x7f5160078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.746+0000 7f5165600640 1 -- 172.21.15.146:0/3483189142 shutdown_connections 2024-04-24T08:25:44.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.746+0000 7f5165600640 1 -- 172.21.15.146:0/3483189142 wait complete. 2024-04-24T08:25:44.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.747+0000 7f5165600640 1 Processor -- start 2024-04-24T08:25:44.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.747+0000 7f5165600640 1 -- start start 2024-04-24T08:25:44.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.747+0000 7f5165600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 0x7f516019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.747+0000 7f5165600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 0x7f516019f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f5165600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f516010e660 0x7f51601a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f5165600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f51601189a0 con 0x7f516010e660 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f5165600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5160118820 con 0x7f5160108fd0 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f5165600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5160118b20 con 0x7f5160108610 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 0x7f516019f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f516010e660 0x7f51601a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 0x7f516019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 0x7f516019f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57170/0 (socket says 172.21.15.146:57170) 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515e000640 1 -- 172.21.15.146:0/2521569199 learned_addr learned my addr 172.21.15.146:0/2521569199 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515ea00640 1 -- 172.21.15.146:0/2521569199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 msgr2=0x7f516019f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515ea00640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 0x7f516019f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515ea00640 1 -- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f516010e660 msgr2=0x7f51601a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515ea00640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f516010e660 0x7f51601a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515ea00640 1 -- 172.21.15.146:0/2521569199 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51601a3e70 con 0x7f5160108610 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.748+0000 7f515e000640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 0x7f516019f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.749+0000 7f515ea00640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 0x7f516019ee80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f515000e9c0 tx=0x7f515000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.749+0000 7f5157400640 1 -- 172.21.15.146:0/2521569199 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515000ccc0 con 0x7f5160108610 2024-04-24T08:25:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.749+0000 7f5157400640 1 -- 172.21.15.146:0/2521569199 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51500035f0 con 0x7f5160108610 2024-04-24T08:25:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.749+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51601a4160 con 0x7f5160108610 2024-04-24T08:25:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.749+0000 7f5157400640 1 -- 172.21.15.146:0/2521569199 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515000ce60 con 0x7f5160108610 2024-04-24T08:25:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.749+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51601ab9f0 con 0x7f5160108610 2024-04-24T08:25:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 Processor -- start 2024-04-24T08:25:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 -- start start 2024-04-24T08:25:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fd5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5603e2f3b1d0 con 0x5603e2fe0940 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5603e2ee1450 con 0x5603e2fd6270 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5168422740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5603e2f37190 con 0x5603e2fd5950 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f5157400640 1 -- 172.21.15.146:0/2521569199 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f51500037a0 con 0x7f5160108610 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f515e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f515e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35770/0 (socket says 172.21.15.146:35770) 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f515f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fd5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.753+0000 7f515e000640 1 -- 172.21.15.146:0/3735651386 learned_addr learned my addr 172.21.15.146:0/3735651386 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515ea00640 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515e000640 1 -- 172.21.15.146:0/3735651386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 msgr2=0x5603e2fd5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515e000640 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fd5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515e000640 1 -- 172.21.15.146:0/3735651386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 msgr2=0x5603e2fe0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515e000640 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515e000640 1 -- 172.21.15.146:0/3735651386 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e2f9e310 con 0x5603e2fe0940 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515f400640 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fd5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515ea00640 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:25:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.754+0000 7f515e000640 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe2d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f514c009a50 tx=0x7f514c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bff7492d097a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.755+0000 7f514b400640 1 -- 172.21.15.146:0/3735651386 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c035040 con 0x5603e2fe0940 2024-04-24T08:25:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.755+0000 7f514b400640 1 -- 172.21.15.146:0/3735651386 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f514c031830 con 0x5603e2fe0940 2024-04-24T08:25:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.755+0000 7f514b400640 1 -- 172.21.15.146:0/3735651386 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c0049c0 con 0x5603e2fe0940 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.755+0000 7f5168422740 1 -- 172.21.15.146:0/3735651386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 msgr2=0x5603e2fe2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.755+0000 7f5168422740 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe2d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f514c009a50 tx=0x7f514c005dc0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.755+0000 7f5157400640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f51440a3da0 0x7f51440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 -- 172.21.15.146:0/3735651386 shutdown_connections 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe2d30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5157400640 1 -- 172.21.15.146:0/2521569199 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f51500d0ac0 con 0x7f5160108610 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 --2- 172.21.15.146:0/3735651386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fd5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 -- 172.21.15.146:0/3735651386 >> 172.21.15.146:0/3735651386 conn(0x5603e2fd1c70 msgr2=0x5603e2fd2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f515f400640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f51440a3da0 0x7f51440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 -- 172.21.15.146:0/3735651386 shutdown_connections 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 -- 172.21.15.146:0/3735651386 wait complete. 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 -- start start 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.756+0000 7f5168422740 1 Processor -- start 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f515f400640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f51440a3da0 0x7f51440a6260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5158005fd0 tx=0x7f5158009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f5168422740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd5950 0x5603e2fe0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f5168422740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd6270 0x5603e2fdd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f5168422740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fe0940 0x5603e2fddc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f5168422740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5603e2fe3770 con 0x5603e2fd6270 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f5168422740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5603e2ef4810 con 0x5603e2fd5950 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f5168422740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5603e2fe38f0 con 0x5603e2fe0940 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f5168422740 10 client.0 _ll_register_callbacks cb 0x5603e2ec2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f515e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd6270 0x5603e2fdd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f515ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd5950 0x5603e2fe0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f515e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd6270 0x5603e2fdd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35782/0 (socket says 172.21.15.146:35782) 2024-04-24T08:25:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f515ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd5950 0x5603e2fe0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57194/0 (socket says 172.21.15.146:57194) 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f515e000640 1 -- 172.21.15.146:0/1225781988 learned_addr learned my addr 172.21.15.146:0/1225781988 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.757+0000 7f515f400640 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fe0940 0x5603e2fddc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f515e000640 1 -- 172.21.15.146:0/1225781988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fe0940 msgr2=0x5603e2fddc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f515e000640 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fe0940 0x5603e2fddc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f515e000640 1 -- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd5950 msgr2=0x5603e2fe0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f515e000640 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd5950 0x5603e2fe0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f515e000640 1 -- 172.21.15.146:0/1225781988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e2f053f0 con 0x5603e2fd6270 2024-04-24T08:25:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f515ea00640 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd5950 0x5603e2fe0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f515e000640 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd6270 0x5603e2fdd650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f514c006010 tx=0x7f514c005290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca016ff320cc9f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f513fe00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f513fe00640 1 -- 172.21.15.146:0/1225781988 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c04e070 con 0x5603e2fd6270 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f513fe00640 1 -- 172.21.15.146:0/1225781988 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f514c004250 con 0x5603e2fd6270 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.758+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f514c031830 con 0x5603e2fd6270 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f513fe00640 1 -- 172.21.15.146:0/1225781988 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f514c05a050 con 0x5603e2fd6270 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 10 client.16158 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5603e2c4d3f0 con 0x5603e2fd6270 2024-04-24T08:25:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f513fe00640 1 -- 172.21.15.146:0/1225781988 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f514c031830 con 0x5603e2fd6270 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f513fe00640 1 -- 172.21.15.146:0/1225781988 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f514c0048e0 con 0x5603e2fd6270 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 10 client.16158 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 20 client.16158 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e2ff0890 0x5603e2ff2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 4 client.16158 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.759+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603e2f48bf0 con 0x5603e2ff0890 2024-04-24T08:25:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.760+0000 7f515ea00640 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e2ff0890 0x5603e2ff2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.760+0000 7f513fe00640 10 client.16158 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.760+0000 7f515ea00640 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e2ff0890 0x5603e2ff2c80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a74f45f96dbb9f6e server_cookie=6993618a6f85c613 in_seq=0 out_seq=0 2024-04-24T08:25:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.761+0000 7f513fe00640 1 -- 172.21.15.146:0/1225781988 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603e2f48bf0 con 0x5603e2ff0890 2024-04-24T08:25:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.761+0000 7f513fe00640 10 client.16158 handle_command_reply: tid=0 2024-04-24T08:25:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.761+0000 7f5168422740 1 client.16158 shutdown 2024-04-24T08:25:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd6270 msgr2=0x5603e2fdd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd6270 0x5603e2fdd650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f514c006010 tx=0x7f514c005290 comp rx=0 tx=0).stop 2024-04-24T08:25:44.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 shutdown_connections 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e2ff0890 0x5603e2ff2c80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fe0940 0x5603e2fddc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd6270 0x5603e2fdd650 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 --2- 172.21.15.146:0/1225781988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd5950 0x5603e2fe0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 >> 172.21.15.146:0/1225781988 conn(0x5603e2fd1c70 msgr2=0x5603e2fd7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f513fe00640 0 client.16158 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 shutdown_connections 2024-04-24T08:25:44.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 1 -- 172.21.15.146:0/1225781988 wait complete. 2024-04-24T08:25:44.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.762+0000 7f5168422740 20 client.16158 trim_cache size 0 max 16384 2024-04-24T08:25:44.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.777+0000 7f5168422740 1 Processor -- start 2024-04-24T08:25:44.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.777+0000 7f5168422740 1 -- start start 2024-04-24T08:25:44.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f5168422740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd5950 0x5603e2fec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f5168422740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd6270 0x5603e2fdfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f5168422740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fe0940 0x5603e2ff8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f5168422740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b6ab0 con 0x5603e2fd5950 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f5168422740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b4380 con 0x5603e2fe0940 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f5168422740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b21d0 con 0x5603e2fd6270 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd6270 0x5603e2fdfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fe0940 0x5603e2ff8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd6270 0x5603e2fdfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48842/0 (socket says 172.21.15.146:48842) 2024-04-24T08:25:44.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 -- 172.21.15.146:0/1491150587 learned_addr learned my addr 172.21.15.146:0/1491150587 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd5950 0x5603e2fec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 -- 172.21.15.146:0/1491150587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fe0940 msgr2=0x5603e2ff8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 --2- 172.21.15.146:0/1491150587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fe0940 0x5603e2ff8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 -- 172.21.15.146:0/1491150587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd5950 msgr2=0x5603e2fec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 --2- 172.21.15.146:0/1491150587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd5950 0x5603e2fec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.778+0000 7f515e000640 1 -- 172.21.15.146:0/1491150587 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e2fd60a0 con 0x5603e2fd6270 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.779+0000 7f515e000640 1 --2- 172.21.15.146:0/1491150587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd6270 0x5603e2fdfe90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f514c02a460 tx=0x7f514c005220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6d85eb493058d03 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.779+0000 7f514aa00640 1 -- 172.21.15.146:0/1491150587 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c04e040 con 0x5603e2fd6270 2024-04-24T08:25:44.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.779+0000 7f514aa00640 1 -- 172.21.15.146:0/1491150587 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f514c0315b0 con 0x5603e2fd6270 2024-04-24T08:25:44.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.779+0000 7f514aa00640 1 -- 172.21.15.146:0/1491150587 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c035040 con 0x5603e2fd6270 2024-04-24T08:25:44.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 -- 172.21.15.146:0/1491150587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd6270 msgr2=0x5603e2fdfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 --2- 172.21.15.146:0/1491150587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd6270 0x5603e2fdfe90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f514c02a460 tx=0x7f514c005220 comp rx=0 tx=0).stop 2024-04-24T08:25:44.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 -- 172.21.15.146:0/1491150587 shutdown_connections 2024-04-24T08:25:44.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 --2- 172.21.15.146:0/1491150587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd6270 0x5603e2fdfe90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 --2- 172.21.15.146:0/1491150587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fe0940 0x5603e2ff8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 --2- 172.21.15.146:0/1491150587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fd5950 0x5603e2fec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 -- 172.21.15.146:0/1491150587 >> 172.21.15.146:0/1491150587 conn(0x5603e2fd1c70 msgr2=0x5603e2fd2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 -- 172.21.15.146:0/1491150587 shutdown_connections 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 -- 172.21.15.146:0/1491150587 wait complete. 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 -- start start 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.780+0000 7f5168422740 1 Processor -- start 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f5168422740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fe0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f5168422740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f5168422740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f5168422740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b2840 con 0x5603e2fe0940 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f5168422740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b3cb0 con 0x5603e2fd6270 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f5168422740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51601b2f10 con 0x5603e2fd5950 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f5168422740 10 client.0 _ll_register_callbacks cb 0x7f514c040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f515ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fe0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f515ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fe0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48850/0 (socket says 172.21.15.146:48850) 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f515ea00640 1 -- 172.21.15.146:0/376657270 learned_addr learned my addr 172.21.15.146:0/376657270 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f515f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.781+0000 7f515e000640 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f515ea00640 1 -- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 msgr2=0x5603e2fe1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f515ea00640 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f515ea00640 1 -- 172.21.15.146:0/376657270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 msgr2=0x5603e2fe17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f515ea00640 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f515ea00640 1 -- 172.21.15.146:0/376657270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e2d177d0 con 0x5603e2fd5950 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f515e000640 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f515ea00640 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fe0d20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5150099440 tx=0x7f5150004590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=243c32b447596c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f513fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:44.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f513fe00640 1 -- 172.21.15.146:0/376657270 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51500a0e20 con 0x5603e2fd5950 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f514c0315b0 con 0x5603e2fd5950 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f513fe00640 1 -- 172.21.15.146:0/376657270 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f51500990e0 con 0x5603e2fd5950 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f513fe00640 1 -- 172.21.15.146:0/376657270 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f51500a56e0 con 0x5603e2fd5950 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f5168422740 10 client.26290 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.782+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5603e2fe8d40 con 0x5603e2fd5950 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f513fe00640 1 -- 172.21.15.146:0/376657270 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f514c0315b0 con 0x5603e2fd5950 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f513fe00640 1 -- 172.21.15.146:0/376657270 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f515009bec0 con 0x5603e2fd5950 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f5168422740 10 client.26290 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f5168422740 10 client.26290 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f5168422740 20 client.26290 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f5168422740 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e3056b70 0x5603e3058f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f5168422740 4 client.26290 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603e2f42a60 con 0x5603e3056b70 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.783+0000 7f515e000640 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e3056b70 0x5603e3058f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f513fe00640 10 client.26290 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f515e000640 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e3056b70 0x5603e3058f60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=601cf21498402242 server_cookie=ef0974ee6f959fc5 in_seq=0 out_seq=0 2024-04-24T08:25:44.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f513fe00640 1 -- 172.21.15.146:0/376657270 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603e2f42a60 con 0x5603e3056b70 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f513fe00640 10 client.26290 handle_command_reply: tid=0 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 client.26290 shutdown 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 msgr2=0x5603e2fe0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fe0d20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5150099440 tx=0x7f5150004590 comp rx=0 tx=0).stop 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 shutdown_connections 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5603e3056b70 0x5603e3058f60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5603e2fe0940 0x5603e2fe17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5603e2fd6270 0x5603e2fe1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 --2- 172.21.15.146:0/376657270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603e2fd5950 0x5603e2fe0d20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f513fe00640 0 client.26290 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:44.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 >> 172.21.15.146:0/376657270 conn(0x5603e2fd1c70 msgr2=0x5603e2fd8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 shutdown_connections 2024-04-24T08:25:44.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 1 -- 172.21.15.146:0/376657270 wait complete. 2024-04-24T08:25:44.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.784+0000 7f5168422740 20 client.26290 trim_cache size 0 max 16384 2024-04-24T08:25:44.794 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:25:44.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f51440a3da0 msgr2=0x7f51440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f51440a3da0 0x7f51440a6260 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5158005fd0 tx=0x7f5158009500 comp rx=0 tx=0).stop 2024-04-24T08:25:44.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 msgr2=0x7f516019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:44.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 0x7f516019ee80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f515000e9c0 tx=0x7f515000ee80 comp rx=0 tx=0).stop 2024-04-24T08:25:44.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 shutdown_connections 2024-04-24T08:25:44.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f51440a3da0 0x7f51440a6260 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f516010e660 0x7f51601a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5160108fd0 0x7f516019f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 --2- 172.21.15.146:0/2521569199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5160108610 0x7f516019ee80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 >> 172.21.15.146:0/2521569199 conn(0x7f5160076540 msgr2=0x7f5160077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 shutdown_connections 2024-04-24T08:25:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:44.785+0000 7f5165600640 1 -- 172.21.15.146:0/2521569199 wait complete. 2024-04-24T08:25:44.802 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:44 smithi146 ceph-mon[27805]: pgmap v139: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-24T08:25:44.802 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:44 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1725352707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:25:45.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:44 smithi177 ceph-mon[33046]: pgmap v139: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-24T08:25:45.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:44 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1725352707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:25:45.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:44 smithi153 ceph-mon[32746]: pgmap v139: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-24T08:25:45.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:44 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1725352707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:25:45.805 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:25:45.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:25:45.806 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:25:46.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 -- 172.21.15.146:0/843683333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 msgr2=0x7f0a8c10e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 --2- 172.21.15.146:0/843683333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 0x7f0a8c10e920 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009ca0 tx=0x7f0a8402a950 comp rx=0 tx=0).stop 2024-04-24T08:25:46.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 -- 172.21.15.146:0/843683333 shutdown_connections 2024-04-24T08:25:46.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 --2- 172.21.15.146:0/843683333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 0x7f0a8c10e920 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 --2- 172.21.15.146:0/843683333 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c103980 0x7f0a8c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 --2- 172.21.15.146:0/843683333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103060 0x7f0a8c103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 -- 172.21.15.146:0/843683333 >> 172.21.15.146:0/843683333 conn(0x7f0a8c0fc880 msgr2=0x7f0a8c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 -- 172.21.15.146:0/843683333 shutdown_connections 2024-04-24T08:25:46.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.022+0000 7f0a91600640 1 -- 172.21.15.146:0/843683333 wait complete. 2024-04-24T08:25:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.023+0000 7f0a91600640 1 Processor -- start 2024-04-24T08:25:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.023+0000 7f0a91600640 1 -- start start 2024-04-24T08:25:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a91600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c103060 0x7f0a8c1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a91600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103980 0x7f0a8c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a91600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 0x7f0a8c19a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a91600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c112260 con 0x7f0a8c103980 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a91600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1120e0 con 0x7f0a8c103060 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a91600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1123e0 con 0x7f0a8c10c480 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103980 0x7f0a8c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103980 0x7f0a8c1a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35822/0 (socket says 172.21.15.146:35822) 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a8b400640 1 -- 172.21.15.146:0/2375098206 learned_addr learned my addr 172.21.15.146:0/2375098206 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a90c00640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 0x7f0a8c19a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.024+0000 7f0a8be00640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c103060 0x7f0a8c1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a8b400640 1 -- 172.21.15.146:0/2375098206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 msgr2=0x7f0a8c19a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a8b400640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 0x7f0a8c19a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a8b400640 1 -- 172.21.15.146:0/2375098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c103060 msgr2=0x7f0a8c1a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a8b400640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c103060 0x7f0a8c1a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a8b400640 1 -- 172.21.15.146:0/2375098206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c19b010 con 0x7f0a8c103980 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a8b400640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103980 0x7f0a8c1a0c10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800e9f0 tx=0x7f0a7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a88c00640 1 -- 172.21.15.146:0/2375098206 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7800ccc0 con 0x7f0a8c103980 2024-04-24T08:25:46.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a88c00640 1 -- 172.21.15.146:0/2375098206 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a7800ce60 con 0x7f0a8c103980 2024-04-24T08:25:46.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c19b300 con 0x7f0a8c103980 2024-04-24T08:25:46.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a88c00640 1 -- 172.21.15.146:0/2375098206 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78010690 con 0x7f0a8c103980 2024-04-24T08:25:46.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.025+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1a7520 con 0x7f0a8c103980 2024-04-24T08:25:46.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 Processor -- start 2024-04-24T08:25:46.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 -- start start 2024-04-24T08:25:46.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a88c00640 1 -- 172.21.15.146:0/2375098206 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f0a780046c0 con 0x7f0a8c103980 2024-04-24T08:25:46.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f75650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75b90 0x5584f9f7fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f80260 0x5584f9f82650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5584f9edae70 con 0x5584f9f75270 2024-04-24T08:25:46.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5584f9e81380 con 0x5584f9f80260 2024-04-24T08:25:46.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a959f5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584f9ed6e30 con 0x5584f9f75b90 2024-04-24T08:25:46.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.029+0000 7f0a90c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f75650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f75650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35830/0 (socket says 172.21.15.146:35830) 2024-04-24T08:25:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f80260 0x5584f9f82650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 -- 172.21.15.146:0/1540037999 learned_addr learned my addr 172.21.15.146:0/1540037999 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a8be00640 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75b90 0x5584f9f7fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 -- 172.21.15.146:0/1540037999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75b90 msgr2=0x5584f9f7fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75b90 0x5584f9f7fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 -- 172.21.15.146:0/1540037999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f80260 msgr2=0x5584f9f82650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f80260 0x5584f9f82650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 -- 172.21.15.146:0/1540037999 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584f9cb57e0 con 0x5584f9f75270 2024-04-24T08:25:46.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.030+0000 7f0a90c00640 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f75650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0a84035000 tx=0x7f0a84005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f01c0815c180a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a81200640 1 -- 172.21.15.146:0/1540037999 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84034040 con 0x5584f9f75270 2024-04-24T08:25:46.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a81200640 1 -- 172.21.15.146:0/1540037999 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a8402f940 con 0x5584f9f75270 2024-04-24T08:25:46.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a81200640 1 -- 172.21.15.146:0/1540037999 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a840042a0 con 0x5584f9f75270 2024-04-24T08:25:46.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a88c00640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0a700a3d20 0x7f0a700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a88c00640 1 -- 172.21.15.146:0/2375098206 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0a780d0b10 con 0x7f0a8c103980 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a8be00640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0a700a3d20 0x7f0a700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a959f5740 1 -- 172.21.15.146:0/1540037999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 msgr2=0x5584f9f75650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.031+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f75650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0a84035000 tx=0x7f0a84005fa0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 -- 172.21.15.146:0/1540037999 shutdown_connections 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75b90 0x5584f9f7fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f80260 0x5584f9f82650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1540037999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f75650 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 -- 172.21.15.146:0/1540037999 >> 172.21.15.146:0/1540037999 conn(0x5584f9f71590 msgr2=0x5584f9f71970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 -- 172.21.15.146:0/1540037999 shutdown_connections 2024-04-24T08:25:46.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 -- 172.21.15.146:0/1540037999 wait complete. 2024-04-24T08:25:46.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a8be00640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0a700a3d20 0x7f0a700a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x5584f9f76830 tx=0x7f0a7c00b410 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 -- start start 2024-04-24T08:25:46.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.032+0000 7f0a959f5740 1 Processor -- start 2024-04-24T08:25:46.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75270 0x5584f9f7d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f7d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f80260 0x5584f9f7dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a959f5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5584f9f82ff0 con 0x5584f9f80260 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a959f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5584f9e94660 con 0x5584f9f75b90 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a959f5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584f9f83170 con 0x5584f9f75270 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a959f5740 10 client.0 _ll_register_callbacks cb 0x5584f9e82aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a8be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75270 0x5584f9f7d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a8be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75270 0x5584f9f7d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48872/0 (socket says 172.21.15.146:48872) 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a8be00640 1 -- 172.21.15.146:0/199521486 learned_addr learned my addr 172.21.15.146:0/199521486 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a90c00640 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f80260 0x5584f9f7dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.033+0000 7f0a8b400640 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f7d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a8be00640 1 -- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 msgr2=0x5584f9f7d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a8be00640 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f7d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a8be00640 1 -- 172.21.15.146:0/199521486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f80260 msgr2=0x5584f9f7dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a8be00640 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f80260 0x5584f9f7dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a8be00640 1 -- 172.21.15.146:0/199521486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584f9d15f70 con 0x5584f9f75270 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a8be00640 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75270 0x5584f9f7d1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c01f550 tx=0x7f0a7c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=153519334d3efacb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a75600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a75600640 1 -- 172.21.15.146:0/199521486 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c0107c0 con 0x5584f9f75270 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a8402f940 con 0x5584f9f75270 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.034+0000 7f0a75600640 1 -- 172.21.15.146:0/199521486 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0a7c017030 con 0x5584f9f75270 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a75600640 1 -- 172.21.15.146:0/199521486 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0a7c010960 con 0x5584f9f75270 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 10 client.26302 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5584f9bad170 con 0x5584f9f75270 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a75600640 1 -- 172.21.15.146:0/199521486 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a8402f940 con 0x5584f9f75270 2024-04-24T08:25:46.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a75600640 1 -- 172.21.15.146:0/199521486 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f0a7c01c430 con 0x5584f9f75270 2024-04-24T08:25:46.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 10 client.26302 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:46.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 10 client.26302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:46.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 20 client.26302 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:46.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9f90560 0x5584f9f92a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 4 client.26302 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:46.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.035+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584f9ee8890 con 0x5584f9f90560 2024-04-24T08:25:46.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.036+0000 7f0a8b400640 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9f90560 0x5584f9f92a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.036+0000 7f0a8b400640 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9f90560 0x5584f9f92a00 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1098b905f2d3e069 server_cookie=48a22cd8fbd552d4 in_seq=0 out_seq=0 2024-04-24T08:25:46.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.036+0000 7f0a75600640 10 client.26302 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a75600640 1 -- 172.21.15.146:0/199521486 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584f9ee8890 con 0x5584f9f90560 2024-04-24T08:25:46.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a75600640 10 client.26302 handle_command_reply: tid=0 2024-04-24T08:25:46.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 client.26302 shutdown 2024-04-24T08:25:46.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75270 msgr2=0x5584f9f7d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75270 0x5584f9f7d1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c01f550 tx=0x7f0a7c004730 comp rx=0 tx=0).stop 2024-04-24T08:25:46.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 shutdown_connections 2024-04-24T08:25:46.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9f90560 0x5584f9f92a00 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f80260 0x5584f9f7dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a75600640 0 client.26302 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f7d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 --2- 172.21.15.146:0/199521486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f75270 0x5584f9f7d1a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.037+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 >> 172.21.15.146:0/199521486 conn(0x5584f9f71590 msgr2=0x5584f9f77290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.038+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 shutdown_connections 2024-04-24T08:25:46.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.038+0000 7f0a959f5740 1 -- 172.21.15.146:0/199521486 wait complete. 2024-04-24T08:25:46.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.038+0000 7f0a959f5740 20 client.26302 trim_cache size 0 max 16384 2024-04-24T08:25:46.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 Processor -- start 2024-04-24T08:25:46.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 -- start start 2024-04-24T08:25:46.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f8bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f779d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f70570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1b2590 con 0x5584f9f75270 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1afe60 con 0x5584f9f75b90 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.057+0000 7f0a959f5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1adcb0 con 0x5584f9f80260 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f8bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f779d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57252/0 (socket says 172.21.15.146:57252) 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 -- 172.21.15.146:0/1913392089 learned_addr learned my addr 172.21.15.146:0/1913392089 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a90c00640 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f70570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 -- 172.21.15.146:0/1913392089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 msgr2=0x5584f9f70570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f70570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 -- 172.21.15.146:0/1913392089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 msgr2=0x5584f9f8bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f8bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 -- 172.21.15.146:0/1913392089 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584f9f6bad0 con 0x5584f9f75b90 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.058+0000 7f0a8b400640 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f779d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800c6a0 tx=0x7f0a780bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14a96213a626aafb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a77e00640 1 -- 172.21.15.146:0/1913392089 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7809c5a0 con 0x5584f9f75b90 2024-04-24T08:25:46.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a77e00640 1 -- 172.21.15.146:0/1913392089 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a780d8040 con 0x5584f9f75b90 2024-04-24T08:25:46.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a77e00640 1 -- 172.21.15.146:0/1913392089 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7809c3c0 con 0x5584f9f75b90 2024-04-24T08:25:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 -- 172.21.15.146:0/1913392089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 msgr2=0x5584f9f779d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f779d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800c6a0 tx=0x7f0a780bd040 comp rx=0 tx=0).stop 2024-04-24T08:25:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 -- 172.21.15.146:0/1913392089 shutdown_connections 2024-04-24T08:25:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f70570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75b90 0x5584f9f779d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 --2- 172.21.15.146:0/1913392089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75270 0x5584f9f8bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 -- 172.21.15.146:0/1913392089 >> 172.21.15.146:0/1913392089 conn(0x5584f9f71590 msgr2=0x5584f9f6fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 -- 172.21.15.146:0/1913392089 shutdown_connections 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.059+0000 7f0a959f5740 1 -- 172.21.15.146:0/1913392089 wait complete. 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 -- start start 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 Processor -- start 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75270 0x5584f9f83d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75b90 0x5584f9f84250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f84790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1ae320 con 0x5584f9f75b90 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1af790 con 0x5584f9f75270 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1ae9f0 con 0x5584f9f80260 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.060+0000 7f0a959f5740 10 client.0 _ll_register_callbacks cb 0x7f0a7c00bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75270 0x5584f9f83d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75b90 0x5584f9f84250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75270 0x5584f9f83d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57256/0 (socket says 172.21.15.146:57256) 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 -- 172.21.15.146:0/822619489 learned_addr learned my addr 172.21.15.146:0/822619489 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a90c00640 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f84790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 -- 172.21.15.146:0/822619489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 msgr2=0x5584f9f84790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f84790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 -- 172.21.15.146:0/822619489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75b90 msgr2=0x5584f9f84250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75b90 0x5584f9f84250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 -- 172.21.15.146:0/822619489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584f9ff5af0 con 0x5584f9f75270 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a8be00640 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75270 0x5584f9f83d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c018040 tx=0x7f0a7c01f270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8517e8ea351b65b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.061+0000 7f0a75600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.062+0000 7f0a75600640 1 -- 172.21.15.146:0/822619489 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c03e060 con 0x5584f9f75270 2024-04-24T08:25:46.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.062+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a780d8040 con 0x5584f9f75270 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.062+0000 7f0a75600640 1 -- 172.21.15.146:0/822619489 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0a7c01ccc0 con 0x5584f9f75270 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.062+0000 7f0a75600640 1 -- 172.21.15.146:0/822619489 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0a7c035040 con 0x5584f9f75270 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.062+0000 7f0a959f5740 10 client.26327 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.062+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5584f9ff6d30 con 0x5584f9f75270 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.062+0000 7f0a75600640 1 -- 172.21.15.146:0/822619489 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a780d8040 con 0x5584f9f75270 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a75600640 1 -- 172.21.15.146:0/822619489 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f0a7c0054f0 con 0x5584f9f75270 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a959f5740 10 client.26327 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:46.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a959f5740 10 client.26327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:46.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a959f5740 20 client.26327 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:46.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a959f5740 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9ff74e0 0x5584f9ff9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a959f5740 4 client.26327 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:25:46.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5584f9ee2700 con 0x5584f9ff74e0 2024-04-24T08:25:46.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.063+0000 7f0a8b400640 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9ff74e0 0x5584f9ff9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.064+0000 7f0a8b400640 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9ff74e0 0x5584f9ff9900 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=366cca626703a319 server_cookie=6f0a0535b9919c1b in_seq=0 out_seq=0 2024-04-24T08:25:46.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.064+0000 7f0a75600640 10 client.26327 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a75600640 1 -- 172.21.15.146:0/822619489 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5584f9ee2700 con 0x5584f9ff74e0 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a75600640 10 client.26327 handle_command_reply: tid=0 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 client.26327 shutdown 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75270 msgr2=0x5584f9f83d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75270 0x5584f9f83d10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c018040 tx=0x7f0a7c01f270 comp rx=0 tx=0).stop 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 shutdown_connections 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5584f9ff74e0 0x5584f9ff9900 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584f9f80260 0x5584f9f84790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a75600640 0 client.26327 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5584f9f75b90 0x5584f9f84250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 --2- 172.21.15.146:0/822619489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5584f9f75270 0x5584f9f83d10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 >> 172.21.15.146:0/822619489 conn(0x5584f9f71590 msgr2=0x5584f9f706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 shutdown_connections 2024-04-24T08:25:46.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 1 -- 172.21.15.146:0/822619489 wait complete. 2024-04-24T08:25:46.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.065+0000 7f0a959f5740 20 client.26327 trim_cache size 0 max 16384 2024-04-24T08:25:46.075 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:25:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:25:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "ac71fedb-ba1e-4084-b26b-88939cd0b325", 2024-04-24T08:25:46.075 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:25:46.075 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:25:46.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0a700a3d20 msgr2=0x7f0a700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0a700a3d20 0x7f0a700a61e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x5584f9f76830 tx=0x7f0a7c00b410 comp rx=0 tx=0).stop 2024-04-24T08:25:46.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103980 msgr2=0x7f0a8c1a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103980 0x7f0a8c1a0c10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800e9f0 tx=0x7f0a7800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 shutdown_connections 2024-04-24T08:25:46.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0a700a3d20 0x7f0a700a61e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a8c10c480 0x7f0a8c19a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a8c103980 0x7f0a8c1a0c10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 --2- 172.21.15.146:0/2375098206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a8c103060 0x7f0a8c1a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 >> 172.21.15.146:0/2375098206 conn(0x7f0a8c0fc880 msgr2=0x7f0a8c100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 shutdown_connections 2024-04-24T08:25:46.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.066+0000 7f0a91600640 1 -- 172.21.15.146:0/2375098206 wait complete. 2024-04-24T08:25:46.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:25:46.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:25:46.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.355+0000 7fa1b1600640 1 -- 172.21.15.146:0/153506162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac1060d0 msgr2=0x7fa1ac10dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.355+0000 7fa1b1600640 1 --2- 172.21.15.146:0/153506162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac1060d0 0x7fa1ac10dca0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa194009900 tx=0x7fa19402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.356+0000 7fa1b1600640 1 -- 172.21.15.146:0/153506162 shutdown_connections 2024-04-24T08:25:46.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.356+0000 7fa1b1600640 1 --2- 172.21.15.146:0/153506162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac1060d0 0x7fa1ac10dca0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.356+0000 7fa1b1600640 1 --2- 172.21.15.146:0/153506162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1ac1019b0 0x7fa1ac1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.356+0000 7fa1b1600640 1 --2- 172.21.15.146:0/153506162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1ac101000 0x7fa1ac1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.356+0000 7fa1b1600640 1 -- 172.21.15.146:0/153506162 >> 172.21.15.146:0/153506162 conn(0x7fa1ac0fc880 msgr2=0x7fa1ac0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.356+0000 7fa1b1600640 1 -- 172.21.15.146:0/153506162 shutdown_connections 2024-04-24T08:25:46.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.356+0000 7fa1b1600640 1 -- 172.21.15.146:0/153506162 wait complete. 2024-04-24T08:25:46.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.357+0000 7fa1b1600640 1 Processor -- start 2024-04-24T08:25:46.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.357+0000 7fa1b1600640 1 -- start start 2024-04-24T08:25:46.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1b1600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac101000 0x7fa1ac1a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1b1600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1ac1060d0 0x7fa1ac1a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1b1600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1ac19eb70 0x7fa1ac19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1b1600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac110080 con 0x7fa1ac19eb70 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1b1600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac10ff00 con 0x7fa1ac101000 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1b1600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac110200 con 0x7fa1ac1060d0 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1abe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac101000 0x7fa1ac1a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1abe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac101000 0x7fa1ac1a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57270/0 (socket says 172.21.15.146:57270) 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1abe00640 1 -- 172.21.15.146:0/373799122 learned_addr learned my addr 172.21.15.146:0/373799122 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1ab400640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1ac1060d0 0x7fa1ac1a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1abe00640 1 -- 172.21.15.146:0/373799122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1ac1060d0 msgr2=0x7fa1ac1a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1abe00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1ac1060d0 0x7fa1ac1a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.358+0000 7fa1abe00640 1 -- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1ac19eb70 msgr2=0x7fa1ac19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1b0c00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1ac19eb70 0x7fa1ac19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1abe00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1ac19eb70 0x7fa1ac19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1abe00640 1 -- 172.21.15.146:0/373799122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ac19f770 con 0x7fa1ac101000 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1b0c00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1ac19eb70 0x7fa1ac19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:25:46.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1abe00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac101000 0x7fa1ac1a4a60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400ebc0 tx=0x7fa1a400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1a8c00640 1 -- 172.21.15.146:0/373799122 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4010630 con 0x7fa1ac101000 2024-04-24T08:25:46.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1a8c00640 1 -- 172.21.15.146:0/373799122 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a4010cb0 con 0x7fa1ac101000 2024-04-24T08:25:46.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1a8c00640 1 -- 172.21.15.146:0/373799122 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4018bd0 con 0x7fa1ac101000 2024-04-24T08:25:46.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ac1ab800 con 0x7fa1ac101000 2024-04-24T08:25:46.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.359+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ac1abbb0 con 0x7fa1ac101000 2024-04-24T08:25:46.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.364+0000 7fa1a8c00640 1 -- 172.21.15.146:0/373799122 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fa1a40107d0 con 0x7fa1ac101000 2024-04-24T08:25:46.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.364+0000 7fa1b5b94740 1 Processor -- start 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.364+0000 7fa1b5b94740 1 -- start start 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdcda30 0x55ea6bdcde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdce350 0x55ea6bdd84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bddae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55ea6bd331f0 con 0x55ea6bdcda30 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ea6bcd9350 con 0x55ea6bdd8a20 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ea6bd2f1b0 con 0x55ea6bdce350 2024-04-24T08:25:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdce350 0x55ea6bdd84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1b0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdcda30 0x55ea6bdcde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdce350 0x55ea6bdd84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48924/0 (socket says 172.21.15.146:48924) 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1abe00640 1 -- 172.21.15.146:0/3818265600 learned_addr learned my addr 172.21.15.146:0/3818265600 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.365+0000 7fa1ab400640 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bddae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1abe00640 1 -- 172.21.15.146:0/3818265600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 msgr2=0x55ea6bddae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1abe00640 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bddae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1abe00640 1 -- 172.21.15.146:0/3818265600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdcda30 msgr2=0x55ea6bdcde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1abe00640 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdcda30 0x55ea6bdcde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1abe00640 1 -- 172.21.15.146:0/3818265600 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea6bd2df50 con 0x55ea6bdce350 2024-04-24T08:25:46.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1abe00640 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdce350 0x55ea6bdd84e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40ce5a0 tx=0x7fa1a40d95f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8a2aac6dc6ca13a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1a8c00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa1900a3de0 0x7fa1900a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.366+0000 7fa1a8c00640 1 -- 172.21.15.146:0/373799122 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa1a40d0f60 con 0x7fa1ac101000 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.367+0000 7fa1a1200640 1 -- 172.21.15.146:0/3818265600 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a40e4070 con 0x55ea6bdce350 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.367+0000 7fa1b0c00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa1900a3de0 0x7fa1900a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.367+0000 7fa1a1200640 1 -- 172.21.15.146:0/3818265600 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a40d9e20 con 0x55ea6bdce350 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.367+0000 7fa1a1200640 1 -- 172.21.15.146:0/3818265600 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a409c810 con 0x55ea6bdce350 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3818265600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdce350 msgr2=0x55ea6bdd84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b0c00640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa1900a3de0 0x7fa1900a62a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa194005c40 tx=0x7fa194005a20 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdce350 0x55ea6bdd84e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40ce5a0 tx=0x7fa1a40d95f0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3818265600 shutdown_connections 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdce350 0x55ea6bdd84e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bddae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3818265600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdcda30 0x55ea6bdcde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3818265600 >> 172.21.15.146:0/3818265600 conn(0x55ea6bdc9d50 msgr2=0x55ea6bdca130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3818265600 shutdown_connections 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3818265600 wait complete. 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.368+0000 7fa1b5b94740 1 -- start start 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.369+0000 7fa1b5b94740 1 Processor -- start 2024-04-24T08:25:46.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.369+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdd59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.369+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdd5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55ea6bddb860 con 0x55ea6bdce350 2024-04-24T08:25:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ea6bcec690 con 0x55ea6bdd8a20 2024-04-24T08:25:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ea6bddb9e0 con 0x55ea6bdcda30 2024-04-24T08:25:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdd59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdd59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48936/0 (socket says 172.21.15.146:48936) 2024-04-24T08:25:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1abe00640 1 -- 172.21.15.146:0/1293973398 learned_addr learned my addr 172.21.15.146:0/1293973398 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1b5b94740 10 client.0 _ll_register_callbacks cb 0x55ea6bd28650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1b0c00640 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.370+0000 7fa1ab400640 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdd5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa1abe00640 1 -- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 msgr2=0x55ea6bdd6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa1abe00640 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa1abe00640 1 -- 172.21.15.146:0/1293973398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 msgr2=0x55ea6bdd5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa1abe00640 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdd5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa1abe00640 1 -- 172.21.15.146:0/1293973398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea6bceb920 con 0x55ea6bdcda30 2024-04-24T08:25:46.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa1abe00640 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdd59e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40d9af0 tx=0x7fa1a4000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84be56a0455b3df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa199600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa199600640 1 -- 172.21.15.146:0/1293973398 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4010e60 con 0x55ea6bdcda30 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa199600640 1 -- 172.21.15.146:0/1293973398 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa1a40d4b80 con 0x55ea6bdcda30 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.371+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1a40d9e20 con 0x55ea6bdcda30 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa199600640 1 -- 172.21.15.146:0/1293973398 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa1a409c810 con 0x55ea6bdcda30 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 10 client.26320 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ea6bb0f7d0 con 0x55ea6bdcda30 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa199600640 1 -- 172.21.15.146:0/1293973398 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a40d9e20 con 0x55ea6bdcda30 2024-04-24T08:25:46.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa199600640 1 -- 172.21.15.146:0/1293973398 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa1a40ce8e0 con 0x55ea6bdcda30 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 10 client.26320 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 10 client.26320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 20 client.26320 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6bde8cc0 0x55ea6bdeb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 4 client.26320 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.372+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea6bd40c10 con 0x55ea6bde8cc0 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.373+0000 7fa1ab400640 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6bde8cc0 0x55ea6bdeb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.373+0000 7fa199600640 10 client.26320 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.373+0000 7fa1ab400640 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6bde8cc0 0x55ea6bdeb0b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=199e31475af23119 server_cookie=27b2223c2103455c in_seq=0 out_seq=0 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.374+0000 7fa199600640 1 -- 172.21.15.146:0/1293973398 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea6bd40c10 con 0x55ea6bde8cc0 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.374+0000 7fa199600640 10 client.26320 handle_command_reply: tid=0 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.374+0000 7fa1b5b94740 1 client.26320 shutdown 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.374+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 msgr2=0x55ea6bdd59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.374+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdd59e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40d9af0 tx=0x7fa1a4000f40 comp rx=0 tx=0).stop 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 shutdown_connections 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6bde8cc0 0x55ea6bdeb0b0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdd5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/1293973398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdd59e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa199600640 0 client.26320 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 >> 172.21.15.146:0/1293973398 conn(0x55ea6bdc9d50 msgr2=0x55ea6bdc8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 shutdown_connections 2024-04-24T08:25:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 1 -- 172.21.15.146:0/1293973398 wait complete. 2024-04-24T08:25:46.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.375+0000 7fa1b5b94740 20 client.26320 trim_cache size 0 max 16384 2024-04-24T08:25:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.394+0000 7fa1b5b94740 1 Processor -- start 2024-04-24T08:25:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.394+0000 7fa1b5b94740 1 -- start start 2024-04-24T08:25:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdcda30 0x55ea6bdc8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdcfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b6a40 con 0x55ea6bdce350 2024-04-24T08:25:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b4310 con 0x55ea6bdcda30 2024-04-24T08:25:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b2160 con 0x55ea6bdd8a20 2024-04-24T08:25:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1ab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdcfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b0c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:48938/0 (socket says 172.21.15.146:48938) 2024-04-24T08:25:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1b0c00640 1 -- 172.21.15.146:0/9728307 learned_addr learned my addr 172.21.15.146:0/9728307 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1ab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdcfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35894/0 (socket says 172.21.15.146:35894) 2024-04-24T08:25:46.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.395+0000 7fa1abe00640 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdcda30 0x55ea6bdc8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa1ab400640 1 -- 172.21.15.146:0/9728307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdd8a20 msgr2=0x55ea6bdd0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa1ab400640 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa1ab400640 1 -- 172.21.15.146:0/9728307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdcda30 msgr2=0x55ea6bdc8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa1ab400640 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdcda30 0x55ea6bdc8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa1ab400640 1 -- 172.21.15.146:0/9728307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea6ba453f0 con 0x55ea6bdce350 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa1b0c00640 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa1ab400640 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdcfcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55ea6bdd6fb0 tx=0x7fa19c01a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117c8d8fca71ef37 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa19be00640 1 -- 172.21.15.146:0/9728307 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00fea0 con 0x55ea6bdce350 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa19be00640 1 -- 172.21.15.146:0/9728307 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19c005500 con 0x55ea6bdce350 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.396+0000 7fa19be00640 1 -- 172.21.15.146:0/9728307 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00f370 con 0x55ea6bdce350 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 -- 172.21.15.146:0/9728307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 msgr2=0x55ea6bdcfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdcfcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55ea6bdd6fb0 tx=0x7fa19c01a7d0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 -- 172.21.15.146:0/9728307 shutdown_connections 2024-04-24T08:25:46.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdd8a20 0x55ea6bdd0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdce350 0x55ea6bdcfcd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/9728307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdcda30 0x55ea6bdc8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 -- 172.21.15.146:0/9728307 >> 172.21.15.146:0/9728307 conn(0x55ea6bdc9d50 msgr2=0x55ea6bdc8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 -- 172.21.15.146:0/9728307 shutdown_connections 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.397+0000 7fa1b5b94740 1 -- 172.21.15.146:0/9728307 wait complete. 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 -- start start 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 Processor -- start 2024-04-24T08:25:46.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdec340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdce350 0x55ea6bdec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdd8a20 0x55ea6be47e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b27d0 con 0x55ea6bdd8a20 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b3c40 con 0x55ea6bdce350 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac1b2ea0 con 0x55ea6bdcda30 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.398+0000 7fa1b5b94740 10 client.0 _ll_register_callbacks cb 0x7fa1a40e2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdce350 0x55ea6bdec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdce350 0x55ea6bdec880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57300/0 (socket says 172.21.15.146:57300) 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 -- 172.21.15.146:0/3677179697 learned_addr learned my addr 172.21.15.146:0/3677179697 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1abe00640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdec340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1b0c00640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdd8a20 0x55ea6be47e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 -- 172.21.15.146:0/3677179697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 msgr2=0x55ea6bdec340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdec340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 -- 172.21.15.146:0/3677179697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdd8a20 msgr2=0x55ea6be47e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdd8a20 0x55ea6be47e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 -- 172.21.15.146:0/3677179697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea6ba050f0 con 0x55ea6bdce350 2024-04-24T08:25:46.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1abe00640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdec340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa1ab400640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdce350 0x55ea6bdec880 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa19c000c00 tx=0x7fa19c010eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd12d48f0a3a1197 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.399+0000 7fa199600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.400+0000 7fa199600640 1 -- 172.21.15.146:0/3677179697 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c02f070 con 0x55ea6bdce350 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.400+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa19c005500 con 0x55ea6bdce350 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.400+0000 7fa199600640 1 -- 172.21.15.146:0/3677179697 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa19c005220 con 0x55ea6bdce350 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.400+0000 7fa199600640 1 -- 172.21.15.146:0/3677179697 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa19c03a050 con 0x55ea6bdce350 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.400+0000 7fa1b5b94740 10 client.26345 fetch_fsmap learned FSMap version 25 2024-04-24T08:25:46.407 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.400+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ea6bcf8810 con 0x55ea6bdce350 2024-04-24T08:25:46.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.400+0000 7fa199600640 1 -- 172.21.15.146:0/3677179697 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19c005500 con 0x55ea6bdce350 2024-04-24T08:25:46.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa199600640 1 -- 172.21.15.146:0/3677179697 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa19c029af0 con 0x55ea6bdce350 2024-04-24T08:25:46.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa1b5b94740 10 client.26345 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa1b5b94740 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa1b5b94740 20 client.26345 populate_metadata read hostname 'smithi146' 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6be4fb30 0x55ea6be51f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa1b5b94740 4 client.26345 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ea6bd3aa80 con 0x55ea6be4fb30 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.401+0000 7fa1abe00640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6be4fb30 0x55ea6be51f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1abe00640 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6be4fb30 0x55ea6be51f50 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f41745d5777a3cba server_cookie=cc9991a439ee67c3 in_seq=0 out_seq=0 2024-04-24T08:25:46.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa199600640 10 client.26345 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa199600640 1 -- 172.21.15.146:0/3677179697 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ea6bd3aa80 con 0x55ea6be4fb30 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa199600640 10 client.26345 handle_command_reply: tid=0 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 client.26345 shutdown 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdce350 msgr2=0x55ea6bdec880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdce350 0x55ea6bdec880 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa19c000c00 tx=0x7fa19c010eb0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 shutdown_connections 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea6be4fb30 0x55ea6be51f50 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea6bdd8a20 0x55ea6be47e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa199600640 0 client.26345 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:25:46.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea6bdce350 0x55ea6bdec880 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 --2- 172.21.15.146:0/3677179697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea6bdcda30 0x55ea6bdec340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 >> 172.21.15.146:0/3677179697 conn(0x55ea6bdc9d50 msgr2=0x55ea6bdd6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 shutdown_connections 2024-04-24T08:25:46.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.402+0000 7fa1b5b94740 1 -- 172.21.15.146:0/3677179697 wait complete. 2024-04-24T08:25:46.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b5b94740 20 client.26345 trim_cache size 0 max 16384 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "ac71fedb-ba1e-4084-b26b-88939cd0b325": { 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "ac71fedb-ba1e-4084-b26b-88939cd0b325", 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:25:46.413 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:25:46.414 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:25:46.414 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa1900a3de0 msgr2=0x7fa1900a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa1900a3de0 0x7fa1900a62a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa194005c40 tx=0x7fa194005a20 comp rx=0 tx=0).stop 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac101000 msgr2=0x7fa1ac1a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac101000 0x7fa1ac1a4a60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400ebc0 tx=0x7fa1a400c6a0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 shutdown_connections 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa1900a3de0 0x7fa1900a62a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1ac1060d0 0x7fa1ac1a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1ac19eb70 0x7fa1ac19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 --2- 172.21.15.146:0/373799122 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1ac101000 0x7fa1ac1a4a60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:25:46.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.403+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 >> 172.21.15.146:0/373799122 conn(0x7fa1ac0fc880 msgr2=0x7fa1ac0fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:25:46.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.404+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 shutdown_connections 2024-04-24T08:25:46.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:25:46.404+0000 7fa1b1600640 1 -- 172.21.15.146:0/373799122 wait complete. 2024-04-24T08:25:46.423 INFO:tasks.cephfs.filesystem:scrub status for tag:ac71fedb-ba1e-4084-b26b-88939cd0b325 - {'path': '/', 'tag': 'ac71fedb-ba1e-4084-b26b-88939cd0b325', 'options': 'recursive,force'} 2024-04-24T08:25:47.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:46 smithi177 ceph-mon[33046]: pgmap v140: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:47.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:46 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:25:47.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:46 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:47.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:46 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:25:47.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:46 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:25:47.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:46 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:47.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:46 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:25:47.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:46 smithi153 ceph-mon[32746]: pgmap v140: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:46 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:25:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:46 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:46 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:25:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:46 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:25:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:46 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:46 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:25:47.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:46 smithi146 ceph-mon[27805]: pgmap v140: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:25:47.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:46 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:25:47.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:46 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:47.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:46 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:25:47.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:46 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:25:47.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:46 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:47.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:46 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:25:48.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:47 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:47 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:25:48.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:47 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:47 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:47 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:47 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:25:48.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:47 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:47 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:47 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:47 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:25:48.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:47 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:48.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:47 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:25:49.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:48 smithi177 ceph-mon[33046]: pgmap v141: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:49.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:48 smithi153 ceph-mon[32746]: pgmap v141: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:49.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:48 smithi146 ceph-mon[27805]: pgmap v141: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:51.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:50 smithi177 ceph-mon[33046]: pgmap v142: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:25:51.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:50 smithi153 ceph-mon[32746]: pgmap v142: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:25:51.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:50 smithi146 ceph-mon[27805]: pgmap v142: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:25:53.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:52 smithi177 ceph-mon[33046]: pgmap v143: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:25:53.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:52 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:25:53.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:52 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:25:53.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:52 smithi153 ceph-mon[32746]: pgmap v143: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:25:53.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:52 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:25:53.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:52 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:25:53.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:52 smithi146 ceph-mon[27805]: pgmap v143: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:25:53.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:52 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:25:53.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:52 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:25:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:54 smithi177 ceph-mon[33046]: pgmap v144: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:55.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:54 smithi153 ceph-mon[32746]: pgmap v144: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:55.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:54 smithi146 ceph-mon[27805]: pgmap v144: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:57.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:56 smithi177 ceph-mon[33046]: pgmap v145: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-24T08:25:57.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:56 smithi153 ceph-mon[32746]: pgmap v145: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-24T08:25:57.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:56 smithi146 ceph-mon[27805]: pgmap v145: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-24T08:25:59.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:25:58 smithi153 ceph-mon[32746]: pgmap v146: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:59.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:25:58 smithi146 ceph-mon[27805]: pgmap v146: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:25:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:25:58 smithi177 ceph-mon[33046]: pgmap v146: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:26:01.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:00 smithi153 ceph-mon[32746]: pgmap v147: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:01.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:00 smithi146 ceph-mon[27805]: pgmap v147: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:01.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:00 smithi177 ceph-mon[33046]: pgmap v147: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:03.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:02 smithi153 ceph-mon[32746]: pgmap v148: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:03.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:02 smithi146 ceph-mon[27805]: pgmap v148: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:03.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:02 smithi177 ceph-mon[33046]: pgmap v148: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:05.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:04 smithi153 ceph-mon[32746]: pgmap v149: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:05.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:04 smithi146 ceph-mon[27805]: pgmap v149: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:05.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:04 smithi177 ceph-mon[33046]: pgmap v149: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:07.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:06 smithi153 ceph-mon[32746]: pgmap v150: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:26:07.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:06 smithi146 ceph-mon[27805]: pgmap v150: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:26:07.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:06 smithi177 ceph-mon[33046]: pgmap v150: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:26:09.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:08 smithi153 ceph-mon[32746]: pgmap v151: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:09.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:08 smithi146 ceph-mon[27805]: pgmap v151: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:09.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:08 smithi177 ceph-mon[33046]: pgmap v151: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:11.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:10 smithi153 ceph-mon[32746]: pgmap v152: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:26:11.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:10 smithi146 ceph-mon[27805]: pgmap v152: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:26:11.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:10 smithi177 ceph-mon[33046]: pgmap v152: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:26:13.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:12 smithi153 ceph-mon[32746]: pgmap v153: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:26:13.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:12 smithi146 ceph-mon[27805]: pgmap v153: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:26:13.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:12 smithi177 ceph-mon[33046]: pgmap v153: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:26:15.117 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:14 smithi177 ceph-mon[33046]: pgmap v154: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:15.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:14 smithi153 ceph-mon[32746]: pgmap v154: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:15.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:14 smithi146 ceph-mon[27805]: pgmap v154: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:26:16.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:26:16.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:26:16.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 -- 172.21.15.146:0/1088983983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8104e00 msgr2=0x7f1ac8108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 --2- 172.21.15.146:0/1088983983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8104e00 0x7f1ac8108e40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009900 tx=0x7f1ab402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 -- 172.21.15.146:0/1088983983 shutdown_connections 2024-04-24T08:26:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 --2- 172.21.15.146:0/1088983983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac8109520 0x7f1ac81110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 --2- 172.21.15.146:0/1088983983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8104e00 0x7f1ac8108e40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 --2- 172.21.15.146:0/1088983983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 0x7f1ac8104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 -- 172.21.15.146:0/1088983983 >> 172.21.15.146:0/1088983983 conn(0x7f1ac80ffc70 msgr2=0x7f1ac8102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 -- 172.21.15.146:0/1088983983 shutdown_connections 2024-04-24T08:26:16.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.642+0000 7f1acf400640 1 -- 172.21.15.146:0/1088983983 wait complete. 2024-04-24T08:26:16.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 Processor -- start 2024-04-24T08:26:16.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 -- start start 2024-04-24T08:26:16.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 0x7f1ac81a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8109520 0x7f1ac81a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac819a960 0x7f1ac819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac8114600 con 0x7f1ac819a960 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac8114480 con 0x7f1ac8109520 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.643+0000 7f1acf400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac8114780 con 0x7f1ac8104450 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac819a960 0x7f1ac819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 0x7f1ac81a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 0x7f1ac81a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45788/0 (socket says 172.21.15.146:45788) 2024-04-24T08:26:16.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 -- 172.21.15.146:0/2256378962 learned_addr learned my addr 172.21.15.146:0/2256378962 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac819a960 0x7f1ac819ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59266/0 (socket says 172.21.15.146:59266) 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8109520 0x7f1ac81a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 -- 172.21.15.146:0/2256378962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8109520 msgr2=0x7f1ac81a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8109520 0x7f1ac81a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 -- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac819a960 msgr2=0x7f1ac819ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac819a960 0x7f1ac819ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.644+0000 7f1acca00640 1 -- 172.21.15.146:0/2256378962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac819b480 con 0x7f1ac8104450 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.645+0000 7f1acca00640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 0x7f1ac81a07b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00e9f0 tx=0x7f1abc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.645+0000 7f1ac5600640 1 -- 172.21.15.146:0/2256378962 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00cd80 con 0x7f1ac8104450 2024-04-24T08:26:16.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.645+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac819b770 con 0x7f1ac8104450 2024-04-24T08:26:16.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.645+0000 7f1ac5600640 1 -- 172.21.15.146:0/2256378962 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc005500 con 0x7f1ac8104450 2024-04-24T08:26:16.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.645+0000 7f1ac5600640 1 -- 172.21.15.146:0/2256378962 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc010690 con 0x7f1ac8104450 2024-04-24T08:26:16.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.645+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac81a79c0 con 0x7f1ac8104450 2024-04-24T08:26:16.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.648+0000 7f1ad2251740 1 Processor -- start 2024-04-24T08:26:16.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.648+0000 7f1ad2251740 1 -- start start 2024-04-24T08:26:16.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.648+0000 7f1ac5600640 1 -- 172.21.15.146:0/2256378962 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f1abc0046c0 con 0x7f1ac8104450 2024-04-24T08:26:16.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e7858dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78598310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e7859ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1ad2251740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555e784f31f0 con 0x555e7858e180 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1ad2251740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555e78499350 con 0x555e7858d860 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1ad2251740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555e784ef1b0 con 0x555e78598850 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e7858dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e7858dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:36916/0 (socket says 172.21.15.146:36916) 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 -- 172.21.15.146:0/1846124447 learned_addr learned my addr 172.21.15.146:0/1846124447 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acca00640 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78598310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e7859ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 -- 172.21.15.146:0/1846124447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 msgr2=0x555e7859ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e7859ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 -- 172.21.15.146:0/1846124447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 msgr2=0x555e78598310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78598310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acd400640 1 -- 172.21.15.146:0/1846124447 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e784edf50 con 0x555e7858d860 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.649+0000 7f1acca00640 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78598310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.650+0000 7f1acd400640 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e7858dc40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009f50 tx=0x7f1ac00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91df316ce0d171ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.650+0000 7f1ab9c00640 1 -- 172.21.15.146:0/1846124447 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0004d20 con 0x555e7858d860 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.650+0000 7f1ab9c00640 1 -- 172.21.15.146:0/1846124447 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac0004050 con 0x555e7858d860 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.650+0000 7f1ab9c00640 1 -- 172.21.15.146:0/1846124447 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0005220 con 0x555e7858d860 2024-04-24T08:26:16.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.650+0000 7f1ac5600640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ab00a3da0 0x7f1ab00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ac5600640 1 -- 172.21.15.146:0/2256378962 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1abc0d0ba0 con 0x7f1ac8104450 2024-04-24T08:26:16.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1acca00640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ab00a3da0 0x7f1ab00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 -- 172.21.15.146:0/1846124447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 msgr2=0x555e7858dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e7858dc40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009f50 tx=0x7f1ac00118d0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 -- 172.21.15.146:0/1846124447 shutdown_connections 2024-04-24T08:26:16.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e7859ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78598310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1846124447 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e7858dc40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 -- 172.21.15.146:0/1846124447 >> 172.21.15.146:0/1846124447 conn(0x555e78589b80 msgr2=0x555e78589f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 -- 172.21.15.146:0/1846124447 shutdown_connections 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1ad2251740 1 -- 172.21.15.146:0/1846124447 wait complete. 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.651+0000 7f1acca00640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ab00a3da0 0x7f1ab00a6260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x555e7858ee20 tx=0x7f1abc004a40 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.652+0000 7f1ad2251740 1 -- start start 2024-04-24T08:26:16.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.652+0000 7f1ad2251740 1 Processor -- start 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.652+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e785957a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.652+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78595ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.652+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.652+0000 7f1ad2251740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555e7859b640 con 0x555e7858e180 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1ad2251740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555e784ac690 con 0x555e7858d860 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1ad2251740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555e7859b7c0 con 0x555e78598850 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1ad2251740 10 client.0 _ll_register_callbacks cb 0x555e784e8650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:16.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1ac7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78595ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1ac7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78595ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59284/0 (socket says 172.21.15.146:59284) 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1ac7e00640 1 -- 172.21.15.146:0/3340947807 learned_addr learned my addr 172.21.15.146:0/3340947807 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1acd400640 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e785957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.653+0000 7f1acd400640 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1acd400640 1 -- 172.21.15.146:0/3340947807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 msgr2=0x555e78596220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1acd400640 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1acd400640 1 -- 172.21.15.146:0/3340947807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 msgr2=0x555e78595ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1acd400640 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78595ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1acd400640 1 -- 172.21.15.146:0/3340947807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e784ab920 con 0x555e7858d860 2024-04-24T08:26:16.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1acd400640 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e785957a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009250 tx=0x7f1ac00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1fc57f773f785be server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1aae000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1aae000640 1 -- 172.21.15.146:0/3340947807 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac002b070 con 0x555e7858d860 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1aae000640 1 -- 172.21.15.146:0/3340947807 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1ac0027640 con 0x555e7858d860 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.654+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac0004050 con 0x555e7858d860 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1aae000640 1 -- 172.21.15.146:0/3340947807 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1ac0037050 con 0x555e7858d860 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 10 client.26357 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555e782cf7d0 con 0x555e7858d860 2024-04-24T08:26:16.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1aae000640 1 -- 172.21.15.146:0/3340947807 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac0004050 con 0x555e7858d860 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1aae000640 1 -- 172.21.15.146:0/3340947807 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f1ac000f6e0 con 0x555e7858d860 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 10 client.26357 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 10 client.26357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 20 client.26357 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e785a8ad0 0x555e785aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 4 client.26357 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e78500c10 con 0x555e785a8ad0 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.655+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e785a8ad0 0x555e785aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.656+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e785a8ad0 0x555e785aaec0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f8ec0d72bf266e server_cookie=7671c954f6ec734f in_seq=0 out_seq=0 2024-04-24T08:26:16.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.656+0000 7f1aae000640 10 client.26357 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:16.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1aae000640 1 -- 172.21.15.146:0/3340947807 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e78500c10 con 0x555e785a8ad0 2024-04-24T08:26:16.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1aae000640 10 client.26357 handle_command_reply: tid=0 2024-04-24T08:26:16.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 client.26357 shutdown 2024-04-24T08:26:16.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 msgr2=0x555e785957a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e785957a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009250 tx=0x7f1ac00047b0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 shutdown_connections 2024-04-24T08:26:16.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e785a8ad0 0x555e785aaec0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e78595ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 --2- 172.21.15.146:0/3340947807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858d860 0x555e785957a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 >> 172.21.15.146:0/3340947807 conn(0x555e78589b80 msgr2=0x555e7858f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1aae000640 0 client.26357 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 shutdown_connections 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 1 -- 172.21.15.146:0/3340947807 wait complete. 2024-04-24T08:26:16.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.657+0000 7f1ad2251740 20 client.26357 trim_cache size 0 max 16384 2024-04-24T08:26:16.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.676+0000 7f1ad2251740 1 Processor -- start 2024-04-24T08:26:16.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.676+0000 7f1ad2251740 1 -- start start 2024-04-24T08:26:16.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858d860 0x555e785a4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858e180 0x555e785947d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1ad2251740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81b28d0 con 0x555e7858d860 2024-04-24T08:26:16.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1ad2251740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81b01a0 con 0x555e7858e180 2024-04-24T08:26:16.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1ad2251740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81ae010 con 0x555e78598850 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858d860 0x555e785a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858d860 0x555e785a4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59298/0 (socket says 172.21.15.146:59298) 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 -- 172.21.15.146:0/1789640173 learned_addr learned my addr 172.21.15.146:0/1789640173 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858e180 0x555e785947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 -- 172.21.15.146:0/1789640173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 msgr2=0x555e78596040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 -- 172.21.15.146:0/1789640173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858e180 msgr2=0x555e785947d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.677+0000 7f1acd400640 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858e180 0x555e785947d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.678+0000 7f1acd400640 1 -- 172.21.15.146:0/1789640173 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e7858dfb0 con 0x555e7858d860 2024-04-24T08:26:16.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.678+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858e180 0x555e785947d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:26:16.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.678+0000 7f1acd400640 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858d860 0x555e785a4500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0012fd0 tx=0x7f1ac0009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af58341cd22c253 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.678+0000 7f1ab9200640 1 -- 172.21.15.146:0/1789640173 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac002b070 con 0x555e7858d860 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.678+0000 7f1ab9200640 1 -- 172.21.15.146:0/1789640173 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac0003ea0 con 0x555e7858d860 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.678+0000 7f1ab9200640 1 -- 172.21.15.146:0/1789640173 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac00198f0 con 0x555e7858d860 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 -- 172.21.15.146:0/1789640173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858d860 msgr2=0x555e785a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858d860 0x555e785a4500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0012fd0 tx=0x7f1ac0009520 comp rx=0 tx=0).stop 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 -- 172.21.15.146:0/1789640173 shutdown_connections 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e78598850 0x555e78596040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e7858e180 0x555e785947d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 --2- 172.21.15.146:0/1789640173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858d860 0x555e785a4500 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 -- 172.21.15.146:0/1789640173 >> 172.21.15.146:0/1789640173 conn(0x555e78589b80 msgr2=0x555e78588390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 -- 172.21.15.146:0/1789640173 shutdown_connections 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 -- 172.21.15.146:0/1789640173 wait complete. 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.679+0000 7f1ad2251740 1 -- start start 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 1 Processor -- start 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e7858d860 0x555e7859bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e7859c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e78598850 0x555e78615710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81ae660 con 0x555e7858e180 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81afad0 con 0x555e78598850 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81aed30 con 0x555e7858d860 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ad2251740 10 client.0 _ll_register_callbacks cb 0x7f1abc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ac7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e7859c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ac7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e7859c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59314/0 (socket says 172.21.15.146:59314) 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.680+0000 7f1ac7e00640 1 -- 172.21.15.146:0/2097633645 learned_addr learned my addr 172.21.15.146:0/2097633645 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1acd400640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e7858d860 0x555e7859bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1acd400640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e78598850 0x555e78615710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1ac7e00640 1 -- 172.21.15.146:0/2097633645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e7858d860 msgr2=0x555e7859bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e7858d860 0x555e7859bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1ac7e00640 1 -- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e78598850 msgr2=0x555e78615710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e78598850 0x555e78615710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1ac7e00640 1 -- 172.21.15.146:0/2097633645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e782053f0 con 0x555e7858e180 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1acd400640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e7858d860 0x555e7859bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1ac7e00640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e7859c140 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4005c00 tx=0x7f1ab40044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b0f4f48683abf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1aae000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1aae000640 1 -- 172.21.15.146:0/2097633645 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab404a020 con 0x555e7858e180 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.681+0000 7f1aae000640 1 -- 172.21.15.146:0/2097633645 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1ab4046330 con 0x555e7858e180 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac0003ea0 con 0x555e7858e180 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1aae000640 1 -- 172.21.15.146:0/2097633645 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1ab4035040 con 0x555e7858e180 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 10 client.16215 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555e781c50f0 con 0x555e7858e180 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1aae000640 1 -- 172.21.15.146:0/2097633645 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac0003ea0 con 0x555e7858e180 2024-04-24T08:26:16.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1aae000640 1 -- 172.21.15.146:0/2097633645 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f1ab4004750 con 0x555e7858e180 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 10 client.16215 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 20 client.16215 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e786172f0 0x555e78619710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 4 client.16215 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.682+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e78594e50 con 0x555e786172f0 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.683+0000 7f1acd400640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e786172f0 0x555e78619710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.683+0000 7f1acd400640 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e786172f0 0x555e78619710 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ac27730bf34e444 server_cookie=b0c642c80d7e6854 in_seq=0 out_seq=0 2024-04-24T08:26:16.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.683+0000 7f1aae000640 10 client.16215 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.683+0000 7f1aae000640 1 -- 172.21.15.146:0/2097633645 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555e78594e50 con 0x555e786172f0 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.683+0000 7f1aae000640 10 client.16215 handle_command_reply: tid=0 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 client.16215 shutdown 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 msgr2=0x555e7859c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e7859c140 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4005c00 tx=0x7f1ab40044f0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 shutdown_connections 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e786172f0 0x555e78619710 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e78598850 0x555e78615710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e7858e180 0x555e7859c140 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 --2- 172.21.15.146:0/2097633645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e7858d860 0x555e7859bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 >> 172.21.15.146:0/2097633645 conn(0x555e78589b80 msgr2=0x555e78588c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:16.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1aae000640 0 client.16215 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:16.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 shutdown_connections 2024-04-24T08:26:16.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 1 -- 172.21.15.146:0/2097633645 wait complete. 2024-04-24T08:26:16.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.684+0000 7f1ad2251740 20 client.16215 trim_cache size 0 max 16384 2024-04-24T08:26:16.693 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:26:16.694 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:26:16.694 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:26:16.694 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ab00a3da0 msgr2=0x7f1ab00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ab00a3da0 0x7f1ab00a6260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x555e7858ee20 tx=0x7f1abc004a40 comp rx=0 tx=0).stop 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 msgr2=0x7f1ac81a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 0x7f1ac81a07b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00e9f0 tx=0x7f1abc00edf0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 shutdown_connections 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1ab00a3da0 0x7f1ab00a6260 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac819a960 0x7f1ac819ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac8109520 0x7f1ac81a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 --2- 172.21.15.146:0/2256378962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1ac8104450 0x7f1ac81a07b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 >> 172.21.15.146:0/2256378962 conn(0x7f1ac80ffc70 msgr2=0x7f1ac8100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:16.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 shutdown_connections 2024-04-24T08:26:16.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.685+0000 7f1acf400640 1 -- 172.21.15.146:0/2256378962 wait complete. 2024-04-24T08:26:16.707 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:26:16.707 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:26:16.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 -- 172.21.15.146:0/1631333943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674104270 msgr2=0x7f4674104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 --2- 172.21.15.146:0/1631333943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674104270 0x7f4674104650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4668009860 tx=0x7f466802a340 comp rx=0 tx=0).stop 2024-04-24T08:26:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 -- 172.21.15.146:0/1631333943 shutdown_connections 2024-04-24T08:26:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 --2- 172.21.15.146:0/1631333943 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4674109340 0x7f4674110f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 --2- 172.21.15.146:0/1631333943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4674104c20 0x7f4674108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 --2- 172.21.15.146:0/1631333943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674104270 0x7f4674104650 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 -- 172.21.15.146:0/1631333943 >> 172.21.15.146:0/1631333943 conn(0x7f46740ffad0 msgr2=0x7f4674101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:16.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 -- 172.21.15.146:0/1631333943 shutdown_connections 2024-04-24T08:26:16.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.912+0000 7f467a400640 1 -- 172.21.15.146:0/1631333943 wait complete. 2024-04-24T08:26:16.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.913+0000 7f467a400640 1 Processor -- start 2024-04-24T08:26:16.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.913+0000 7f467a400640 1 -- start start 2024-04-24T08:26:16.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.913+0000 7f467a400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4674104270 0x7f46741a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.913+0000 7f467a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674109340 0x7f46741a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f467a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f467419a750 0x7f467419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f467a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f46741143d0 con 0x7f467419a750 2024-04-24T08:26:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f467a400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4674114250 con 0x7f4674104270 2024-04-24T08:26:16.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f467a400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4674114550 con 0x7f4674109340 2024-04-24T08:26:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4679000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4674104270 0x7f46741a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4679a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f467419a750 0x7f467419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4673e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674109340 0x7f46741a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4679a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f467419a750 0x7f467419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59342/0 (socket says 172.21.15.146:59342) 2024-04-24T08:26:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4679a00640 1 -- 172.21.15.146:0/674107998 learned_addr learned my addr 172.21.15.146:0/674107998 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4673e00640 1 -- 172.21.15.146:0/674107998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4674104270 msgr2=0x7f46741a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4673e00640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4674104270 0x7f46741a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4673e00640 1 -- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f467419a750 msgr2=0x7f467419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:16.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4673e00640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f467419a750 0x7f467419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:16.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4673e00640 1 -- 172.21.15.146:0/674107998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467419b350 con 0x7f4674109340 2024-04-24T08:26:16.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4679a00640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f467419a750 0x7f467419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:16.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.914+0000 7f4679000640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4674104270 0x7f46741a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:16.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.915+0000 7f4673e00640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674109340 0x7f46741a0b80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4660005fd0 tx=0x7f466000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.915+0000 7f4671600640 1 -- 172.21.15.146:0/674107998 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466000ccc0 con 0x7f4674109340 2024-04-24T08:26:16.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.915+0000 7f4671600640 1 -- 172.21.15.146:0/674107998 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f466000ee70 con 0x7f4674109340 2024-04-24T08:26:16.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.915+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46741a73e0 con 0x7f4674109340 2024-04-24T08:26:16.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.915+0000 7f4671600640 1 -- 172.21.15.146:0/674107998 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46600187e0 con 0x7f4674109340 2024-04-24T08:26:16.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.915+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46741a77e0 con 0x7f4674109340 2024-04-24T08:26:16.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.916+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4674105740 con 0x7f4674109340 2024-04-24T08:26:16.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.919+0000 7f4671600640 1 -- 172.21.15.146:0/674107998 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f4660010430 con 0x7f4674109340 2024-04-24T08:26:16.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.920+0000 7f4671600640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f465c0a3d20 0x7f465c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:16.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.920+0000 7f4671600640 1 -- 172.21.15.146:0/674107998 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f46600d0a00 con 0x7f4674109340 2024-04-24T08:26:16.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.920+0000 7f4679000640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f465c0a3d20 0x7f465c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:16.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.921+0000 7f4679000640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f465c0a3d20 0x7f465c0a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f466802a930 tx=0x7f46680056d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:16.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:16.925+0000 7f4671600640 1 -- 172.21.15.146:0/674107998 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f466009ca00 con 0x7f4674109340 2024-04-24T08:26:17.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.167+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f467410c570 con 0x7f4674109340 2024-04-24T08:26:17.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.168+0000 7f4671600640 1 -- 172.21.15.146:0/674107998 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11892 (secure 0 0 0) 0x7f46600a0d00 con 0x7f4674109340 2024-04-24T08:26:17.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:26:17.170 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:23:07.211437+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24568,24569,24553]},"id":1}]} 2024-04-24T08:26:17.170 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 25 2024-04-24T08:26:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f465c0a3d20 msgr2=0x7f465c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f465c0a3d20 0x7f465c0a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f466802a930 tx=0x7f46680056d0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674109340 msgr2=0x7f46741a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674109340 0x7f46741a0b80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4660005fd0 tx=0x7f466000ec20 comp rx=0 tx=0).stop 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 shutdown_connections 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f465c0a3d20 0x7f465c0a61e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4674109340 0x7f46741a0b80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f467419a750 0x7f467419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 --2- 172.21.15.146:0/674107998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4674104270 0x7f46741a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 >> 172.21.15.146:0/674107998 conn(0x7f46740ffad0 msgr2=0x7f4674101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 shutdown_connections 2024-04-24T08:26:17.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.171+0000 7f467a400640 1 -- 172.21.15.146:0/674107998 wait complete. 2024-04-24T08:26:17.188 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:16 smithi146 ceph-mon[27805]: pgmap v155: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:26:17.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:26:17.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:16 smithi153 ceph-mon[32746]: pgmap v155: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:26:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:16 smithi177 ceph-mon[33046]: pgmap v155: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:26:17.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 -- 172.21.15.146:0/4075228301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540112d40 msgr2=0x7f0540115130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 --2- 172.21.15.146:0/4075228301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540112d40 0x7f0540115130 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0538009cd0 tx=0x7f053802a8f0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 -- 172.21.15.146:0/4075228301 shutdown_connections 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 --2- 172.21.15.146:0/4075228301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540076e50 0x7f0540112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 --2- 172.21.15.146:0/4075228301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540112d40 0x7f0540115130 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 --2- 172.21.15.146:0/4075228301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076530 0x7f0540076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 -- 172.21.15.146:0/4075228301 >> 172.21.15.146:0/4075228301 conn(0x7f05400ffb70 msgr2=0x7f0540101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 -- 172.21.15.146:0/4075228301 shutdown_connections 2024-04-24T08:26:17.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.393+0000 7f053fe00640 1 -- 172.21.15.146:0/4075228301 wait complete. 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.394+0000 7f053fe00640 1 Processor -- start 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.394+0000 7f053fe00640 1 -- start start 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540076530 0x7f054019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076e50 0x7f054019f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540112d40 0x7f05401a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f05401188d0 con 0x7f0540076e50 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053fe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0540118750 con 0x7f0540076530 2024-04-24T08:26:17.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0540118a50 con 0x7f0540112d40 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076e50 0x7f054019f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540076530 0x7f054019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076e50 0x7f054019f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59358/0 (socket says 172.21.15.146:59358) 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540076530 0x7f054019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:36984/0 (socket says 172.21.15.146:36984) 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 -- 172.21.15.146:0/2279216781 learned_addr learned my addr 172.21.15.146:0/2279216781 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053f400640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540112d40 0x7f05401a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 -- 172.21.15.146:0/2279216781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540112d40 msgr2=0x7f05401a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540112d40 0x7f05401a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 -- 172.21.15.146:0/2279216781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540076530 msgr2=0x7f054019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540076530 0x7f054019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053e000640 1 -- 172.21.15.146:0/2279216781 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05401a3d40 con 0x7f0540076e50 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.395+0000 7f053ea00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540076530 0x7f054019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.396+0000 7f053e000640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076e50 0x7f054019f290 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f052800e9f0 tx=0x7f052800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.396+0000 7f0537400640 1 -- 172.21.15.146:0/2279216781 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052800cd80 con 0x7f0540076e50 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.396+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05401a4030 con 0x7f0540076e50 2024-04-24T08:26:17.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.396+0000 7f0537400640 1 -- 172.21.15.146:0/2279216781 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0528005500 con 0x7f0540076e50 2024-04-24T08:26:17.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.396+0000 7f0537400640 1 -- 172.21.15.146:0/2279216781 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528010690 con 0x7f0540076e50 2024-04-24T08:26:17.401 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.396+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05401ab8c0 con 0x7f0540076e50 2024-04-24T08:26:17.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 Processor -- start 2024-04-24T08:26:17.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f0537400640 1 -- 172.21.15.146:0/2279216781 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f05280046c0 con 0x7f0540076e50 2024-04-24T08:26:17.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 -- start start 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde02f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55daddd681d0 con 0x55dadde0db30 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55daddd0e450 con 0x55dadde03460 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f05483cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55daddd64190 con 0x55dadde02b40 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f053f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde02f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f053f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde02f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45864/0 (socket says 172.21.15.146:45864) 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f053e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.400+0000 7f053f400640 1 -- 172.21.15.146:0/1702468300 learned_addr learned my addr 172.21.15.146:0/1702468300 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f053ea00640 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f053f400640 1 -- 172.21.15.146:0/1702468300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 msgr2=0x55dadde0d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f053f400640 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f053f400640 1 -- 172.21.15.146:0/1702468300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 msgr2=0x55dadde0ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f053f400640 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f053f400640 1 -- 172.21.15.146:0/1702468300 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dadddcb310 con 0x55dadde02b40 2024-04-24T08:26:17.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f053f400640 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde02f20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f054006b460 tx=0x7f05380093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28d2301adb602454 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f0537400640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f05240a3d20 0x7f05240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.401+0000 7f0537400640 1 -- 172.21.15.146:0/2279216781 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f05280d0b70 con 0x7f0540076e50 2024-04-24T08:26:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f053ea00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f05240a3d20 0x7f05240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f052f400640 1 -- 172.21.15.146:0/1702468300 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538030720 con 0x55dadde02b40 2024-04-24T08:26:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f052f400640 1 -- 172.21.15.146:0/1702468300 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0538036030 con 0x55dadde02b40 2024-04-24T08:26:17.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f052f400640 1 -- 172.21.15.146:0/1702468300 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05380043c0 con 0x55dadde02b40 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f05483cc740 1 -- 172.21.15.146:0/1702468300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 msgr2=0x55dadde02f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f053ea00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f05240a3d20 0x7f05240a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55dadde04100 tx=0x7f053000b410 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f05483cc740 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde02f20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f054006b460 tx=0x7f05380093e0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f05483cc740 1 -- 172.21.15.146:0/1702468300 shutdown_connections 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f05483cc740 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f05483cc740 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f05483cc740 1 --2- 172.21.15.146:0/1702468300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde02f20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.402+0000 7f05483cc740 1 -- 172.21.15.146:0/1702468300 >> 172.21.15.146:0/1702468300 conn(0x55dadddfee60 msgr2=0x55dadddff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.403+0000 7f05483cc740 1 -- 172.21.15.146:0/1702468300 shutdown_connections 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.403+0000 7f05483cc740 1 -- 172.21.15.146:0/1702468300 wait complete. 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.403+0000 7f05483cc740 1 -- start start 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.403+0000 7f05483cc740 1 Processor -- start 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f05483cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde0d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f05483cc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f05483cc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f05483cc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dadde10960 con 0x55dadde0db30 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f05483cc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55daddd21810 con 0x55dadde03460 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f05483cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dadde10ae0 con 0x55dadde02b40 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f05483cc740 10 client.0 _ll_register_callbacks cb 0x55daddcef140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f053e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f053ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde0d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f053e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37002/0 (socket says 172.21.15.146:37002) 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f053e000640 1 -- 172.21.15.146:0/175051113 learned_addr learned my addr 172.21.15.146:0/175051113 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.404+0000 7f053f400640 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f053e000640 1 -- 172.21.15.146:0/175051113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 msgr2=0x55dadde0d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f053e000640 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde0d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f053e000640 1 -- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 msgr2=0x55dadde0afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f053e000640 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f053e000640 1 -- 172.21.15.146:0/175051113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daddd323f0 con 0x55dadde03460 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f053e000640 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0a880 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f05280ce330 tx=0x7f05280d42a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58386b275f259102 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f0523e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:17.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f0523e00640 1 -- 172.21.15.146:0/175051113 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052809c660 con 0x55dadde03460 2024-04-24T08:26:17.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f0523e00640 1 -- 172.21.15.146:0/175051113 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f05280a5300 con 0x55dadde03460 2024-04-24T08:26:17.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0538036030 con 0x55dadde03460 2024-04-24T08:26:17.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f0523e00640 1 -- 172.21.15.146:0/175051113 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f052809c480 con 0x55dadde03460 2024-04-24T08:26:17.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f05483cc740 10 client.26378 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:17.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.405+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dadda7a3f0 con 0x55dadde03460 2024-04-24T08:26:17.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f0523e00640 1 -- 172.21.15.146:0/175051113 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0538036030 con 0x55dadde03460 2024-04-24T08:26:17.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f0523e00640 1 -- 172.21.15.146:0/175051113 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f05280b8030 con 0x55dadde03460 2024-04-24T08:26:17.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f05483cc740 10 client.26378 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:17.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f05483cc740 10 client.26378 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:26:17.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f05483cc740 20 client.26378 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f05483cc740 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde1dd20 0x55dadde20110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f05483cc740 4 client.26378 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daddd75bf0 con 0x55dadde1dd20 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.406+0000 7f053ea00640 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde1dd20 0x55dadde20110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.407+0000 7f0523e00640 10 client.26378 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.407+0000 7f053ea00640 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde1dd20 0x55dadde20110 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b0fbbd31fab4c1d server_cookie=cec5bb20f3ed5f20 in_seq=0 out_seq=0 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f0523e00640 1 -- 172.21.15.146:0/175051113 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55daddd75bf0 con 0x55dadde1dd20 2024-04-24T08:26:17.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f0523e00640 10 client.26378 handle_command_reply: tid=0 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 client.26378 shutdown 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 msgr2=0x55dadde0a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0a880 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f05280ce330 tx=0x7f05280d42a0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 shutdown_connections 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde1dd20 0x55dadde20110 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadde0a880 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f0523e00640 0 client.26378 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:17.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 --2- 172.21.15.146:0/175051113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde0d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 >> 172.21.15.146:0/175051113 conn(0x55dadddfee60 msgr2=0x55dadde04c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 shutdown_connections 2024-04-24T08:26:17.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 1 -- 172.21.15.146:0/175051113 wait complete. 2024-04-24T08:26:17.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.408+0000 7f05483cc740 20 client.26378 trim_cache size 0 max 16384 2024-04-24T08:26:17.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 Processor -- start 2024-04-24T08:26:17.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 -- start start 2024-04-24T08:26:17.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde19820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadddfddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f05401b68d0 con 0x55dadde0db30 2024-04-24T08:26:17.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05401b41a0 con 0x55dadde03460 2024-04-24T08:26:17.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f05483cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05401b1ff0 con 0x55dadde02b40 2024-04-24T08:26:17.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f053f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f053f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59390/0 (socket says 172.21.15.146:59390) 2024-04-24T08:26:17.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f053f400640 1 -- 172.21.15.146:0/2277384203 learned_addr learned my addr 172.21.15.146:0/2277384203 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f053ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde19820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.423+0000 7f053e000640 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadddfddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f053f400640 1 -- 172.21.15.146:0/2277384203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 msgr2=0x55dadde19820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f053f400640 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde19820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f053f400640 1 -- 172.21.15.146:0/2277384203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 msgr2=0x55dadddfddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f053f400640 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadddfddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f053f400640 1 -- 172.21.15.146:0/2277384203 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daddb447d0 con 0x55dadde0db30 2024-04-24T08:26:17.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f053ea00640 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde19820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:17.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f053f400640 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0d0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f053802af70 tx=0x7f0538030f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acea1e7dc992de server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f052ea00640 1 -- 172.21.15.146:0/2277384203 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538052020 con 0x55dadde0db30 2024-04-24T08:26:17.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f052ea00640 1 -- 172.21.15.146:0/2277384203 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0538036070 con 0x55dadde0db30 2024-04-24T08:26:17.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.424+0000 7f052ea00640 1 -- 172.21.15.146:0/2277384203 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538004040 con 0x55dadde0db30 2024-04-24T08:26:17.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 -- 172.21.15.146:0/2277384203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 msgr2=0x55dadde0d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0d0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f053802af70 tx=0x7f0538030f30 comp rx=0 tx=0).stop 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 -- 172.21.15.146:0/2277384203 shutdown_connections 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde0db30 0x55dadde0d0c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde03460 0x55dadddfddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 --2- 172.21.15.146:0/2277384203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde02b40 0x55dadde19820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 -- 172.21.15.146:0/2277384203 >> 172.21.15.146:0/2277384203 conn(0x55dadddfee60 msgr2=0x55dadddfd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 -- 172.21.15.146:0/2277384203 shutdown_connections 2024-04-24T08:26:17.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.425+0000 7f05483cc740 1 -- 172.21.15.146:0/2277384203 wait complete. 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 -- start start 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 Processor -- start 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde02b40 0x55dadde0efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde03460 0x55dadde0f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde0db30 0x55dadde14370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f05401b2660 con 0x55dadde02b40 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f05401b3ad0 con 0x55dadde0db30 2024-04-24T08:26:17.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05401b2d30 con 0x55dadde03460 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.426+0000 7f05483cc740 10 client.0 _ll_register_callbacks cb 0x7f05280d43a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde02b40 0x55dadde0efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde02b40 0x55dadde0efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59400/0 (socket says 172.21.15.146:59400) 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 -- 172.21.15.146:0/435677175 learned_addr learned my addr 172.21.15.146:0/435677175 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053f400640 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde0db30 0x55dadde14370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053e000640 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde03460 0x55dadde0f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 -- 172.21.15.146:0/435677175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde03460 msgr2=0x55dadde0f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde03460 0x55dadde0f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 -- 172.21.15.146:0/435677175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde0db30 msgr2=0x55dadde14370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde0db30 0x55dadde14370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 -- 172.21.15.146:0/435677175 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dadde148b0 con 0x55dadde02b40 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f053ea00640 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde02b40 0x55dadde0efb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0530010960 tx=0x7f05300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c54d280183f068c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.427+0000 7f0523e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f0523e00640 1 -- 172.21.15.146:0/435677175 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053001e8d0 con 0x55dadde02b40 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f0523e00640 1 -- 172.21.15.146:0/435677175 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f053001ea70 con 0x55dadde02b40 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0538036070 con 0x55dadde02b40 2024-04-24T08:26:17.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f0523e00640 1 -- 172.21.15.146:0/435677175 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f053001ec70 con 0x55dadde02b40 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 10 client.16239 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dadde84950 con 0x55dadde02b40 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f0523e00640 1 -- 172.21.15.146:0/435677175 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0538036070 con 0x55dadde02b40 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f0523e00640 1 -- 172.21.15.146:0/435677175 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f053003b8b0 con 0x55dadde02b40 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 10 client.16239 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 10 client.16239 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 20 client.16239 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:17.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde85030 0x55dadde87470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 4 client.16239 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:26:17.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.428+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55daddd6fa60 con 0x55dadde85030 2024-04-24T08:26:17.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.429+0000 7f053e000640 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde85030 0x55dadde87470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.429+0000 7f0523e00640 10 client.16239 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:17.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.429+0000 7f053e000640 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde85030 0x55dadde87470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17c2ae0a7d56088f server_cookie=83c02d796cd462a9 in_seq=0 out_seq=0 2024-04-24T08:26:17.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.429+0000 7f0523e00640 1 -- 172.21.15.146:0/435677175 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55daddd6fa60 con 0x55dadde85030 2024-04-24T08:26:17.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.429+0000 7f0523e00640 10 client.16239 handle_command_reply: tid=0 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.429+0000 7f05483cc740 1 client.16239 shutdown 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde02b40 msgr2=0x55dadde0efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde02b40 0x55dadde0efb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0530010960 tx=0x7f05300046c0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 shutdown_connections 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55dadde85030 0x55dadde87470 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dadde03460 0x55dadde0f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dadde0db30 0x55dadde14370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 --2- 172.21.15.146:0/435677175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dadde02b40 0x55dadde0efb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f0523e00640 0 client.16239 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:17.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 >> 172.21.15.146:0/435677175 conn(0x55dadddfee60 msgr2=0x55dadddfdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 shutdown_connections 2024-04-24T08:26:17.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 1 -- 172.21.15.146:0/435677175 wait complete. 2024-04-24T08:26:17.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.430+0000 7f05483cc740 20 client.16239 trim_cache size 0 max 16384 2024-04-24T08:26:17.439 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:26:17.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f05240a3d20 msgr2=0x7f05240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f05240a3d20 0x7f05240a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55dadde04100 tx=0x7f053000b410 comp rx=0 tx=0).stop 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076e50 msgr2=0x7f054019f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076e50 0x7f054019f290 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f052800e9f0 tx=0x7f052800edf0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 shutdown_connections 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f05240a3d20 0x7f05240a61e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0540112d40 0x7f05401a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0540076e50 0x7f054019f290 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 --2- 172.21.15.146:0/2279216781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0540076530 0x7f054019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 >> 172.21.15.146:0/2279216781 conn(0x7f05400ffb70 msgr2=0x7f0540101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 shutdown_connections 2024-04-24T08:26:17.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.431+0000 7f053fe00640 1 -- 172.21.15.146:0/2279216781 wait complete. 2024-04-24T08:26:17.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:26:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 -- 172.21.15.146:0/3630636768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf00108f40 msgr2=0x7faf001153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 --2- 172.21.15.146:0/3630636768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf00108f40 0x7faf001153b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7faef8009930 tx=0x7faef802a420 comp rx=0 tx=0).stop 2024-04-24T08:26:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 -- 172.21.15.146:0/3630636768 shutdown_connections 2024-04-24T08:26:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 --2- 172.21.15.146:0/3630636768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf00108f40 0x7faf001153b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 --2- 172.21.15.146:0/3630636768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 0x7faf00108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 --2- 172.21.15.146:0/3630636768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf0010d500 0x7faf0010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 -- 172.21.15.146:0/3630636768 >> 172.21.15.146:0/3630636768 conn(0x7faf00077e20 msgr2=0x7faf00106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 -- 172.21.15.146:0/3630636768 shutdown_connections 2024-04-24T08:26:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.649+0000 7faf04a00640 1 -- 172.21.15.146:0/3630636768 wait complete. 2024-04-24T08:26:17.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.650+0000 7faf04a00640 1 Processor -- start 2024-04-24T08:26:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.650+0000 7faf04a00640 1 -- start start 2024-04-24T08:26:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faf04a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 0x7faf001a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faf04a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf00108f40 0x7faf001a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faf04a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf0010d500 0x7faf0019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faf04a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faf001188d0 con 0x7faf0010d500 2024-04-24T08:26:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faf04a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf00118750 con 0x7faf00108580 2024-04-24T08:26:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faf04a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf00118a50 con 0x7faf00108f40 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faefe800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 0x7faf001a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faeff200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf0010d500 0x7faf0019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faef5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf00108f40 0x7faf001a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faefe800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 0x7faf001a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37054/0 (socket says 172.21.15.146:37054) 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faefe800640 1 -- 172.21.15.146:0/1415391575 learned_addr learned my addr 172.21.15.146:0/1415391575 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faefe800640 1 -- 172.21.15.146:0/1415391575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf00108f40 msgr2=0x7faf001a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faefe800640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf00108f40 0x7faf001a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faefe800640 1 -- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf0010d500 msgr2=0x7faf0019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.651+0000 7faefe800640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf0010d500 0x7faf0019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faefe800640 1 -- 172.21.15.146:0/1415391575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0019f270 con 0x7faf00108580 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faef5e00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf00108f40 0x7faf001a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faefe800640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 0x7faf001a4940 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faef000eb70 tx=0x7faef000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faef7e00640 1 -- 172.21.15.146:0/1415391575 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0003b70 con 0x7faf00108580 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0019f560 con 0x7faf00108580 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faef7e00640 1 -- 172.21.15.146:0/1415391575 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef000ee70 con 0x7faf00108580 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faef7e00640 1 -- 172.21.15.146:0/1415391575 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0010830 con 0x7faf00108580 2024-04-24T08:26:17.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.652+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf001ab800 con 0x7faf00108580 2024-04-24T08:26:17.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.655+0000 7faf081a5740 1 Processor -- start 2024-04-24T08:26:17.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.655+0000 7faf081a5740 1 -- start start 2024-04-24T08:26:17.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.655+0000 7faf081a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f5b40 0x5646d16f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.655+0000 7faf081a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d17005f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.655+0000 7faf081a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1702f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faf081a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5646d165b1d0 con 0x5646d16f6460 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faf081a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646d1601450 con 0x5646d1700b30 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faf081a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5646d1657190 con 0x5646d16f5b40 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faef7e00640 1 -- 172.21.15.146:0/1415391575 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7faef0004020 con 0x7faf00108580 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f5b40 0x5646d16f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faefe800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d17005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f5b40 0x5646d16f5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45932/0 (socket says 172.21.15.146:45932) 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faef5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1702f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 -- 172.21.15.146:0/1276900513 learned_addr learned my addr 172.21.15.146:0/1276900513 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 -- 172.21.15.146:0/1276900513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 msgr2=0x5646d1702f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 --2- 172.21.15.146:0/1276900513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1702f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 -- 172.21.15.146:0/1276900513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 msgr2=0x5646d17005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 --2- 172.21.15.146:0/1276900513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d17005f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.656+0000 7faeff200640 1 -- 172.21.15.146:0/1276900513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646d16be310 con 0x5646d16f5b40 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.657+0000 7faeff200640 1 --2- 172.21.15.146:0/1276900513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f5b40 0x5646d16f5f20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faef8009930 tx=0x7faef8005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7352a0da9e340d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.657+0000 7faeef400640 1 -- 172.21.15.146:0/1276900513 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8040070 con 0x5646d16f5b40 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.657+0000 7faeef400640 1 -- 172.21.15.146:0/1276900513 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef80318f0 con 0x5646d16f5b40 2024-04-24T08:26:17.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.657+0000 7faeef400640 1 -- 172.21.15.146:0/1276900513 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8035040 con 0x5646d16f5b40 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.657+0000 7faef7e00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3da0 0x7faee40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.657+0000 7faef7e00640 1 -- 172.21.15.146:0/1415391575 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faef00d0e10 con 0x7faf00108580 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faefe800640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3da0 0x7faee40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 -- 172.21.15.146:0/1276900513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f5b40 msgr2=0x5646d16f5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 --2- 172.21.15.146:0/1276900513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f5b40 0x5646d16f5f20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faef8009930 tx=0x7faef8005dc0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 -- 172.21.15.146:0/1276900513 shutdown_connections 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 --2- 172.21.15.146:0/1276900513 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1702f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 --2- 172.21.15.146:0/1276900513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d17005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 --2- 172.21.15.146:0/1276900513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f5b40 0x5646d16f5f20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 -- 172.21.15.146:0/1276900513 >> 172.21.15.146:0/1276900513 conn(0x5646d16f1e60 msgr2=0x5646d16f2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 -- 172.21.15.146:0/1276900513 shutdown_connections 2024-04-24T08:26:17.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faf081a5740 1 -- 172.21.15.146:0/1276900513 wait complete. 2024-04-24T08:26:17.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.658+0000 7faefe800640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3da0 0x7faee40a6260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x5646d16f7100 tx=0x7faef00d48c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 -- start start 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 Processor -- start 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d17004f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d16fd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d16fddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5646d1703960 con 0x5646d16f5b40 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5646d1614810 con 0x5646d1700b30 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5646d1703ae0 con 0x5646d16f6460 2024-04-24T08:26:17.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faf081a5740 10 client.0 _ll_register_callbacks cb 0x5646d15e2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faef5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d16fd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faef5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d16fd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:45936/0 (socket says 172.21.15.146:45936) 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faef5e00640 1 -- 172.21.15.146:0/2105210428 learned_addr learned my addr 172.21.15.146:0/2105210428 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.659+0000 7faeff200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d17004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faeff200640 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d16fddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faef5e00640 1 -- 172.21.15.146:0/2105210428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 msgr2=0x5646d16fddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faef5e00640 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d16fddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faef5e00640 1 -- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 msgr2=0x5646d17004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faef5e00640 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d17004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faef5e00640 1 -- 172.21.15.146:0/2105210428 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646d16253f0 con 0x5646d16f6460 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faeff200640 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d17004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faef5e00640 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d16fd7e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faee8012d20 tx=0x7faee800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bca51e86065c336d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faee3e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faee3e00640 1 -- 172.21.15.146:0/2105210428 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8012110 con 0x5646d16f6460 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef80318f0 con 0x5646d16f6460 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.660+0000 7faee3e00640 1 -- 172.21.15.146:0/2105210428 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faee8004030 con 0x5646d16f6460 2024-04-24T08:26:17.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faee3e00640 1 -- 172.21.15.146:0/2105210428 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faee80122b0 con 0x5646d16f6460 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 10 client.26380 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5646d136d3f0 con 0x5646d16f6460 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faee3e00640 1 -- 172.21.15.146:0/2105210428 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef80318f0 con 0x5646d16f6460 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faee3e00640 1 -- 172.21.15.146:0/2105210428 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7faee8010430 con 0x5646d16f6460 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 10 client.26380 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 10 client.26380 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 20 client.26380 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:17.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1710c80 0x5646d17130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 4 client.26380 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.661+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646d1668bf0 con 0x5646d1710c80 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.662+0000 7faeff200640 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1710c80 0x5646d17130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.662+0000 7faeff200640 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1710c80 0x5646d17130b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba3d4bd233876e93 server_cookie=e1206e8cabf5932c in_seq=0 out_seq=0 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.662+0000 7faee3e00640 10 client.26380 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faee3e00640 1 -- 172.21.15.146:0/2105210428 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646d1668bf0 con 0x5646d1710c80 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faee3e00640 10 client.26380 handle_command_reply: tid=0 2024-04-24T08:26:17.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 client.26380 shutdown 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 msgr2=0x5646d16fd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d16fd7e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faee8012d20 tx=0x7faee800f790 comp rx=0 tx=0).stop 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 shutdown_connections 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1710c80 0x5646d17130b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d16fd7e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d16fddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faee3e00640 0 client.26380 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 --2- 172.21.15.146:0/2105210428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d17004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 >> 172.21.15.146:0/2105210428 conn(0x5646d16f1e60 msgr2=0x5646d16f7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 shutdown_connections 2024-04-24T08:26:17.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 1 -- 172.21.15.146:0/2105210428 wait complete. 2024-04-24T08:26:17.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.663+0000 7faf081a5740 20 client.26380 trim_cache size 0 max 16384 2024-04-24T08:26:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.677+0000 7faf081a5740 1 Processor -- start 2024-04-24T08:26:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.677+0000 7faf081a5740 1 -- start start 2024-04-24T08:26:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faf081a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d16f5b40 0x5646d16fc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faf081a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d16f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faf081a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d1700b30 0x5646d1700020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faf081a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faf001b6950 con 0x5646d16f6460 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faf081a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf001b4220 con 0x5646d16f5b40 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faf081a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf001b2070 con 0x5646d1700b30 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d16f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d16f0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59436/0 (socket says 172.21.15.146:59436) 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 -- 172.21.15.146:0/940832034 learned_addr learned my addr 172.21.15.146:0/940832034 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faeff200640 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d16f5b40 0x5646d16fc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faeff200640 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d1700b30 0x5646d1700020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 -- 172.21.15.146:0/940832034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d1700b30 msgr2=0x5646d1700020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d1700b30 0x5646d1700020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 -- 172.21.15.146:0/940832034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d16f5b40 msgr2=0x5646d16fc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d16f5b40 0x5646d16fc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.678+0000 7faef5e00640 1 -- 172.21.15.146:0/940832034 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646d14377d0 con 0x5646d16f6460 2024-04-24T08:26:17.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faef5e00640 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d16f0dd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7faee8010820 tx=0x7faee8012990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a769a58ef444c140 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faeeea00640 1 -- 172.21.15.146:0/940832034 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8029070 con 0x5646d16f6460 2024-04-24T08:26:17.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faeeea00640 1 -- 172.21.15.146:0/940832034 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee8010430 con 0x5646d16f6460 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faeeea00640 1 -- 172.21.15.146:0/940832034 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8003850 con 0x5646d16f6460 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faf081a5740 1 -- 172.21.15.146:0/940832034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 msgr2=0x5646d16f0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faf081a5740 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d16f0dd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7faee8010820 tx=0x7faee8012990 comp rx=0 tx=0).stop 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faf081a5740 1 -- 172.21.15.146:0/940832034 shutdown_connections 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faf081a5740 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d1700b30 0x5646d1700020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faf081a5740 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f6460 0x5646d16f0dd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faf081a5740 1 --2- 172.21.15.146:0/940832034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d16f5b40 0x5646d16fc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.679+0000 7faf081a5740 1 -- 172.21.15.146:0/940832034 >> 172.21.15.146:0/940832034 conn(0x5646d16f1e60 msgr2=0x5646d16f06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.680+0000 7faf081a5740 1 -- 172.21.15.146:0/940832034 shutdown_connections 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.680+0000 7faf081a5740 1 -- 172.21.15.146:0/940832034 wait complete. 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.680+0000 7faf081a5740 1 -- start start 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.680+0000 7faf081a5740 1 Processor -- start 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.680+0000 7faf081a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d1701af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.680+0000 7faf081a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d1702030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.680+0000 7faf081a5740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1707270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faf081a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faf001b26e0 con 0x5646d16f5b40 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faf081a5740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf001b3b50 con 0x5646d1700b30 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faf081a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf001b2db0 con 0x5646d16f6460 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faf081a5740 10 client.0 _ll_register_callbacks cb 0x7faee8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d1701af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d1701af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59444/0 (socket says 172.21.15.146:59444) 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 -- 172.21.15.146:0/3539797217 learned_addr learned my addr 172.21.15.146:0/3539797217 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faef5e00640 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d1702030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1707270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 -- 172.21.15.146:0/3539797217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 msgr2=0x5646d1702030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d1702030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 -- 172.21.15.146:0/3539797217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 msgr2=0x5646d1707270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1707270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 -- 172.21.15.146:0/3539797217 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646d17077b0 con 0x5646d16f5b40 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faeff200640 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d1701af0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7faef80055a0 tx=0x7faef8005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18464a5d7c5fb214 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.681+0000 7faee3e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:17.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faee3e00640 1 -- 172.21.15.146:0/3539797217 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8035040 con 0x5646d16f5b40 2024-04-24T08:26:17.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faee3e00640 1 -- 172.21.15.146:0/3539797217 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faef80316f0 con 0x5646d16f5b40 2024-04-24T08:26:17.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faee8010430 con 0x5646d16f5b40 2024-04-24T08:26:17.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faee3e00640 1 -- 172.21.15.146:0/3539797217 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faef8061050 con 0x5646d16f5b40 2024-04-24T08:26:17.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 10 client.16257 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:17.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5646d1777860 con 0x5646d16f5b40 2024-04-24T08:26:17.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faee3e00640 1 -- 172.21.15.146:0/3539797217 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee8010430 con 0x5646d16f5b40 2024-04-24T08:26:17.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faee3e00640 1 -- 172.21.15.146:0/3539797217 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7faef80503f0 con 0x5646d16f5b40 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 10 client.16257 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 10 client.16257 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 20 client.16257 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1777b90 0x5646d1779f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 4 client.16257 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.682+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646d1662a60 con 0x5646d1777b90 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.683+0000 7faef5e00640 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1777b90 0x5646d1779f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.683+0000 7faef5e00640 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1777b90 0x5646d1779f80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41c9fe2b6174c382 server_cookie=ab65300852d6f8f5 in_seq=0 out_seq=0 2024-04-24T08:26:17.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.683+0000 7faee3e00640 10 client.16257 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.683+0000 7faee3e00640 1 -- 172.21.15.146:0/3539797217 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646d1662a60 con 0x5646d1777b90 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.683+0000 7faee3e00640 10 client.16257 handle_command_reply: tid=0 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.683+0000 7faf081a5740 1 client.16257 shutdown 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 msgr2=0x5646d1701af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d1701af0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7faef80055a0 tx=0x7faef8005530 comp rx=0 tx=0).stop 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 shutdown_connections 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5646d1777b90 0x5646d1779f80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646d16f6460 0x5646d1702030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5646d1700b30 0x5646d1707270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 --2- 172.21.15.146:0/3539797217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5646d16f5b40 0x5646d1701af0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faee3e00640 0 client.16257 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:17.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 >> 172.21.15.146:0/3539797217 conn(0x5646d16f1e60 msgr2=0x5646d16f06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 shutdown_connections 2024-04-24T08:26:17.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 1 -- 172.21.15.146:0/3539797217 wait complete. 2024-04-24T08:26:17.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf081a5740 20 client.16257 trim_cache size 0 max 16384 2024-04-24T08:26:17.693 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:26:17.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3da0 msgr2=0x7faee40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf04a00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3da0 0x7faee40a6260 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x5646d16f7100 tx=0x7faef00d48c0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 msgr2=0x7faf001a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.684+0000 7faf04a00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 0x7faf001a4940 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faef000eb70 tx=0x7faef000c6a0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 shutdown_connections 2024-04-24T08:26:17.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faee40a3da0 0x7faee40a6260 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf00108f40 0x7faf001a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf0010d500 0x7faf0019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 --2- 172.21.15.146:0/1415391575 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf00108580 0x7faf001a4940 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 >> 172.21.15.146:0/1415391575 conn(0x7faf00077e20 msgr2=0x7faf00078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 shutdown_connections 2024-04-24T08:26:17.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.685+0000 7faf04a00640 1 -- 172.21.15.146:0/1415391575 wait complete. 2024-04-24T08:26:17.702 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:26:17.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 -- 172.21.15.146:0/2768067600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948101000 msgr2=0x7fa9481013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 --2- 172.21.15.146:0/2768067600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948101000 0x7fa9481013e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009900 tx=0x7fa93c02a410 comp rx=0 tx=0).stop 2024-04-24T08:26:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 -- 172.21.15.146:0/2768067600 shutdown_connections 2024-04-24T08:26:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 --2- 172.21.15.146:0/2768067600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9481060d0 0x7fa94810dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 --2- 172.21.15.146:0/2768067600 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481019b0 0x7fa9481059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 --2- 172.21.15.146:0/2768067600 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948101000 0x7fa9481013e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 -- 172.21.15.146:0/2768067600 >> 172.21.15.146:0/2768067600 conn(0x7fa9480fc880 msgr2=0x7fa9480feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 -- 172.21.15.146:0/2768067600 shutdown_connections 2024-04-24T08:26:17.880 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.877+0000 7fa94f000640 1 -- 172.21.15.146:0/2768067600 wait complete. 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.878+0000 7fa94f000640 1 Processor -- start 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.878+0000 7fa94f000640 1 -- start start 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa948101000 0x7fa9481a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94f000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481060d0 0x7fa9481a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94f000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94819a7b0 0x7fa94819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481122d0 con 0x7fa948101000 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94f000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa948112150 con 0x7fa9481060d0 2024-04-24T08:26:17.882 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94f000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa948112450 con 0x7fa94819a7b0 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481060d0 0x7fa9481a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa948101000 0x7fa9481a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481060d0 0x7fa9481a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37092/0 (socket says 172.21.15.146:37092) 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 -- 172.21.15.146:0/3577594376 learned_addr learned my addr 172.21.15.146:0/3577594376 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94e600640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94819a7b0 0x7fa94819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 -- 172.21.15.146:0/3577594376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94819a7b0 msgr2=0x7fa94819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94819a7b0 0x7fa94819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.883 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 -- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa948101000 msgr2=0x7fa9481a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa948101000 0x7fa9481a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94d200640 1 -- 172.21.15.146:0/3577594376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94819b340 con 0x7fa9481060d0 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.879+0000 7fa94dc00640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa948101000 0x7fa9481a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.880+0000 7fa94d200640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481060d0 0x7fa9481a0b80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa93400e9a0 tx=0x7fa93400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.880+0000 7fa946000640 1 -- 172.21.15.146:0/3577594376 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93400cd80 con 0x7fa9481060d0 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.880+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94819b630 con 0x7fa9481060d0 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.880+0000 7fa946000640 1 -- 172.21.15.146:0/3577594376 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9340035f0 con 0x7fa9481060d0 2024-04-24T08:26:17.884 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.880+0000 7fa946000640 1 -- 172.21.15.146:0/3577594376 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934010670 con 0x7fa9481060d0 2024-04-24T08:26:17.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.880+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9481a7810 con 0x7fa9481060d0 2024-04-24T08:26:17.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.883+0000 7fa953428740 1 Processor -- start 2024-04-24T08:26:17.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.883+0000 7fa946000640 1 -- 172.21.15.146:0/3577594376 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fa9340037a0 con 0x7fa9481060d0 2024-04-24T08:26:17.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.883+0000 7fa953428740 1 -- start start 2024-04-24T08:26:17.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa953428740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981b96730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa953428740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa953428740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa953428740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561981af31d0 con 0x561981ba1340 2024-04-24T08:26:17.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa953428740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561981a99450 con 0x561981b96c70 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa953428740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561981aef190 con 0x561981b96350 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94e600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981b96730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59464/0 (socket says 172.21.15.146:59464) 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37098/0 (socket says 172.21.15.146:37098) 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94d200640 1 -- 172.21.15.146:0/3444969059 learned_addr learned my addr 172.21.15.146:0/3444969059 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94dc00640 1 -- 172.21.15.146:0/3444969059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 msgr2=0x561981b96730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94dc00640 1 --2- 172.21.15.146:0/3444969059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981b96730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94dc00640 1 -- 172.21.15.146:0/3444969059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 msgr2=0x561981ba3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94dc00640 1 --2- 172.21.15.146:0/3444969059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.884+0000 7fa94dc00640 1 -- 172.21.15.146:0/3444969059 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561981b56310 con 0x561981b96c70 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.885+0000 7fa94dc00640 1 --2- 172.21.15.146:0/3444969059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba0e00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02aa00 tx=0x7fa93c005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51eb0b2af4a9960e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.885+0000 7fa946000640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9300a3d20 0x7fa9300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.885+0000 7fa946000640 1 -- 172.21.15.146:0/3577594376 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa9340d0a80 con 0x7fa9481060d0 2024-04-24T08:26:17.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.885+0000 7fa94e600640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9300a3d20 0x7fa9300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.885+0000 7fa93aa00640 1 -- 172.21.15.146:0/3444969059 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c035040 con 0x561981b96c70 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.885+0000 7fa93aa00640 1 -- 172.21.15.146:0/3444969059 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c004760 con 0x561981b96c70 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.885+0000 7fa93aa00640 1 -- 172.21.15.146:0/3444969059 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c0038e0 con 0x561981b96c70 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 -- 172.21.15.146:0/3444969059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 msgr2=0x561981ba0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa94e600640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9300a3d20 0x7fa9300a61e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa940005fd0 tx=0x7fa940009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 --2- 172.21.15.146:0/3444969059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba0e00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa93c02aa00 tx=0x7fa93c005de0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 -- 172.21.15.146:0/3444969059 shutdown_connections 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 --2- 172.21.15.146:0/3444969059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 --2- 172.21.15.146:0/3444969059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba0e00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 --2- 172.21.15.146:0/3444969059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981b96730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 -- 172.21.15.146:0/3444969059 >> 172.21.15.146:0/3444969059 conn(0x561981b92670 msgr2=0x561981b92a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 -- 172.21.15.146:0/3444969059 shutdown_connections 2024-04-24T08:26:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 -- 172.21.15.146:0/3444969059 wait complete. 2024-04-24T08:26:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.886+0000 7fa953428740 1 -- start start 2024-04-24T08:26:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 1 Processor -- start 2024-04-24T08:26:17.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981b9e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981b9e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981ba1340 0x561981b9eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561981ba4170 con 0x561981b96350 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561981aac810 con 0x561981b96c70 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561981ba42f0 con 0x561981ba1340 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa953428740 10 client.0 _ll_register_callbacks cb 0x561981a7a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa94d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981b9e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981b9e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa94d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981b9e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37102/0 (socket says 172.21.15.146:37102) 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa94d200640 1 -- 172.21.15.146:0/3244417156 learned_addr learned my addr 172.21.15.146:0/3244417156 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.887+0000 7fa94e600640 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981ba1340 0x561981b9eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa94d200640 1 -- 172.21.15.146:0/3244417156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981ba1340 msgr2=0x561981b9eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa94d200640 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981ba1340 0x561981b9eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa94d200640 1 -- 172.21.15.146:0/3244417156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 msgr2=0x561981b9e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa94d200640 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981b9e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa94d200640 1 -- 172.21.15.146:0/3244417156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561981abd3f0 con 0x561981b96c70 2024-04-24T08:26:17.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa94d200640 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981b9e640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa9340d0d20 tx=0x7fa9340095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc531636f4655f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa92f400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa92f400640 1 -- 172.21.15.146:0/3244417156 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9340da040 con 0x561981b96c70 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa92f400640 1 -- 172.21.15.146:0/3244417156 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa9340bbb90 con 0x561981b96c70 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa93c004760 con 0x561981b96c70 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa92f400640 1 -- 172.21.15.146:0/3244417156 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa93409c510 con 0x561981b96c70 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa953428740 10 client.26414 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.888+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5619818053f0 con 0x561981b96c70 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa92f400640 1 -- 172.21.15.146:0/3244417156 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c004760 con 0x561981b96c70 2024-04-24T08:26:17.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa92f400640 1 -- 172.21.15.146:0/3244417156 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa9340870b0 con 0x561981b96c70 2024-04-24T08:26:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa953428740 10 client.26414 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa953428740 10 client.26414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa953428740 20 client.26414 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa953428740 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981bb1530 0x561981bb3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa953428740 4 client.26414 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:17.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561981b00bf0 con 0x561981bb1530 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.889+0000 7fa94dc00640 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981bb1530 0x561981bb3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.890+0000 7fa94dc00640 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981bb1530 0x561981bb3980 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a822da9d6ff483af server_cookie=12560b2ed4989d0c in_seq=0 out_seq=0 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.890+0000 7fa92f400640 10 client.26414 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa92f400640 1 -- 172.21.15.146:0/3244417156 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561981b00bf0 con 0x561981bb1530 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa92f400640 10 client.26414 handle_command_reply: tid=0 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 client.26414 shutdown 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 msgr2=0x561981b9e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981b9e640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa9340d0d20 tx=0x7fa9340095a0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 shutdown_connections 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981bb1530 0x561981bb3980 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981ba1340 0x561981b9eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981b9e640 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 --2- 172.21.15.146:0/3244417156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981b9e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa92f400640 0 client.26414 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 >> 172.21.15.146:0/3244417156 conn(0x561981b92670 msgr2=0x561981b98470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 shutdown_connections 2024-04-24T08:26:17.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.891+0000 7fa953428740 1 -- 172.21.15.146:0/3244417156 wait complete. 2024-04-24T08:26:17.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.892+0000 7fa953428740 20 client.26414 trim_cache size 0 max 16384 2024-04-24T08:26:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 Processor -- start 2024-04-24T08:26:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 -- start start 2024-04-24T08:26:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981bad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96c70 0x561981b9d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981ba1340 0x561981b9e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481ae250 con 0x561981b96350 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481abb20 con 0x561981ba1340 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa953428740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481a9990 con 0x561981b96c70 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981bad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa94d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96c70 0x561981b9d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981bad030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59484/0 (socket says 172.21.15.146:59484) 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa94dc00640 1 -- 172.21.15.146:0/3064909106 learned_addr learned my addr 172.21.15.146:0/3064909106 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.906+0000 7fa94e600640 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981ba1340 0x561981b9e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa94d200640 1 -- 172.21.15.146:0/3064909106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981ba1340 msgr2=0x561981b9e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa94d200640 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981ba1340 0x561981b9e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa94d200640 1 -- 172.21.15.146:0/3064909106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 msgr2=0x561981bad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa94d200640 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981bad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa94d200640 1 -- 172.21.15.146:0/3064909106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561981b96aa0 con 0x561981b96c70 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa94dc00640 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981bad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa94d200640 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96c70 0x561981b9d260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa934009380 tx=0x7fa934003d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d31d64af112bf77 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa93a000640 1 -- 172.21.15.146:0/3064909106 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93409c330 con 0x561981b96c70 2024-04-24T08:26:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa93a000640 1 -- 172.21.15.146:0/3064909106 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93409d0d0 con 0x561981b96c70 2024-04-24T08:26:17.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.907+0000 7fa93a000640 1 -- 172.21.15.146:0/3064909106 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9340bbb90 con 0x561981b96c70 2024-04-24T08:26:17.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 -- 172.21.15.146:0/3064909106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96c70 msgr2=0x561981b9d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96c70 0x561981b9d260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa934009380 tx=0x7fa934003d20 comp rx=0 tx=0).stop 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 -- 172.21.15.146:0/3064909106 shutdown_connections 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96c70 0x561981b9d260 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981ba1340 0x561981b9e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 --2- 172.21.15.146:0/3064909106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981b96350 0x561981bad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 -- 172.21.15.146:0/3064909106 >> 172.21.15.146:0/3064909106 conn(0x561981b92670 msgr2=0x561981b9e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 -- 172.21.15.146:0/3064909106 shutdown_connections 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 -- 172.21.15.146:0/3064909106 wait complete. 2024-04-24T08:26:17.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 -- start start 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.908+0000 7fa953428740 1 Processor -- start 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa953428740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981ba2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa953428740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa953428740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa953428740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481a9fe0 con 0x561981ba1340 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa953428740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481ab450 con 0x561981b96c70 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa953428740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481aa6b0 con 0x561981b96350 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa953428740 10 client.0 _ll_register_callbacks cb 0x7fa93400ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981ba2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981ba2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46008/0 (socket says 172.21.15.146:46008) 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94dc00640 1 -- 172.21.15.146:0/1856027164 learned_addr learned my addr 172.21.15.146:0/1856027164 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37128/0 (socket says 172.21.15.146:37128) 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94e600640 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94dc00640 1 -- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 msgr2=0x561981ba2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94dc00640 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94dc00640 1 -- 172.21.15.146:0/1856027164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 msgr2=0x561981ba2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.909+0000 7fa94dc00640 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa94dc00640 1 -- 172.21.15.146:0/1856027164 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619818cf7d0 con 0x561981b96350 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa94d200640 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa94dc00640 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981ba2050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009900 tx=0x7fa93c005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61c639e6b2ef5caa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa92f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa92f400640 1 -- 172.21.15.146:0/1856027164 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c058060 con 0x561981b96350 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa93409d0d0 con 0x561981b96350 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa92f400640 1 -- 172.21.15.146:0/1856027164 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fa93c003cf0 con 0x561981b96350 2024-04-24T08:26:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa92f400640 1 -- 172.21.15.146:0/1856027164 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa93c035040 con 0x561981b96350 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa953428740 10 client.26404 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.910+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561981c17be0 con 0x561981b96350 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa92f400640 1 -- 172.21.15.146:0/1856027164 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93409d0d0 con 0x561981b96350 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa92f400640 1 -- 172.21.15.146:0/1856027164 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa93c053310 con 0x561981b96350 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa953428740 10 client.26404 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa953428740 10 client.26404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa953428740 20 client.26404 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:17.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa953428740 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981c17f70 0x561981c1a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa953428740 4 client.26404 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561981afaa60 con 0x561981c17f70 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa94d200640 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981c17f70 0x561981c1a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.911+0000 7fa94d200640 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981c17f70 0x561981c1a390 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdb0983432f1ac71 server_cookie=7742bd6aa03a02dc in_seq=0 out_seq=0 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa92f400640 10 client.26404 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa92f400640 1 -- 172.21.15.146:0/1856027164 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561981afaa60 con 0x561981c17f70 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa92f400640 10 client.26404 handle_command_reply: tid=0 2024-04-24T08:26:17.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 client.26404 shutdown 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 msgr2=0x561981ba2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981ba2050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa93c009900 tx=0x7fa93c005770 comp rx=0 tx=0).stop 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 shutdown_connections 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x561981c17f70 0x561981c1a390 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561981ba1340 0x561981ba2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561981b96c70 0x561981ba2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa92f400640 0 client.26404 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 --2- 172.21.15.146:0/1856027164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561981b96350 0x561981ba2050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 >> 172.21.15.146:0/1856027164 conn(0x561981b92670 msgr2=0x561981b9e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 shutdown_connections 2024-04-24T08:26:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 1 -- 172.21.15.146:0/1856027164 wait complete. 2024-04-24T08:26:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.912+0000 7fa953428740 20 client.26404 trim_cache size 0 max 16384 2024-04-24T08:26:17.922 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:26:17.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9300a3d20 msgr2=0x7fa9300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9300a3d20 0x7fa9300a61e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa940005fd0 tx=0x7fa940009500 comp rx=0 tx=0).stop 2024-04-24T08:26:17.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481060d0 msgr2=0x7fa9481a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481060d0 0x7fa9481a0b80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa93400e9a0 tx=0x7fa93400ee60 comp rx=0 tx=0).stop 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 shutdown_connections 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9300a3d20 0x7fa9300a61e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa94819a7b0 0x7fa94819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9481060d0 0x7fa9481a0b80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 --2- 172.21.15.146:0/3577594376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa948101000 0x7fa9481a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 >> 172.21.15.146:0/3577594376 conn(0x7fa9480fc880 msgr2=0x7fa9480fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 shutdown_connections 2024-04-24T08:26:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:17.913+0000 7fa94f000640 1 -- 172.21.15.146:0/3577594376 wait complete. 2024-04-24T08:26:17.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:17 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/674107998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:26:18.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:17 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/674107998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:26:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:17 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/674107998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:26:18.931 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:26:18.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:26:18.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:26:19.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 -- 172.21.15.146:0/4171530559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c112d40 msgr2=0x7fcf2c115130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 --2- 172.21.15.146:0/4171530559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c112d40 0x7fcf2c115130 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009ca0 tx=0x7fcf2402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 -- 172.21.15.146:0/4171530559 shutdown_connections 2024-04-24T08:26:19.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 --2- 172.21.15.146:0/4171530559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c112d40 0x7fcf2c115130 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 --2- 172.21.15.146:0/4171530559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c076e50 0x7fcf2c112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 --2- 172.21.15.146:0/4171530559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c076530 0x7fcf2c076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 -- 172.21.15.146:0/4171530559 >> 172.21.15.146:0/4171530559 conn(0x7fcf2c0ffb10 msgr2=0x7fcf2c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 -- 172.21.15.146:0/4171530559 shutdown_connections 2024-04-24T08:26:19.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.151+0000 7fcf30e00640 1 -- 172.21.15.146:0/4171530559 wait complete. 2024-04-24T08:26:19.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 Processor -- start 2024-04-24T08:26:19.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 -- start start 2024-04-24T08:26:19.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c076530 0x7fcf2c19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c076e50 0x7fcf2c19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c112d40 0x7fcf2c1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c118860 con 0x7fcf2c076530 2024-04-24T08:26:19.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1186e0 con 0x7fcf2c112d40 2024-04-24T08:26:19.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.152+0000 7fcf30e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1189e0 con 0x7fcf2c076e50 2024-04-24T08:26:19.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c076530 0x7fcf2c19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c076530 0x7fcf2c19ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59512/0 (socket says 172.21.15.146:59512) 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 -- 172.21.15.146:0/3748523406 learned_addr learned my addr 172.21.15.146:0/3748523406 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c076e50 0x7fcf2c19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2be00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c112d40 0x7fcf2c1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 -- 172.21.15.146:0/3748523406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c076e50 msgr2=0x7fcf2c19f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c076e50 0x7fcf2c19f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 -- 172.21.15.146:0/3748523406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c112d40 msgr2=0x7fcf2c1a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c112d40 0x7fcf2c1a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.153+0000 7fcf2b400640 1 -- 172.21.15.146:0/3748523406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf2c1a3d30 con 0x7fcf2c076530 2024-04-24T08:26:19.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.154+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c076530 0x7fcf2c19ed40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00e9f0 tx=0x7fcf1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.154+0000 7fcf28c00640 1 -- 172.21.15.146:0/3748523406 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00cd80 con 0x7fcf2c076530 2024-04-24T08:26:19.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.154+0000 7fcf28c00640 1 -- 172.21.15.146:0/3748523406 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf1c010910 con 0x7fcf2c076530 2024-04-24T08:26:19.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.154+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf2c1a4020 con 0x7fcf2c076530 2024-04-24T08:26:19.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.154+0000 7fcf28c00640 1 -- 172.21.15.146:0/3748523406 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c018990 con 0x7fcf2c076530 2024-04-24T08:26:19.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.154+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf2c1ab8b0 con 0x7fcf2c076530 2024-04-24T08:26:19.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.157+0000 7fcf35216740 1 Processor -- start 2024-04-24T08:26:19.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.157+0000 7fcf28c00640 1 -- 172.21.15.146:0/3748523406 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fcf1c010ac0 con 0x7fcf2c076530 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf35216740 1 -- start start 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf35216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972d1270 0x5623972d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf35216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1b90 0x5623972dbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf35216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972dc260 0x5623972de650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf35216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x562397236e70 con 0x5623972d1270 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf35216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5623971dd380 con 0x5623972dc260 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf35216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562397232e30 con 0x5623972d1b90 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1b90 0x5623972dbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972d1270 0x5623972d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1b90 0x5623972dbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46034/0 (socket says 172.21.15.146:46034) 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972d1270 0x5623972d1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59522/0 (socket says 172.21.15.146:59522) 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf2b400640 1 -- 172.21.15.146:0/3728346182 learned_addr learned my addr 172.21.15.146:0/3728346182 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.158+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972dc260 0x5623972de650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf23e00640 1 -- 172.21.15.146:0/3728346182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1b90 msgr2=0x5623972dbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1b90 0x5623972dbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf23e00640 1 -- 172.21.15.146:0/3728346182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972d1270 msgr2=0x5623972d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972d1270 0x5623972d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf23e00640 1 -- 172.21.15.146:0/3728346182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623970117e0 con 0x5623972dc260 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1b90 0x5623972dbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972dc260 0x5623972de650 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009bf0 tx=0x7fcf14011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ec0ba0802e9461a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf28c00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcf100a3d20 0x7fcf100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf20c00640 1 -- 172.21.15.146:0/3728346182 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14004310 con 0x5623972dc260 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf20c00640 1 -- 172.21.15.146:0/3728346182 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf140044b0 con 0x5623972dc260 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.159+0000 7fcf20c00640 1 -- 172.21.15.146:0/3728346182 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14019a80 con 0x5623972dc260 2024-04-24T08:26:19.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf28c00640 1 -- 172.21.15.146:0/3748523406 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcf1c0d0b70 con 0x7fcf2c076530 2024-04-24T08:26:19.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf2be00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcf100a3d20 0x7fcf100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf35216740 1 -- 172.21.15.146:0/3728346182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972dc260 msgr2=0x5623972de650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf35216740 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972dc260 0x5623972de650 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009bf0 tx=0x7fcf14011440 comp rx=0 tx=0).stop 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf2be00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcf100a3d20 0x7fcf100a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcf24005e30 tx=0x7fcf24009360 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf35216740 1 -- 172.21.15.146:0/3728346182 shutdown_connections 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf35216740 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1b90 0x5623972dbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf35216740 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972dc260 0x5623972de650 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf35216740 1 --2- 172.21.15.146:0/3728346182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972d1270 0x5623972d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.160+0000 7fcf35216740 1 -- 172.21.15.146:0/3728346182 >> 172.21.15.146:0/3728346182 conn(0x5623972cd590 msgr2=0x5623972cd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.161+0000 7fcf35216740 1 -- 172.21.15.146:0/3728346182 shutdown_connections 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.161+0000 7fcf35216740 1 -- 172.21.15.146:0/3728346182 wait complete. 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.161+0000 7fcf35216740 1 -- start start 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.161+0000 7fcf35216740 1 Processor -- start 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf35216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972d9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf35216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf35216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972d9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf35216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5623972df050 con 0x5623972dc260 2024-04-24T08:26:19.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf35216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5623971f0660 con 0x5623972d1b90 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf35216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5623972df1d0 con 0x5623972d1270 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf35216740 10 client.0 _ll_register_callbacks cb 0x5623971deaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972d9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972d9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972d9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46040/0 (socket says 172.21.15.146:46040) 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf23e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 -- 172.21.15.146:0/1655662680 learned_addr learned my addr 172.21.15.146:0/1655662680 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 -- 172.21.15.146:0/1655662680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 msgr2=0x5623972d96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 -- 172.21.15.146:0/1655662680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 msgr2=0x5623972d9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972d9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.162+0000 7fcf2b400640 1 -- 172.21.15.146:0/1655662680 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562397071f70 con 0x5623972d1270 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf2b400640 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972d9160 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c098fd0 tx=0x7fcf1c09c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5fa172a5428ddc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf19600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf19600640 1 -- 172.21.15.146:0/1655662680 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c09cb70 con 0x5623972d1270 2024-04-24T08:26:19.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf19600640 1 -- 172.21.15.146:0/1655662680 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcf1c0ce4f0 con 0x5623972d1270 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf140044b0 con 0x5623972d1270 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf19600640 1 -- 172.21.15.146:0/1655662680 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcf1c0ce6f0 con 0x5623972d1270 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf35216740 10 client.26416 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562396f09170 con 0x5623972d1270 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.163+0000 7fcf19600640 1 -- 172.21.15.146:0/1655662680 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf140044b0 con 0x5623972d1270 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf19600640 1 -- 172.21.15.146:0/1655662680 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fcf1c09bfc0 con 0x5623972d1270 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf35216740 10 client.26416 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf35216740 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:19.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf35216740 20 client.26416 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf35216740 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5623972ec4c0 0x5623972ee910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf35216740 4 client.26416 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562397244890 con 0x5623972ec4c0 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.164+0000 7fcf23e00640 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5623972ec4c0 0x5623972ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.165+0000 7fcf23e00640 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5623972ec4c0 0x5623972ee910 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6df9177c4106f6e4 server_cookie=91dffde949b0432c in_seq=0 out_seq=0 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.165+0000 7fcf19600640 10 client.26416 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf19600640 1 -- 172.21.15.146:0/1655662680 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562397244890 con 0x5623972ec4c0 2024-04-24T08:26:19.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf19600640 10 client.26416 handle_command_reply: tid=0 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 client.26416 shutdown 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 msgr2=0x5623972d9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972d9160 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c098fd0 tx=0x7fcf1c09c940 comp rx=0 tx=0).stop 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 shutdown_connections 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5623972ec4c0 0x5623972ee910 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972d9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf19600640 0 client.26416 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 --2- 172.21.15.146:0/1655662680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972d9160 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 >> 172.21.15.146:0/1655662680 conn(0x5623972cd590 msgr2=0x5623972d3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 shutdown_connections 2024-04-24T08:26:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 1 -- 172.21.15.146:0/1655662680 wait complete. 2024-04-24T08:26:19.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.166+0000 7fcf35216740 20 client.26416 trim_cache size 0 max 16384 2024-04-24T08:26:19.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.185+0000 7fcf35216740 1 Processor -- start 2024-04-24T08:26:19.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.185+0000 7fcf35216740 1 -- start start 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf35216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf35216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf35216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf35216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1b6b10 con 0x5623972dc260 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf35216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1b43e0 con 0x5623972d1b90 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf35216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1b2230 con 0x5623972d1270 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf23e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972f41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59538/0 (socket says 172.21.15.146:59538) 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf2be00640 1 -- 172.21.15.146:0/3174957814 learned_addr learned my addr 172.21.15.146:0/3174957814 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf23e00640 1 -- 172.21.15.146:0/3174957814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 msgr2=0x5623972e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf23e00640 1 -- 172.21.15.146:0/3174957814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 msgr2=0x5623972f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.186+0000 7fcf23e00640 1 -- 172.21.15.146:0/3174957814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623972d2500 con 0x5623972d1b90 2024-04-24T08:26:19.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.187+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d30d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf14011940 tx=0x7fcf14019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc965ff016cbdbb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.187+0000 7fcf1be00640 1 -- 172.21.15.146:0/3174957814 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1401f040 con 0x5623972d1b90 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.187+0000 7fcf1be00640 1 -- 172.21.15.146:0/3174957814 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf14019450 con 0x5623972d1b90 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.187+0000 7fcf1be00640 1 -- 172.21.15.146:0/3174957814 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14003940 con 0x5623972d1b90 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 -- 172.21.15.146:0/3174957814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 msgr2=0x5623972d30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d30d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf14011940 tx=0x7fcf14019740 comp rx=0 tx=0).stop 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 -- 172.21.15.146:0/3174957814 shutdown_connections 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972d30d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 --2- 172.21.15.146:0/3174957814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 -- 172.21.15.146:0/3174957814 >> 172.21.15.146:0/3174957814 conn(0x5623972cd590 msgr2=0x5623972d29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 -- 172.21.15.146:0/3174957814 shutdown_connections 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 -- 172.21.15.146:0/3174957814 wait complete. 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 -- start start 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.188+0000 7fcf35216740 1 Processor -- start 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf35216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf35216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972e4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf35216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972e1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf35216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1b28a0 con 0x5623972dc260 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf35216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1b3d10 con 0x5623972d1b90 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf35216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf2c1b2f70 con 0x5623972d1270 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf35216740 10 client.0 _ll_register_callbacks cb 0x7fcf1c0d4310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf23e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46054/0 (socket says 172.21.15.146:46054) 2024-04-24T08:26:19.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf2b400640 1 -- 172.21.15.146:0/3680510214 learned_addr learned my addr 172.21.15.146:0/3680510214 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf2be00640 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972e1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf2b400640 1 -- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 msgr2=0x5623972e4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972e4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.189+0000 7fcf2b400640 1 -- 172.21.15.146:0/3680510214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 msgr2=0x5623972e1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972e1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf2b400640 1 -- 172.21.15.146:0/3680510214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623972e1b60 con 0x5623972d1270 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972e4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:19.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf2b400640 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e4450 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c09ae20 tx=0x7fcf1c09ca70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b326d9482a6d6bc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf19600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf19600640 1 -- 172.21.15.146:0/3680510214 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c09c6b0 con 0x5623972d1270 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf19600640 1 -- 172.21.15.146:0/3680510214 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fcf1c0988c0 con 0x5623972d1270 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf14019450 con 0x5623972d1270 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf19600640 1 -- 172.21.15.146:0/3680510214 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcf1c0a0e70 con 0x5623972d1270 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf35216740 10 client.26422 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5623973534a0 con 0x5623972d1270 2024-04-24T08:26:19.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.190+0000 7fcf19600640 1 -- 172.21.15.146:0/3680510214 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf14019450 con 0x5623972d1270 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf19600640 1 -- 172.21.15.146:0/3680510214 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fcf1c09ba30 con 0x5623972d1270 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf35216740 10 client.26422 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf35216740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf35216740 20 client.26422 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf35216740 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x562397353c20 0x562397356040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf35216740 4 client.26422 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56239723e700 con 0x562397353c20 2024-04-24T08:26:19.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.191+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x562397353c20 0x562397356040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.192+0000 7fcf19600640 10 client.26422 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.192+0000 7fcf23e00640 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x562397353c20 0x562397356040 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa2ee44ab062d8ca server_cookie=ce58051bb011f098 in_seq=0 out_seq=0 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf19600640 1 -- 172.21.15.146:0/3680510214 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56239723e700 con 0x562397353c20 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf19600640 10 client.26422 handle_command_reply: tid=0 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 client.26422 shutdown 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 msgr2=0x5623972e4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e4450 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c09ae20 tx=0x7fcf1c09ca70 comp rx=0 tx=0).stop 2024-04-24T08:26:19.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 shutdown_connections 2024-04-24T08:26:19.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x562397353c20 0x562397356040 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5623972dc260 0x5623972e1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5623972d1b90 0x5623972e4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 --2- 172.21.15.146:0/3680510214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5623972d1270 0x5623972e4450 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf19600640 0 client.26422 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 >> 172.21.15.146:0/3680510214 conn(0x5623972cd590 msgr2=0x5623972d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 shutdown_connections 2024-04-24T08:26:19.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 1 -- 172.21.15.146:0/3680510214 wait complete. 2024-04-24T08:26:19.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.193+0000 7fcf35216740 20 client.26422 trim_cache size 0 max 16384 2024-04-24T08:26:19.203 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:26:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:26:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "7b95b4fc-47c7-4abc-bb52-130c9bacdae3", 2024-04-24T08:26:19.203 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:26:19.203 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:26:19.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcf100a3d20 msgr2=0x7fcf100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcf100a3d20 0x7fcf100a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcf24005e30 tx=0x7fcf24009360 comp rx=0 tx=0).stop 2024-04-24T08:26:19.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c076530 msgr2=0x7fcf2c19ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c076530 0x7fcf2c19ed40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00e9f0 tx=0x7fcf1c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 shutdown_connections 2024-04-24T08:26:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcf100a3d20 0x7fcf100a61e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf2c076e50 0x7fcf2c19f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf2c112d40 0x7fcf2c1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 --2- 172.21.15.146:0/3748523406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf2c076530 0x7fcf2c19ed40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 >> 172.21.15.146:0/3748523406 conn(0x7fcf2c0ffb10 msgr2=0x7fcf2c1147e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 shutdown_connections 2024-04-24T08:26:19.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.194+0000 7fcf30e00640 1 -- 172.21.15.146:0/3748523406 wait complete. 2024-04-24T08:26:19.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:18 smithi153 ceph-mon[32746]: pgmap v156: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:26:19.214 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:18 smithi146 ceph-mon[27805]: pgmap v156: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:26:19.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:26:19.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:26:19.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:18 smithi177 ceph-mon[33046]: pgmap v156: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:26:19.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.470+0000 7f6a4a200640 1 -- 172.21.15.146:0/1755054128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44108610 msgr2=0x7f6a44108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.470+0000 7f6a4a200640 1 --2- 172.21.15.146:0/1755054128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44108610 0x7f6a44108a90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009900 tx=0x7f6a3002a400 comp rx=0 tx=0).stop 2024-04-24T08:26:19.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.470+0000 7f6a4a200640 1 -- 172.21.15.146:0/1755054128 shutdown_connections 2024-04-24T08:26:19.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.470+0000 7f6a4a200640 1 --2- 172.21.15.146:0/1755054128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a44108fd0 0x7f6a44115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.470+0000 7f6a4a200640 1 --2- 172.21.15.146:0/1755054128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a44108610 0x7f6a44108a90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.470+0000 7f6a4a200640 1 --2- 172.21.15.146:0/1755054128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a4410e660 0x7f6a4410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.470+0000 7f6a4a200640 1 -- 172.21.15.146:0/1755054128 >> 172.21.15.146:0/1755054128 conn(0x7f6a44076540 msgr2=0x7f6a44078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.471+0000 7f6a4a200640 1 -- 172.21.15.146:0/1755054128 shutdown_connections 2024-04-24T08:26:19.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.471+0000 7f6a4a200640 1 -- 172.21.15.146:0/1755054128 wait complete. 2024-04-24T08:26:19.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.472+0000 7f6a4a200640 1 Processor -- start 2024-04-24T08:26:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.472+0000 7f6a4a200640 1 -- start start 2024-04-24T08:26:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.472+0000 7f6a4a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a44108610 0x7f6a4419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.472+0000 7f6a4a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a44108fd0 0x7f6a4419f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.472+0000 7f6a4a200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a4410e660 0x7f6a441a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.472+0000 7f6a4a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441189f0 con 0x7f6a44108610 2024-04-24T08:26:19.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.473+0000 7f6a4a200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a44118870 con 0x7f6a4410e660 2024-04-24T08:26:19.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.473+0000 7f6a4a200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6a44118b70 con 0x7f6a44108fd0 2024-04-24T08:26:19.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.473+0000 7f6a43e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a4410e660 0x7f6a441a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.473+0000 7f6a43e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a4410e660 0x7f6a441a3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37208/0 (socket says 172.21.15.146:37208) 2024-04-24T08:26:19.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.473+0000 7f6a43e00640 1 -- 172.21.15.146:0/3367095097 learned_addr learned my addr 172.21.15.146:0/3367095097 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.473+0000 7f6a42a00640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a44108fd0 0x7f6a4419f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.473+0000 7f6a43400640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a44108610 0x7f6a4419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a42a00640 1 -- 172.21.15.146:0/3367095097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a4410e660 msgr2=0x7f6a441a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a42a00640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a4410e660 0x7f6a441a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a42a00640 1 -- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a44108610 msgr2=0x7f6a4419ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a42a00640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a44108610 0x7f6a4419ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a42a00640 1 -- 172.21.15.146:0/3367095097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a441a3e10 con 0x7f6a44108fd0 2024-04-24T08:26:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a42a00640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a44108fd0 0x7f6a4419f390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009230 tx=0x7f6a30003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a3be00640 1 -- 172.21.15.146:0/3367095097 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3003f070 con 0x7f6a44108fd0 2024-04-24T08:26:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.474+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a441a40a0 con 0x7f6a44108fd0 2024-04-24T08:26:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.475+0000 7f6a3be00640 1 -- 172.21.15.146:0/3367095097 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a30031c20 con 0x7f6a44108fd0 2024-04-24T08:26:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.475+0000 7f6a3be00640 1 -- 172.21.15.146:0/3367095097 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30035030 con 0x7f6a44108fd0 2024-04-24T08:26:19.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.475+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4406ba20 con 0x7f6a44108fd0 2024-04-24T08:26:19.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.478+0000 7f6a3be00640 1 -- 172.21.15.146:0/3367095097 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f6a30042070 con 0x7f6a44108fd0 2024-04-24T08:26:19.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.480+0000 7f6a3be00640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a280a3d20 0x7f6a280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.480+0000 7f6a3be00640 1 -- 172.21.15.146:0/3367095097 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6a300f0b90 con 0x7f6a44108fd0 2024-04-24T08:26:19.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.481+0000 7f6a43400640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a280a3d20 0x7f6a280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.482+0000 7f6a43400640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a280a3d20 0x7f6a280a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005fd0 tx=0x7f6a3400b410 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.483+0000 7f6a4cf88740 1 Processor -- start 2024-04-24T08:26:19.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.483+0000 7f6a4cf88740 1 -- start start 2024-04-24T08:26:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec691c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec69c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55baec5f71f0 con 0x55baec691860 2024-04-24T08:26:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55baec59d350 con 0x55baec69c850 2024-04-24T08:26:19.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55baec5f31b0 con 0x55baec692180 2024-04-24T08:26:19.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec69c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec69c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46076/0 (socket says 172.21.15.146:46076) 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a42a00640 1 -- 172.21.15.146:0/1921574857 learned_addr learned my addr 172.21.15.146:0/1921574857 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a43e00640 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.484+0000 7f6a43400640 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec691c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.485+0000 7f6a42a00640 1 -- 172.21.15.146:0/1921574857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 msgr2=0x55baec69ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.485+0000 7f6a42a00640 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.485+0000 7f6a42a00640 1 -- 172.21.15.146:0/1921574857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 msgr2=0x55baec691c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.485+0000 7f6a42a00640 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec691c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.485+0000 7f6a42a00640 1 -- 172.21.15.146:0/1921574857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baec5f1f50 con 0x55baec692180 2024-04-24T08:26:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.485+0000 7f6a43400640 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec691c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.485+0000 7f6a42a00640 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec69c310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6a300b8ff0 tx=0x7f6a30003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=494508c001701941 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.486+0000 7f6a2fe00640 1 -- 172.21.15.146:0/1921574857 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30049050 con 0x55baec692180 2024-04-24T08:26:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.486+0000 7f6a2fe00640 1 -- 172.21.15.146:0/1921574857 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a300c5750 con 0x55baec692180 2024-04-24T08:26:19.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.486+0000 7f6a2fe00640 1 -- 172.21.15.146:0/1921574857 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a300b9200 con 0x55baec692180 2024-04-24T08:26:19.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 -- 172.21.15.146:0/1921574857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 msgr2=0x55baec69c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec69c310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6a300b8ff0 tx=0x7f6a30003f90 comp rx=0 tx=0).stop 2024-04-24T08:26:19.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 -- 172.21.15.146:0/1921574857 shutdown_connections 2024-04-24T08:26:19.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec69c310 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/1921574857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec691c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 -- 172.21.15.146:0/1921574857 >> 172.21.15.146:0/1921574857 conn(0x55baec68db80 msgr2=0x55baec68df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 -- 172.21.15.146:0/1921574857 shutdown_connections 2024-04-24T08:26:19.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.487+0000 7f6a4cf88740 1 -- 172.21.15.146:0/1921574857 wait complete. 2024-04-24T08:26:19.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.488+0000 7f6a4cf88740 1 -- start start 2024-04-24T08:26:19.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.488+0000 7f6a4cf88740 1 Processor -- start 2024-04-24T08:26:19.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.489+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec691860 0x55baec699880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.489+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec699dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.489+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec69a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.489+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55baec69f400 con 0x55baec69c850 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.489+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55baec5b0690 con 0x55baec691860 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.489+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55baec69f580 con 0x55baec692180 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.489+0000 7f6a4cf88740 10 client.0 _ll_register_callbacks cb 0x55baec5ec650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.490+0000 7f6a43400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec691860 0x55baec699880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.490+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec699dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.490+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec699dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46084/0 (socket says 172.21.15.146:46084) 2024-04-24T08:26:19.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.490+0000 7f6a42a00640 1 -- 172.21.15.146:0/836029733 learned_addr learned my addr 172.21.15.146:0/836029733 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.490+0000 7f6a43e00640 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec69a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a43400640 1 -- 172.21.15.146:0/836029733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 msgr2=0x55baec699dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a43400640 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec699dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a43400640 1 -- 172.21.15.146:0/836029733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 msgr2=0x55baec69a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a43400640 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec69a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a43400640 1 -- 172.21.15.146:0/836029733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baec5af920 con 0x55baec691860 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a42a00640 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec699dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a43400640 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec691860 0x55baec699880 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6a34003e90 tx=0x7f6a34004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5664179a313d710 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a2cc00640 1 -- 172.21.15.146:0/836029733 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3401f640 con 0x55baec691860 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a2cc00640 1 -- 172.21.15.146:0/836029733 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6a3401f7e0 con 0x55baec691860 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.491+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a300c5750 con 0x55baec691860 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.492+0000 7f6a2cc00640 1 -- 172.21.15.146:0/836029733 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6a3401f9e0 con 0x55baec691860 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.492+0000 7f6a4cf88740 10 client.26447 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:19.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.492+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55baec3d37d0 con 0x55baec691860 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.492+0000 7f6a2cc00640 1 -- 172.21.15.146:0/836029733 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a300c5750 con 0x55baec691860 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.492+0000 7f6a2cc00640 1 -- 172.21.15.146:0/836029733 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6a34028470 con 0x55baec691860 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.492+0000 7f6a4cf88740 10 client.26447 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.493+0000 7f6a4cf88740 10 client.26447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.493+0000 7f6a4cf88740 20 client.26447 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.493+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec6ac990 0x55baec6aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.493+0000 7f6a4cf88740 4 client.26447 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:19.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.493+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baec604c10 con 0x55baec6ac990 2024-04-24T08:26:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.493+0000 7f6a42a00640 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec6ac990 0x55baec6aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.494+0000 7f6a42a00640 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec6ac990 0x55baec6aedc0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b771aea1b33c2a30 server_cookie=8233cadd23ca0bfb in_seq=0 out_seq=0 2024-04-24T08:26:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.494+0000 7f6a2cc00640 10 client.26447 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.495+0000 7f6a2cc00640 1 -- 172.21.15.146:0/836029733 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55baec604c10 con 0x55baec6ac990 2024-04-24T08:26:19.498 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.495+0000 7f6a2cc00640 10 client.26447 handle_command_reply: tid=0 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.495+0000 7f6a4cf88740 1 client.26447 shutdown 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec691860 msgr2=0x55baec699880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec691860 0x55baec699880 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6a34003e90 tx=0x7f6a34004730 comp rx=0 tx=0).stop 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 shutdown_connections 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec6ac990 0x55baec6aedc0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec699dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec69a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/836029733 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec691860 0x55baec699880 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 >> 172.21.15.146:0/836029733 conn(0x55baec68db80 msgr2=0x55baec693980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.496+0000 7f6a2cc00640 0 client.26447 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.497+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 shutdown_connections 2024-04-24T08:26:19.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.497+0000 7f6a4cf88740 1 -- 172.21.15.146:0/836029733 wait complete. 2024-04-24T08:26:19.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.497+0000 7f6a4cf88740 20 client.26447 trim_cache size 0 max 16384 2024-04-24T08:26:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 Processor -- start 2024-04-24T08:26:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 -- start start 2024-04-24T08:26:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec68df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec6a8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441bf470 con 0x55baec691860 2024-04-24T08:26:19.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441bcd40 con 0x55baec69c850 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.521+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6a440723f0 con 0x55baec692180 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec6a8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a43400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec68df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec6a8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46094/0 (socket says 172.21.15.146:46094) 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 -- 172.21.15.146:0/4225484830 learned_addr learned my addr 172.21.15.146:0/4225484830 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a43e00640 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 -- 172.21.15.146:0/4225484830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 msgr2=0x55baec69b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 -- 172.21.15.146:0/4225484830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 msgr2=0x55baec68df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec68df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 -- 172.21.15.146:0/4225484830 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baec69efb0 con 0x55baec692180 2024-04-24T08:26:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.522+0000 7f6a42a00640 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec6a8210 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a300092a0 tx=0x7f6a3002a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47987ec2baf37bc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a2f400640 1 -- 172.21.15.146:0/4225484830 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30049040 con 0x55baec692180 2024-04-24T08:26:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a2f400640 1 -- 172.21.15.146:0/4225484830 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a300bae90 con 0x55baec692180 2024-04-24T08:26:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a2f400640 1 -- 172.21.15.146:0/4225484830 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a300c5520 con 0x55baec692180 2024-04-24T08:26:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a4cf88740 1 -- 172.21.15.146:0/4225484830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 msgr2=0x55baec6a8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec6a8210 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a300092a0 tx=0x7f6a3002a400 comp rx=0 tx=0).stop 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a4cf88740 1 -- 172.21.15.146:0/4225484830 shutdown_connections 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec692180 0x55baec6a8210 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec69c850 0x55baec69b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/4225484830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec691860 0x55baec68df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.523+0000 7f6a4cf88740 1 -- 172.21.15.146:0/4225484830 >> 172.21.15.146:0/4225484830 conn(0x55baec68db80 msgr2=0x55baec69b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.524+0000 7f6a4cf88740 1 -- 172.21.15.146:0/4225484830 shutdown_connections 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.524+0000 7f6a4cf88740 1 -- 172.21.15.146:0/4225484830 wait complete. 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.524+0000 7f6a4cf88740 1 -- start start 2024-04-24T08:26:19.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.524+0000 7f6a4cf88740 1 Processor -- start 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec691860 0x55baec6969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec692180 0x55baec696f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a4cf88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec711f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6a44072a60 con 0x55baec69c850 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a44073ed0 con 0x55baec692180 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a4cf88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6a44073130 con 0x55baec691860 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a4cf88740 10 client.0 _ll_register_callbacks cb 0x7f6a34003870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:19.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec692180 0x55baec696f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a43e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec711f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec692180 0x55baec696f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37244/0 (socket says 172.21.15.146:37244) 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 -- 172.21.15.146:0/242961894 learned_addr learned my addr 172.21.15.146:0/242961894 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a43400640 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec691860 0x55baec6969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 -- 172.21.15.146:0/242961894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec691860 msgr2=0x55baec6969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec691860 0x55baec6969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 -- 172.21.15.146:0/242961894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 msgr2=0x55baec711f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:19.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec711f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 -- 172.21.15.146:0/242961894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baec3093f0 con 0x55baec692180 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a42a00640 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec692180 0x55baec696f20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6a30003fd0 tx=0x7f6a30005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f23c1090c1f73e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.525+0000 7f6a2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a2cc00640 1 -- 172.21.15.146:0/242961894 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30049070 con 0x55baec692180 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a2cc00640 1 -- 172.21.15.146:0/242961894 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6a300c54c0 con 0x55baec692180 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a300bae90 con 0x55baec692180 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a2cc00640 1 -- 172.21.15.146:0/242961894 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6a300c5750 con 0x55baec692180 2024-04-24T08:26:19.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 10 client.26453 fetch_fsmap learned FSMap version 25 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55baec2c90f0 con 0x55baec692180 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a2cc00640 1 -- 172.21.15.146:0/242961894 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a300bae90 con 0x55baec692180 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a2cc00640 1 -- 172.21.15.146:0/242961894 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6a3004c060 con 0x55baec692180 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 10 client.26453 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 10 client.26453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 20 client.26453 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec713530 0x55baec715920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:19.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 4 client.26453 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.526+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55baec5fea80 con 0x55baec713530 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.527+0000 7f6a43400640 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec713530 0x55baec715920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.527+0000 7f6a43400640 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec713530 0x55baec715920 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=412afe1156716d5b server_cookie=60aa17c3fb559468 in_seq=0 out_seq=0 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.527+0000 7f6a2cc00640 10 client.26453 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.527+0000 7f6a2cc00640 1 -- 172.21.15.146:0/242961894 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55baec5fea80 con 0x55baec713530 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.527+0000 7f6a2cc00640 10 client.26453 handle_command_reply: tid=0 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.527+0000 7f6a4cf88740 1 client.26453 shutdown 2024-04-24T08:26:19.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec692180 msgr2=0x55baec696f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec692180 0x55baec696f20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6a30003fd0 tx=0x7f6a30005de0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 shutdown_connections 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55baec713530 0x55baec715920 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55baec69c850 0x55baec711f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55baec692180 0x55baec696f20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a2cc00640 0 client.26453 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 --2- 172.21.15.146:0/242961894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec691860 0x55baec6969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 >> 172.21.15.146:0/242961894 conn(0x55baec68db80 msgr2=0x55baec69b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 shutdown_connections 2024-04-24T08:26:19.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 1 -- 172.21.15.146:0/242961894 wait complete. 2024-04-24T08:26:19.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4cf88740 20 client.26453 trim_cache size 0 max 16384 2024-04-24T08:26:19.538 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:26:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:26:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:26:19.538 INFO:teuthology.orchestra.run.smithi146.stdout: "7b95b4fc-47c7-4abc-bb52-130c9bacdae3": { 2024-04-24T08:26:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:26:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "7b95b4fc-47c7-4abc-bb52-130c9bacdae3", 2024-04-24T08:26:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:26:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:26:19.539 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:26:19.539 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:26:19.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.528+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a280a3d20 msgr2=0x7f6a280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a280a3d20 0x7f6a280a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005fd0 tx=0x7f6a3400b410 comp rx=0 tx=0).stop 2024-04-24T08:26:19.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a44108fd0 msgr2=0x7f6a4419f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:19.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a44108fd0 0x7f6a4419f390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009230 tx=0x7f6a30003b90 comp rx=0 tx=0).stop 2024-04-24T08:26:19.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 shutdown_connections 2024-04-24T08:26:19.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6a280a3d20 0x7f6a280a61e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a44108fd0 0x7f6a4419f390 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a4410e660 0x7f6a441a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 --2- 172.21.15.146:0/3367095097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a44108610 0x7f6a4419ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:19.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 >> 172.21.15.146:0/3367095097 conn(0x7f6a44076540 msgr2=0x7f6a44077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:19.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 shutdown_connections 2024-04-24T08:26:19.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:19.529+0000 7f6a4a200640 1 -- 172.21.15.146:0/3367095097 wait complete. 2024-04-24T08:26:19.547 INFO:tasks.cephfs.filesystem:scrub status for tag:7b95b4fc-47c7-4abc-bb52-130c9bacdae3 - {'path': '/', 'tag': '7b95b4fc-47c7-4abc-bb52-130c9bacdae3', 'options': 'recursive,force'} 2024-04-24T08:26:20.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:19 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:26:20.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:19 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:20.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:19 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:26:20.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:19 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:26:20.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:19 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:20.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:19 smithi153 ceph-mon[32746]: pgmap v157: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:26:20.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:19 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:26:20.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:19 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:26:20.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:19 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:20.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:19 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:26:20.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:19 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:26:20.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:19 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:20.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:19 smithi146 ceph-mon[27805]: pgmap v157: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:26:20.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:19 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:26:20.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:19 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:26:20.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:19 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:20.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:19 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:26:20.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:19 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:26:20.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:19 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:20.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:19 smithi177 ceph-mon[33046]: pgmap v157: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:26:20.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:19 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:26:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:20 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:20 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:21.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:20 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:21.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:20 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:20 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:20 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:22.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:21 smithi153 ceph-mon[32746]: pgmap v158: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:26:22.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:21 smithi146 ceph-mon[27805]: pgmap v158: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:26:22.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:21 smithi177 ceph-mon[33046]: pgmap v158: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:26:24.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:24 smithi177 ceph-mon[33046]: pgmap v159: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:26:24.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:24 smithi153 ceph-mon[32746]: pgmap v159: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:26:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:24 smithi146 ceph-mon[27805]: pgmap v159: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:26:26.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:26 smithi177 ceph-mon[33046]: pgmap v160: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:26.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:26 smithi153 ceph-mon[32746]: pgmap v160: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:26 smithi146 ceph-mon[27805]: pgmap v160: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:27.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:27 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:26:27.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:27 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:26:27.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:27 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:26:27.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:27 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:26:27.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:27 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:26:27.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:27 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:26:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:28 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:26:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:28 smithi177 ceph-mon[33046]: pgmap v161: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:26:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:28 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:26:28.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:28 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:26:28.566 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:28 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:26:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:28 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:26:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:28 smithi153 ceph-mon[32746]: pgmap v161: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:26:28.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:28 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:26:28.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:28 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:26:28.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:28 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:26:28.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:28 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:26:28.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:28 smithi146 ceph-mon[27805]: pgmap v161: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:26:28.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:28 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:26:28.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:28 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:26:28.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:28 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:26:30.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:30 smithi153 ceph-mon[32746]: pgmap v162: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:30.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:30 smithi146 ceph-mon[27805]: pgmap v162: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:30 smithi177 ceph-mon[33046]: pgmap v162: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:32.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:32 smithi153 ceph-mon[32746]: pgmap v163: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:32.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:32 smithi146 ceph-mon[27805]: pgmap v163: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:32 smithi177 ceph-mon[33046]: pgmap v163: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:26:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:34 smithi153 ceph-mon[32746]: pgmap v164: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:26:34.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:34 smithi146 ceph-mon[27805]: pgmap v164: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:26:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:34 smithi177 ceph-mon[33046]: pgmap v164: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:26:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:36 smithi153 ceph-mon[32746]: pgmap v165: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:36.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:36 smithi146 ceph-mon[27805]: pgmap v165: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:36 smithi177 ceph-mon[33046]: pgmap v165: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:38.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:38 smithi153 ceph-mon[32746]: pgmap v166: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:38.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:38 smithi146 ceph-mon[27805]: pgmap v166: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:38 smithi177 ceph-mon[33046]: pgmap v166: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:40 smithi153 ceph-mon[32746]: pgmap v167: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:40.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:40 smithi146 ceph-mon[27805]: pgmap v167: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:40 smithi177 ceph-mon[33046]: pgmap v167: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:42.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:42 smithi153 ceph-mon[32746]: pgmap v168: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:42.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:42 smithi146 ceph-mon[27805]: pgmap v168: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:42 smithi177 ceph-mon[33046]: pgmap v168: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:43.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:43 smithi153 ceph-mon[32746]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:26:43.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:43 smithi146 ceph-mon[27805]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:26:43.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:43 smithi177 ceph-mon[33046]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:26:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:44 smithi153 ceph-mon[32746]: pgmap v169: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:44.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:44 smithi146 ceph-mon[27805]: pgmap v169: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:44 smithi177 ceph-mon[33046]: pgmap v169: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:46.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:46 smithi153 ceph-mon[32746]: pgmap v170: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:46 smithi146 ceph-mon[27805]: pgmap v170: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:46 smithi177 ceph-mon[33046]: pgmap v170: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:48.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:48 smithi153 ceph-mon[32746]: pgmap v171: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:48 smithi146 ceph-mon[27805]: pgmap v171: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:48 smithi177 ceph-mon[33046]: pgmap v171: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:49.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:26:49.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:26:49.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 -- 172.21.15.146:0/3686864208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584104e00 msgr2=0x7fc584108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 --2- 172.21.15.146:0/3686864208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584104e00 0x7fc584108e40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009900 tx=0x7fc56c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 -- 172.21.15.146:0/3686864208 shutdown_connections 2024-04-24T08:26:49.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 --2- 172.21.15.146:0/3686864208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584104e00 0x7fc584108e40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 --2- 172.21.15.146:0/3686864208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc584109520 0x7fc5841110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 --2- 172.21.15.146:0/3686864208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 0x7fc584104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 -- 172.21.15.146:0/3686864208 >> 172.21.15.146:0/3686864208 conn(0x7fc5840ffc70 msgr2=0x7fc584102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:49.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 -- 172.21.15.146:0/3686864208 shutdown_connections 2024-04-24T08:26:49.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.764+0000 7fc588a00640 1 -- 172.21.15.146:0/3686864208 wait complete. 2024-04-24T08:26:49.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.765+0000 7fc588a00640 1 Processor -- start 2024-04-24T08:26:49.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc588a00640 1 -- start start 2024-04-24T08:26:49.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc588a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 0x7fc5841a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc588a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584109520 0x7fc5841a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc588a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc58419a950 0x7fc58419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc588a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841145b0 con 0x7fc584104450 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc588a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc584114430 con 0x7fc58419a950 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc588a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc584114730 con 0x7fc584109520 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc581e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 0x7fc5841a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc581e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 0x7fc5841a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38414/0 (socket says 172.21.15.146:38414) 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc581e00640 1 -- 172.21.15.146:0/2714344944 learned_addr learned my addr 172.21.15.146:0/2714344944 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:49.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc582800640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc58419a950 0x7fc58419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.766+0000 7fc57be00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584109520 0x7fc5841a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc581e00640 1 -- 172.21.15.146:0/2714344944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584109520 msgr2=0x7fc5841a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc581e00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584109520 0x7fc5841a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc581e00640 1 -- 172.21.15.146:0/2714344944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc58419a950 msgr2=0x7fc58419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc581e00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc58419a950 0x7fc58419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc581e00640 1 -- 172.21.15.146:0/2714344944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58419b470 con 0x7fc584104450 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc582800640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc58419a950 0x7fc58419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc57be00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584109520 0x7fc5841a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.767+0000 7fc581e00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 0x7fc5841a07a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc57400e9f0 tx=0x7fc57400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:49.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.768+0000 7fc57aa00640 1 -- 172.21.15.146:0/2714344944 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57400cd80 con 0x7fc584104450 2024-04-24T08:26:49.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.768+0000 7fc57aa00640 1 -- 172.21.15.146:0/2714344944 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc574010910 con 0x7fc584104450 2024-04-24T08:26:49.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.768+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58419b760 con 0x7fc584104450 2024-04-24T08:26:49.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.768+0000 7fc57aa00640 1 -- 172.21.15.146:0/2714344944 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574018990 con 0x7fc584104450 2024-04-24T08:26:49.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.768+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5841a79b0 con 0x7fc584104450 2024-04-24T08:26:49.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.771+0000 7fc58b7ba740 1 Processor -- start 2024-04-24T08:26:49.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.771+0000 7fc57aa00640 1 -- 172.21.15.146:0/2714344944 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fc574010ac0 con 0x7fc584104450 2024-04-24T08:26:49.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.771+0000 7fc58b7ba740 1 -- start start 2024-04-24T08:26:49.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.771+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee4860 0x555952ee4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.771+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952eef310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952eef850 0x555952ef1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555952e4a1f0 con 0x555952ee4860 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555952df0350 con 0x555952ee5180 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555952e461b0 con 0x555952eef850 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee4860 0x555952ee4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc581e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952eef310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee4860 0x555952ee4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38430/0 (socket says 172.21.15.146:38430) 2024-04-24T08:26:49.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 -- 172.21.15.146:0/2375336208 learned_addr learned my addr 172.21.15.146:0/2375336208 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc57be00640 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952eef850 0x555952ef1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 -- 172.21.15.146:0/2375336208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952eef850 msgr2=0x555952ef1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952eef850 0x555952ef1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 -- 172.21.15.146:0/2375336208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 msgr2=0x555952eef310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952eef310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc582800640 1 -- 172.21.15.146:0/2375336208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555952e44f50 con 0x555952ee4860 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.772+0000 7fc57be00640 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952eef850 0x555952ef1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:49.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.773+0000 7fc582800640 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee4860 0x555952ee4c40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc57c011790 tx=0x7fc57c011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7919071acfe1eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.773+0000 7fc57aa00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc5680a3d20 0x7fc5680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.773+0000 7fc573000640 1 -- 172.21.15.146:0/2375336208 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c003c30 con 0x555952ee4860 2024-04-24T08:26:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.773+0000 7fc573000640 1 -- 172.21.15.146:0/2375336208 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57c00fe80 con 0x555952ee4860 2024-04-24T08:26:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.773+0000 7fc573000640 1 -- 172.21.15.146:0/2375336208 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0193e0 con 0x555952ee4860 2024-04-24T08:26:49.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.773+0000 7fc581e00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc5680a3d20 0x7fc5680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc57aa00640 1 -- 172.21.15.146:0/2714344944 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc5740d0b80 con 0x7fc584104450 2024-04-24T08:26:49.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2375336208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee4860 msgr2=0x555952ee4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee4860 0x555952ee4c40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc57c011790 tx=0x7fc57c011c50 comp rx=0 tx=0).stop 2024-04-24T08:26:49.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc581e00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc5680a3d20 0x7fc5680a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x555952ee5e20 tx=0x7fc5740d4630 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:49.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2375336208 shutdown_connections 2024-04-24T08:26:49.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952eef850 0x555952ef1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952eef310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2375336208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee4860 0x555952ee4c40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2375336208 >> 172.21.15.146:0/2375336208 conn(0x555952ee0b80 msgr2=0x555952ee0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2375336208 shutdown_connections 2024-04-24T08:26:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.774+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2375336208 wait complete. 2024-04-24T08:26:49.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 -- start start 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 Processor -- start 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952eec790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952eeccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952eef850 0x555952eed210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555952ef2640 con 0x555952ee5180 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555952e03690 con 0x555952eef850 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.775+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555952ef27c0 con 0x555952ee4860 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc58b7ba740 10 client.0 _ll_register_callbacks cb 0x555952e3f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:49.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952eeccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952eeccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38434/0 (socket says 172.21.15.146:38434) 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 -- 172.21.15.146:0/2649429023 learned_addr learned my addr 172.21.15.146:0/2649429023 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc582800640 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952eec790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc582800640 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952eef850 0x555952eed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 -- 172.21.15.146:0/2649429023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 msgr2=0x555952eec790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952eec790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 -- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952eef850 msgr2=0x555952eed210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952eef850 0x555952eed210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.776+0000 7fc57be00640 1 -- 172.21.15.146:0/2649429023 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555952e02920 con 0x555952ee5180 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc57be00640 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952eeccd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009900 tx=0x7fc56c0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772e9856cb6e50a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc567400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc567400640 1 -- 172.21.15.146:0/2649429023 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c035040 con 0x555952ee5180 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc567400640 1 -- 172.21.15.146:0/2649429023 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc56c004250 con 0x555952ee5180 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555952e57c10 con 0x555952ee5180 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc567400640 1 -- 172.21.15.146:0/2649429023 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc56c031a10 con 0x555952ee5180 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc58b7ba740 10 client.16323 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:49.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555952c267d0 con 0x555952ee5180 2024-04-24T08:26:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.777+0000 7fc567400640 1 -- 172.21.15.146:0/2649429023 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555952e57c10 con 0x555952ee5180 2024-04-24T08:26:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc567400640 1 -- 172.21.15.146:0/2649429023 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc56c0465f0 con 0x555952ee5180 2024-04-24T08:26:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc58b7ba740 10 client.16323 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc58b7ba740 10 client.16323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc58b7ba740 20 client.16323 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952effbe0 0x555952f01fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc58b7ba740 4 client.16323 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555952e51a80 con 0x555952effbe0 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.778+0000 7fc582800640 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952effbe0 0x555952f01fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.779+0000 7fc582800640 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952effbe0 0x555952f01fd0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aae48300d51a1dc6 server_cookie=7fbb765dd9c41c70 in_seq=0 out_seq=0 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.779+0000 7fc567400640 10 client.16323 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc567400640 1 -- 172.21.15.146:0/2649429023 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555952e51a80 con 0x555952effbe0 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc567400640 10 client.16323 handle_command_reply: tid=0 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 client.16323 shutdown 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 msgr2=0x555952eeccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952eeccd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009900 tx=0x7fc56c0317e0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 shutdown_connections 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952effbe0 0x555952f01fd0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952eef850 0x555952eed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952eeccd0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc567400640 0 client.16323 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2649429023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952eec790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 >> 172.21.15.146:0/2649429023 conn(0x555952ee0b80 msgr2=0x555952ee6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:49.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 shutdown_connections 2024-04-24T08:26:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2649429023 wait complete. 2024-04-24T08:26:49.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.780+0000 7fc58b7ba740 20 client.16323 trim_cache size 0 max 16384 2024-04-24T08:26:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.799+0000 7fc58b7ba740 1 Processor -- start 2024-04-24T08:26:49.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.799+0000 7fc58b7ba740 1 -- start start 2024-04-24T08:26:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.799+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee0b80 0x555952efb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.799+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee4860 0x555952eed030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.799+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952f07890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.799+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841b28a0 con 0x555952ee5180 2024-04-24T08:26:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.799+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841b0170 con 0x555952ee4860 2024-04-24T08:26:49.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841adfc0 con 0x555952ee0b80 2024-04-24T08:26:49.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee4860 0x555952eed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee4860 0x555952eed030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33590/0 (socket says 172.21.15.146:33590) 2024-04-24T08:26:49.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 -- 172.21.15.146:0/2245358352 learned_addr learned my addr 172.21.15.146:0/2245358352 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc582800640 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952f07890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc582800640 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee0b80 0x555952efb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 -- 172.21.15.146:0/2245358352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee0b80 msgr2=0x555952efb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee0b80 0x555952efb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 -- 172.21.15.146:0/2245358352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 msgr2=0x555952f07890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952f07890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc57be00640 1 -- 172.21.15.146:0/2245358352 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555952f02340 con 0x555952ee4860 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.800+0000 7fc582800640 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952f07890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:49.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.801+0000 7fc57be00640 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee4860 0x555952eed030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc56c046f80 tx=0x7fc56c005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d023ff06d866 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.801+0000 7fc572600640 1 -- 172.21.15.146:0/2245358352 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c035070 con 0x555952ee4860 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.801+0000 7fc572600640 1 -- 172.21.15.146:0/2245358352 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc56c031bb0 con 0x555952ee4860 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.801+0000 7fc572600640 1 -- 172.21.15.146:0/2245358352 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c053070 con 0x555952ee4860 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2245358352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee4860 msgr2=0x555952eed030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee4860 0x555952eed030 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc56c046f80 tx=0x7fc56c005780 comp rx=0 tx=0).stop 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2245358352 shutdown_connections 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee5180 0x555952f07890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee4860 0x555952eed030 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/2245358352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee0b80 0x555952efb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2245358352 >> 172.21.15.146:0/2245358352 conn(0x555952eeb3d0 msgr2=0x555952ee6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2245358352 shutdown_connections 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 -- 172.21.15.146:0/2245358352 wait complete. 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 -- start start 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.802+0000 7fc58b7ba740 1 Processor -- start 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee0b80 0x555952ef3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952ef4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc58b7ba740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952f6d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841ae630 con 0x555952ee0b80 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841afaa0 con 0x555952ee5180 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc58b7ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5841aed00 con 0x555952ee4860 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc58b7ba740 10 client.0 _ll_register_callbacks cb 0x7fc56c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc582800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee0b80 0x555952ef3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc582800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee0b80 0x555952ef3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38446/0 (socket says 172.21.15.146:38446) 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc582800640 1 -- 172.21.15.146:0/1443514880 learned_addr learned my addr 172.21.15.146:0/1443514880 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc57be00640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952ef4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.803+0000 7fc582800640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952f6d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc57be00640 1 -- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 msgr2=0x555952f6d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc57be00640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952f6d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc57be00640 1 -- 172.21.15.146:0/1443514880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee0b80 msgr2=0x555952ef3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:49.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc57be00640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee0b80 0x555952ef3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc57be00640 1 -- 172.21.15.146:0/1443514880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555952b5c3f0 con 0x555952ee4860 2024-04-24T08:26:49.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc582800640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee0b80 0x555952ef3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:49.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc57be00640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952ef4170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x555952eef5d0 tx=0x7fc56c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a6f439413bfa824 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:49.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc567400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:49.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc567400640 1 -- 172.21.15.146:0/1443514880 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c053070 con 0x555952ee4860 2024-04-24T08:26:49.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc567400640 1 -- 172.21.15.146:0/1443514880 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc56c046480 con 0x555952ee4860 2024-04-24T08:26:49.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.804+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc56c031bb0 con 0x555952ee4860 2024-04-24T08:26:49.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc567400640 1 -- 172.21.15.146:0/1443514880 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc56c035040 con 0x555952ee4860 2024-04-24T08:26:49.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 10 client.26458 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:49.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555952b1c0f0 con 0x555952ee4860 2024-04-24T08:26:49.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc567400640 1 -- 172.21.15.146:0/1443514880 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc56c031bb0 con 0x555952ee4860 2024-04-24T08:26:49.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc567400640 1 -- 172.21.15.146:0/1443514880 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc56c03d450 con 0x555952ee4860 2024-04-24T08:26:49.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 10 client.26458 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:49.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 10 client.26458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:49.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 20 client.26458 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952eef850 0x555952f714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 4 client.26458 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.805+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555952dcd360 con 0x555952eef850 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc582800640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952eef850 0x555952f714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc582800640 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952eef850 0x555952f714f0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9da1c15ad6b5e28a server_cookie=28ef9113ec7dd912 in_seq=0 out_seq=0 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc567400640 10 client.26458 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc567400640 1 -- 172.21.15.146:0/1443514880 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555952dcd360 con 0x555952eef850 2024-04-24T08:26:49.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc567400640 10 client.26458 handle_command_reply: tid=0 2024-04-24T08:26:49.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc58b7ba740 1 client.26458 shutdown 2024-04-24T08:26:49.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 msgr2=0x555952ef4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.806+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952ef4170 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x555952eef5d0 tx=0x7fc56c005fd0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 shutdown_connections 2024-04-24T08:26:49.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555952eef850 0x555952f714f0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555952ee4860 0x555952ef4170 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555952ee5180 0x555952f6d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc567400640 0 client.26458 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 --2- 172.21.15.146:0/1443514880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555952ee0b80 0x555952ef3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 >> 172.21.15.146:0/1443514880 conn(0x555952eeb3d0 msgr2=0x555952edf390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 shutdown_connections 2024-04-24T08:26:49.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 1 -- 172.21.15.146:0/1443514880 wait complete. 2024-04-24T08:26:49.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc58b7ba740 20 client.26458 trim_cache size 0 max 16384 2024-04-24T08:26:49.818 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:26:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:26:49.818 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:26:49.818 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:26:49.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc5680a3d20 msgr2=0x7fc5680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc588a00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc5680a3d20 0x7fc5680a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x555952ee5e20 tx=0x7fc5740d4630 comp rx=0 tx=0).stop 2024-04-24T08:26:49.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 msgr2=0x7fc5841a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:49.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.807+0000 7fc588a00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 0x7fc5841a07a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc57400e9f0 tx=0x7fc57400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 shutdown_connections 2024-04-24T08:26:49.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc5680a3d20 0x7fc5680a61e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc584109520 0x7fc5841a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc58419a950 0x7fc58419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 --2- 172.21.15.146:0/2714344944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc584104450 0x7fc5841a07a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:49.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 >> 172.21.15.146:0/2714344944 conn(0x7fc5840ffc70 msgr2=0x7fc584100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:49.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 shutdown_connections 2024-04-24T08:26:49.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:49.808+0000 7fc588a00640 1 -- 172.21.15.146:0/2714344944 wait complete. 2024-04-24T08:26:49.827 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:26:49.828 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:26:50.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.021+0000 7f06ae200640 1 -- 172.21.15.146:0/3108987784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8104260 msgr2=0x7f06a8104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.021+0000 7f06ae200640 1 --2- 172.21.15.146:0/3108987784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8104260 0x7f06a8104640 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f069c009900 tx=0x7f069c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.021+0000 7f06ae200640 1 -- 172.21.15.146:0/3108987784 shutdown_connections 2024-04-24T08:26:50.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.021+0000 7f06ae200640 1 --2- 172.21.15.146:0/3108987784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8109330 0x7f06a8110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.021+0000 7f06ae200640 1 --2- 172.21.15.146:0/3108987784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a8104c10 0x7f06a8108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.021+0000 7f06ae200640 1 --2- 172.21.15.146:0/3108987784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8104260 0x7f06a8104640 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.022+0000 7f06ae200640 1 -- 172.21.15.146:0/3108987784 >> 172.21.15.146:0/3108987784 conn(0x7f06a80ffaa0 msgr2=0x7f06a8101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.022+0000 7f06ae200640 1 -- 172.21.15.146:0/3108987784 shutdown_connections 2024-04-24T08:26:50.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.022+0000 7f06ae200640 1 -- 172.21.15.146:0/3108987784 wait complete. 2024-04-24T08:26:50.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 Processor -- start 2024-04-24T08:26:50.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 -- start start 2024-04-24T08:26:50.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8104260 0x7f06a81a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8109330 0x7f06a81a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a819a780 0x7f06a819ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8114440 con 0x7f06a819a780 2024-04-24T08:26:50.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81142c0 con 0x7f06a8109330 2024-04-24T08:26:50.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.023+0000 7f06ae200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81145c0 con 0x7f06a8104260 2024-04-24T08:26:50.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8104260 0x7f06a81a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8104260 0x7f06a81a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47210/0 (socket says 172.21.15.146:47210) 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 -- 172.21.15.146:0/2019719362 learned_addr learned my addr 172.21.15.146:0/2019719362 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ad800640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a819a780 0x7f06a819ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06a7e00640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8109330 0x7f06a81a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 -- 172.21.15.146:0/2019719362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8109330 msgr2=0x7f06a81a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8109330 0x7f06a81a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 -- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a819a780 msgr2=0x7f06a819ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a819a780 0x7f06a819ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 -- 172.21.15.146:0/2019719362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a819b380 con 0x7f06a8104260 2024-04-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06a7e00640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8109330 0x7f06a81a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:50.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.024+0000 7f06ace00640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8104260 0x7f06a81a0670 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f069c004770 tx=0x7f069c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.025+0000 7f06a5600640 1 -- 172.21.15.146:0/2019719362 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c03f070 con 0x7f06a8104260 2024-04-24T08:26:50.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.025+0000 7f06a5600640 1 -- 172.21.15.146:0/2019719362 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f069c031ca0 con 0x7f06a8104260 2024-04-24T08:26:50.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.025+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a819b610 con 0x7f06a8104260 2024-04-24T08:26:50.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.025+0000 7f06a5600640 1 -- 172.21.15.146:0/2019719362 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c035030 con 0x7f06a8104260 2024-04-24T08:26:50.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.025+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a81a7750 con 0x7f06a8104260 2024-04-24T08:26:50.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.026+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a8105730 con 0x7f06a8104260 2024-04-24T08:26:50.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.028+0000 7f06a5600640 1 -- 172.21.15.146:0/2019719362 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f069c042070 con 0x7f06a8104260 2024-04-24T08:26:50.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.029+0000 7f06a5600640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f06900a3da0 0x7f06900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.030+0000 7f06a5600640 1 -- 172.21.15.146:0/2019719362 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f069c0f1350 con 0x7f06a8104260 2024-04-24T08:26:50.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.030+0000 7f06a7e00640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f06900a3da0 0x7f06900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.030+0000 7f06a7e00640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f06900a3da0 0x7f06900a6260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f06940058c0 tx=0x7f0694005850 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.032+0000 7f06a5600640 1 -- 172.21.15.146:0/2019719362 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f069c0bdab0 con 0x7f06a8104260 2024-04-24T08:26:50.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.276+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06a810c560 con 0x7f06a8104260 2024-04-24T08:26:50.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.277+0000 7f06a5600640 1 -- 172.21.15.146:0/2019719362 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f069c0bd3b0 con 0x7f06a8104260 2024-04-24T08:26:50.279 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:26:50.280 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:26:50.280 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:26:50.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.280+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f06900a3da0 msgr2=0x7f06900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.280+0000 7f06ae200640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f06900a3da0 0x7f06900a6260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f06940058c0 tx=0x7f0694005850 comp rx=0 tx=0).stop 2024-04-24T08:26:50.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.280+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8104260 msgr2=0x7f06a81a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.280+0000 7f06ae200640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8104260 0x7f06a81a0670 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f069c004770 tx=0x7f069c003c10 comp rx=0 tx=0).stop 2024-04-24T08:26:50.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 shutdown_connections 2024-04-24T08:26:50.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f06900a3da0 0x7f06900a6260 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a819a780 0x7f06a819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06a8109330 0x7f06a81a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 --2- 172.21.15.146:0/2019719362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a8104260 0x7f06a81a0670 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 >> 172.21.15.146:0/2019719362 conn(0x7f06a80ffaa0 msgr2=0x7f06a8101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 shutdown_connections 2024-04-24T08:26:50.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.281+0000 7f06ae200640 1 -- 172.21.15.146:0/2019719362 wait complete. 2024-04-24T08:26:50.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:26:50.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 -- 172.21.15.146:0/262003241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90109520 msgr2=0x7fdf901110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 --2- 172.21.15.146:0/262003241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90109520 0x7fdf901110f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009ca0 tx=0x7fdf8802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 -- 172.21.15.146:0/262003241 shutdown_connections 2024-04-24T08:26:50.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 --2- 172.21.15.146:0/262003241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90109520 0x7fdf901110f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 --2- 172.21.15.146:0/262003241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf90104e00 0x7fdf90108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 --2- 172.21.15.146:0/262003241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90104450 0x7fdf90104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 -- 172.21.15.146:0/262003241 >> 172.21.15.146:0/262003241 conn(0x7fdf900ffc70 msgr2=0x7fdf90102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 -- 172.21.15.146:0/262003241 shutdown_connections 2024-04-24T08:26:50.506 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.503+0000 7fdf97600640 1 -- 172.21.15.146:0/262003241 wait complete. 2024-04-24T08:26:50.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 Processor -- start 2024-04-24T08:26:50.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 -- start start 2024-04-24T08:26:50.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90104450 0x7fdf901a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90109520 0x7fdf901a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf9019a8d0 0x7fdf9019ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90114610 con 0x7fdf90109520 2024-04-24T08:26:50.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90114490 con 0x7fdf9019a8d0 2024-04-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.504+0000 7fdf97600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90114790 con 0x7fdf90104450 2024-04-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90109520 0x7fdf901a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90109520 0x7fdf901a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38468/0 (socket says 172.21.15.146:38468) 2024-04-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 -- 172.21.15.146:0/1237686361 learned_addr learned my addr 172.21.15.146:0/1237686361 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf94c00640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90104450 0x7fdf901a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf95600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf9019a8d0 0x7fdf9019ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 -- 172.21.15.146:0/1237686361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90104450 msgr2=0x7fdf901a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90104450 0x7fdf901a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 -- 172.21.15.146:0/1237686361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf9019a8d0 msgr2=0x7fdf9019ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf9019a8d0 0x7fdf9019ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.505+0000 7fdf8fe00640 1 -- 172.21.15.146:0/1237686361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9019b3f0 con 0x7fdf90109520 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.506+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90109520 0x7fdf901a0c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000e9c0 tx=0x7fdf8000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.506+0000 7fdf8d600640 1 -- 172.21.15.146:0/1237686361 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000cd80 con 0x7fdf90109520 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.506+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9019b6e0 con 0x7fdf90109520 2024-04-24T08:26:50.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.506+0000 7fdf8d600640 1 -- 172.21.15.146:0/1237686361 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf80005500 con 0x7fdf90109520 2024-04-24T08:26:50.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.506+0000 7fdf8d600640 1 -- 172.21.15.146:0/1237686361 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80010690 con 0x7fdf90109520 2024-04-24T08:26:50.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.506+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901a7930 con 0x7fdf90109520 2024-04-24T08:26:50.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.510+0000 7fdf9a37e740 1 Processor -- start 2024-04-24T08:26:50.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.510+0000 7fdf9a37e740 1 -- start start 2024-04-24T08:26:50.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f13bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f146400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f148d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564e4f0a11d0 con 0x564e4f146940 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e4f047450 con 0x564e4f13c270 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e4f09d190 con 0x564e4f13b950 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf8d600640 1 -- 172.21.15.146:0/1237686361 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fdf800046c0 con 0x7fdf90109520 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf95600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f13bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf95600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f13bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47230/0 (socket says 172.21.15.146:47230) 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf95600640 1 -- 172.21.15.146:0/1433214626 learned_addr learned my addr 172.21.15.146:0/1433214626 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf94c00640 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf95600640 1 -- 172.21.15.146:0/1433214626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 msgr2=0x564e4f146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf95600640 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf95600640 1 -- 172.21.15.146:0/1433214626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 msgr2=0x564e4f148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.511+0000 7fdf95600640 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.512+0000 7fdf95600640 1 -- 172.21.15.146:0/1433214626 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e4f104310 con 0x564e4f13b950 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.512+0000 7fdf95600640 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f13bd30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009ca0 tx=0x7fdf880093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3def19e7419af31 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.512+0000 7fdf7dc00640 1 -- 172.21.15.146:0/1433214626 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88031db0 con 0x564e4f13b950 2024-04-24T08:26:50.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.512+0000 7fdf7dc00640 1 -- 172.21.15.146:0/1433214626 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf88036030 con 0x564e4f13b950 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.512+0000 7fdf7dc00640 1 -- 172.21.15.146:0/1433214626 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88004330 con 0x564e4f13b950 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.512+0000 7fdf8d600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdf780a3d20 0x7fdf780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf8d600640 1 -- 172.21.15.146:0/1237686361 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdf800d0b40 con 0x7fdf90109520 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 -- 172.21.15.146:0/1433214626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 msgr2=0x564e4f13bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f13bd30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009ca0 tx=0x7fdf880093e0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf94c00640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdf780a3d20 0x7fdf780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 -- 172.21.15.146:0/1433214626 shutdown_connections 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/1433214626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f13bd30 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 -- 172.21.15.146:0/1433214626 >> 172.21.15.146:0/1433214626 conn(0x564e4f137c70 msgr2=0x564e4f138050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 -- 172.21.15.146:0/1433214626 shutdown_connections 2024-04-24T08:26:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.513+0000 7fdf9a37e740 1 -- 172.21.15.146:0/1433214626 wait complete. 2024-04-24T08:26:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf94c00640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdf780a3d20 0x7fdf780a61e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x564e4f13cf10 tx=0x7fdf84005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 -- start start 2024-04-24T08:26:50.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 Processor -- start 2024-04-24T08:26:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f13b950 0x564e4f1463c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f1436b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f146940 0x564e4f143df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564e4f1495f0 con 0x564e4f13b950 2024-04-24T08:26:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e4f05a810 con 0x564e4f146940 2024-04-24T08:26:50.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e4f149770 con 0x564e4f13c270 2024-04-24T08:26:50.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.514+0000 7fdf9a37e740 10 client.0 _ll_register_callbacks cb 0x564e4f028140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:50.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf94c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f13b950 0x564e4f1463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f1436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf94c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f13b950 0x564e4f1463c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38494/0 (socket says 172.21.15.146:38494) 2024-04-24T08:26:50.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf94c00640 1 -- 172.21.15.146:0/4217930175 learned_addr learned my addr 172.21.15.146:0/4217930175 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf95600640 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f146940 0x564e4f143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf8fe00640 1 -- 172.21.15.146:0/4217930175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f146940 msgr2=0x564e4f143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f146940 0x564e4f143df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf8fe00640 1 -- 172.21.15.146:0/4217930175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f13b950 msgr2=0x564e4f1463c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f13b950 0x564e4f1463c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf8fe00640 1 -- 172.21.15.146:0/4217930175 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e4f06b3f0 con 0x564e4f13c270 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f1436b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdf800ce240 tx=0x7fdf8009c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65c228b3c4f54f02 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.515+0000 7fdf72000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:50.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf72000640 1 -- 172.21.15.146:0/4217930175 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800ce6c0 con 0x564e4f13c270 2024-04-24T08:26:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf72000640 1 -- 172.21.15.146:0/4217930175 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdf800a5270 con 0x564e4f13c270 2024-04-24T08:26:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf88036030 con 0x564e4f13c270 2024-04-24T08:26:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf72000640 1 -- 172.21.15.146:0/4217930175 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdf8009ca90 con 0x564e4f13c270 2024-04-24T08:26:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf9a37e740 10 client.26476 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564e4edb33f0 con 0x564e4f13c270 2024-04-24T08:26:50.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf72000640 1 -- 172.21.15.146:0/4217930175 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf88036030 con 0x564e4f13c270 2024-04-24T08:26:50.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.516+0000 7fdf72000640 1 -- 172.21.15.146:0/4217930175 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fdf8009c000 con 0x564e4f13c270 2024-04-24T08:26:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf9a37e740 10 client.26476 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf9a37e740 10 client.26476 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:26:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf9a37e740 20 client.26476 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f156a00 0x564e4f158df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf9a37e740 4 client.26476 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e4f0aebf0 con 0x564e4f156a00 2024-04-24T08:26:50.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf94c00640 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f156a00 0x564e4f158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf72000640 10 client.26476 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.517+0000 7fdf94c00640 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f156a00 0x564e4f158df0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ddb6eeb2b30a0ca8 server_cookie=e402e54a642ee168 in_seq=0 out_seq=0 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf72000640 1 -- 172.21.15.146:0/4217930175 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e4f0aebf0 con 0x564e4f156a00 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf72000640 10 client.26476 handle_command_reply: tid=0 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 client.26476 shutdown 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 msgr2=0x564e4f1436b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f1436b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdf800ce240 tx=0x7fdf8009c920 comp rx=0 tx=0).stop 2024-04-24T08:26:50.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 shutdown_connections 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f156a00 0x564e4f158df0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f1436b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f146940 0x564e4f143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf72000640 0 client.26476 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.518+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/4217930175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f13b950 0x564e4f1463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.519+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 >> 172.21.15.146:0/4217930175 conn(0x564e4f137c70 msgr2=0x564e4f13da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.519+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 shutdown_connections 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.519+0000 7fdf9a37e740 1 -- 172.21.15.146:0/4217930175 wait complete. 2024-04-24T08:26:50.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.519+0000 7fdf9a37e740 20 client.26476 trim_cache size 0 max 16384 2024-04-24T08:26:50.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 Processor -- start 2024-04-24T08:26:50.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 -- start start 2024-04-24T08:26:50.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f13e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b2830 con 0x564e4f146940 2024-04-24T08:26:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901b0100 con 0x564e4f13c270 2024-04-24T08:26:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901adf50 con 0x564e4f13b950 2024-04-24T08:26:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf95600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f13e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f1524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47250/0 (socket says 172.21.15.146:47250) 2024-04-24T08:26:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.534+0000 7fdf94c00640 1 -- 172.21.15.146:0/3886438649 learned_addr learned my addr 172.21.15.146:0/3886438649 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf94c00640 1 -- 172.21.15.146:0/3886438649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 msgr2=0x564e4f13e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf94c00640 1 --2- 172.21.15.146:0/3886438649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f13e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf94c00640 1 -- 172.21.15.146:0/3886438649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 msgr2=0x564e4f145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf94c00640 1 --2- 172.21.15.146:0/3886438649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf94c00640 1 -- 172.21.15.146:0/3886438649 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e4ee7d7d0 con 0x564e4f13b950 2024-04-24T08:26:50.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf94c00640 1 --2- 172.21.15.146:0/3886438649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f1524e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x564e4f159630 tx=0x7fdf8401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a70f137c8157100a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf7d200640 1 -- 172.21.15.146:0/3886438649 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84003f10 con 0x564e4f13b950 2024-04-24T08:26:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf7d200640 1 -- 172.21.15.146:0/3886438649 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf84005020 con 0x564e4f13b950 2024-04-24T08:26:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.535+0000 7fdf7d200640 1 -- 172.21.15.146:0/3886438649 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf840047a0 con 0x564e4f13b950 2024-04-24T08:26:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3886438649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 msgr2=0x564e4f1524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3886438649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f1524e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x564e4f159630 tx=0x7fdf8401b000 comp rx=0 tx=0).stop 2024-04-24T08:26:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3886438649 shutdown_connections 2024-04-24T08:26:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3886438649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3886438649 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13c270 0x564e4f13e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3886438649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13b950 0x564e4f1524e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3886438649 >> 172.21.15.146:0/3886438649 conn(0x564e4f137c70 msgr2=0x564e4f13d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3886438649 shutdown_connections 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3886438649 wait complete. 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 -- start start 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.536+0000 7fdf9a37e740 1 Processor -- start 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13b950 0x564e4f14eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f140b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf9a37e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f141180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901ae5c0 con 0x564e4f146940 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901afa30 con 0x564e4f13b950 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf9a37e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901aec90 con 0x564e4f13c270 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf9a37e740 10 client.0 _ll_register_callbacks cb 0x7fdf80009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf95600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf8fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf95600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38506/0 (socket says 172.21.15.146:38506) 2024-04-24T08:26:50.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf95600640 1 -- 172.21.15.146:0/3176535255 learned_addr learned my addr 172.21.15.146:0/3176535255 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.537+0000 7fdf94c00640 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13b950 0x564e4f14eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf8fe00640 1 -- 172.21.15.146:0/3176535255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13b950 msgr2=0x564e4f14eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13b950 0x564e4f14eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf8fe00640 1 -- 172.21.15.146:0/3176535255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 msgr2=0x564e4f141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf8fe00640 1 -- 172.21.15.146:0/3176535255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e4f13fb70 con 0x564e4f13c270 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf8fe00640 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f140b50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf80099410 tx=0x7fdf8009c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87b7d70b4e3120ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf72000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf72000640 1 -- 172.21.15.146:0/3176535255 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80098d30 con 0x564e4f13c270 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf72000640 1 -- 172.21.15.146:0/3176535255 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fdf8009bee0 con 0x564e4f13c270 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf84005020 con 0x564e4f13c270 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.538+0000 7fdf72000640 1 -- 172.21.15.146:0/3176535255 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdf800a5470 con 0x564e4f13c270 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 10 client.26482 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564e4f1bd270 con 0x564e4f13c270 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf72000640 1 -- 172.21.15.146:0/3176535255 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf84005020 con 0x564e4f13c270 2024-04-24T08:26:50.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf72000640 1 -- 172.21.15.146:0/3176535255 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fdf800a20f0 con 0x564e4f13c270 2024-04-24T08:26:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 10 client.26482 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 10 client.26482 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:26:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 20 client.26482 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f1bd9c0 0x564e4f1bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 4 client.26482 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e4f0a8a60 con 0x564e4f1bd9c0 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.539+0000 7fdf94c00640 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f1bd9c0 0x564e4f1bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf94c00640 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f1bd9c0 0x564e4f1bfde0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41f9e7e5f3e8c31c server_cookie=60a3a667ad76bbca in_seq=0 out_seq=0 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf72000640 10 client.26482 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf72000640 1 -- 172.21.15.146:0/3176535255 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e4f0a8a60 con 0x564e4f1bd9c0 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf72000640 10 client.26482 handle_command_reply: tid=0 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 client.26482 shutdown 2024-04-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 msgr2=0x564e4f140b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f140b50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf80099410 tx=0x7fdf8009c990 comp rx=0 tx=0).stop 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 shutdown_connections 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x564e4f1bd9c0 0x564e4f1bfde0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4f13c270 0x564e4f140b50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e4f146940 0x564e4f141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf72000640 0 client.26482 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 --2- 172.21.15.146:0/3176535255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e4f13b950 0x564e4f14eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.540+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 >> 172.21.15.146:0/3176535255 conn(0x564e4f137c70 msgr2=0x564e4f13d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 shutdown_connections 2024-04-24T08:26:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf9a37e740 1 -- 172.21.15.146:0/3176535255 wait complete. 2024-04-24T08:26:50.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf9a37e740 20 client.26482 trim_cache size 0 max 16384 2024-04-24T08:26:50.552 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:26:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdf780a3d20 msgr2=0x7fdf780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdf780a3d20 0x7fdf780a61e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x564e4f13cf10 tx=0x7fdf84005f90 comp rx=0 tx=0).stop 2024-04-24T08:26:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90109520 msgr2=0x7fdf901a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90109520 0x7fdf901a0c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000e9c0 tx=0x7fdf8000ee80 comp rx=0 tx=0).stop 2024-04-24T08:26:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 shutdown_connections 2024-04-24T08:26:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdf780a3d20 0x7fdf780a61e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf9019a8d0 0x7fdf9019ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf90109520 0x7fdf901a0c60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 --2- 172.21.15.146:0/1237686361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf90104450 0x7fdf901a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.541+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 >> 172.21.15.146:0/1237686361 conn(0x7fdf900ffc70 msgr2=0x7fdf90101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.542+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 shutdown_connections 2024-04-24T08:26:50.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.542+0000 7fdf97600640 1 -- 172.21.15.146:0/1237686361 wait complete. 2024-04-24T08:26:50.558 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:50 smithi146 ceph-mon[27805]: pgmap v172: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:50.558 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:50 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2019719362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:26:50.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:26:50.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:50 smithi153 ceph-mon[32746]: pgmap v172: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:50.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:50 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2019719362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:26:50.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.760+0000 7f1d94a00640 1 -- 172.21.15.146:0/478912324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d901071c0 msgr2=0x7f1d9010ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.760+0000 7f1d94a00640 1 --2- 172.21.15.146:0/478912324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d901071c0 0x7f1d9010ed90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009ca0 tx=0x7f1d8802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.761+0000 7f1d94a00640 1 -- 172.21.15.146:0/478912324 shutdown_connections 2024-04-24T08:26:50.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.761+0000 7f1d94a00640 1 --2- 172.21.15.146:0/478912324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d901071c0 0x7f1d9010ed90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.761+0000 7f1d94a00640 1 --2- 172.21.15.146:0/478912324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d90102aa0 0x7f1d90106ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.761+0000 7f1d94a00640 1 --2- 172.21.15.146:0/478912324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d901020f0 0x7f1d901024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.761+0000 7f1d94a00640 1 -- 172.21.15.146:0/478912324 >> 172.21.15.146:0/478912324 conn(0x7f1d900fd9d0 msgr2=0x7f1d900ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.761+0000 7f1d94a00640 1 -- 172.21.15.146:0/478912324 shutdown_connections 2024-04-24T08:26:50.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.761+0000 7f1d94a00640 1 -- 172.21.15.146:0/478912324 wait complete. 2024-04-24T08:26:50.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 Processor -- start 2024-04-24T08:26:50.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 -- start start 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d901020f0 0x7f1d901a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d901071c0 0x7f1d901a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9019a7b0 0x7f1d9019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1d901122b0 con 0x7f1d901020f0 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d90112130 con 0x7f1d901071c0 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d94a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d90112430 con 0x7f1d9019a7b0 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.762+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d901020f0 0x7f1d901a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d901020f0 0x7f1d901a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38528/0 (socket says 172.21.15.146:38528) 2024-04-24T08:26:50.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 -- 172.21.15.146:0/169101990 learned_addr learned my addr 172.21.15.146:0/169101990 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d87e00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d901071c0 0x7f1d901a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8f200640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9019a7b0 0x7f1d9019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 -- 172.21.15.146:0/169101990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9019a7b0 msgr2=0x7f1d9019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9019a7b0 0x7f1d9019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 -- 172.21.15.146:0/169101990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d901071c0 msgr2=0x7f1d901a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d901071c0 0x7f1d901a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 -- 172.21.15.146:0/169101990 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9019b340 con 0x7f1d901020f0 2024-04-24T08:26:50.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d87e00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d901071c0 0x7f1d901a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:50.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.763+0000 7f1d8e800640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d901020f0 0x7f1d901a0640 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000e9f0 tx=0x7f1d8000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.764+0000 7f1d87400640 1 -- 172.21.15.146:0/169101990 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8000cd80 con 0x7f1d901020f0 2024-04-24T08:26:50.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.764+0000 7f1d87400640 1 -- 172.21.15.146:0/169101990 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d80010910 con 0x7f1d901020f0 2024-04-24T08:26:50.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.764+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9019b630 con 0x7f1d901020f0 2024-04-24T08:26:50.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.764+0000 7f1d87400640 1 -- 172.21.15.146:0/169101990 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80018990 con 0x7f1d901020f0 2024-04-24T08:26:50.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.764+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d901a7810 con 0x7f1d901020f0 2024-04-24T08:26:50.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d98261740 1 Processor -- start 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d87400640 1 -- 172.21.15.146:0/169101990 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f1d80010ac0 con 0x7f1d901020f0 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d98261740 1 -- start start 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d98261740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9457f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d98261740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee94625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d98261740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee9464f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d98261740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dee93bd1d0 con 0x55dee9457b40 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.767+0000 7f1d98261740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dee9363450 con 0x55dee9462b30 2024-04-24T08:26:50.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d98261740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dee93b9190 con 0x55dee9458460 2024-04-24T08:26:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d8f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9457f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee9464f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee9464f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33684/0 (socket says 172.21.15.146:33684) 2024-04-24T08:26:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 -- 172.21.15.146:0/3917906411 learned_addr learned my addr 172.21.15.146:0/3917906411 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d8f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9457f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38540/0 (socket says 172.21.15.146:38540) 2024-04-24T08:26:50.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d8e800640 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee94625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 -- 172.21.15.146:0/3917906411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 msgr2=0x55dee94625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee94625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 -- 172.21.15.146:0/3917906411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 msgr2=0x55dee9457f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9457f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d87e00640 1 -- 172.21.15.146:0/3917906411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dee9420310 con 0x55dee9462b30 2024-04-24T08:26:50.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.768+0000 7f1d8e800640 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee94625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.769+0000 7f1d87e00640 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee9464f20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009bf0 tx=0x7f1d780117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab2f899ecf2ca019 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.769+0000 7f1d87400640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1d740a3d20 0x7f1d740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.769+0000 7f1d7f400640 1 -- 172.21.15.146:0/3917906411 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d78012450 con 0x55dee9462b30 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.769+0000 7f1d7f400640 1 -- 172.21.15.146:0/3917906411 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d78012ad0 con 0x55dee9462b30 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.769+0000 7f1d7f400640 1 -- 172.21.15.146:0/3917906411 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7800fbc0 con 0x55dee9462b30 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.769+0000 7f1d87400640 1 -- 172.21.15.146:0/169101990 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1d800d0bd0 con 0x7f1d901020f0 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.769+0000 7f1d8f200640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1d740a3d20 0x7f1d740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 -- 172.21.15.146:0/3917906411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 msgr2=0x55dee9464f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee9464f20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009bf0 tx=0x7f1d780117b0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d8f200640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1d740a3d20 0x7f1d740a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1d88006010 tx=0x7f1d88005d80 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 -- 172.21.15.146:0/3917906411 shutdown_connections 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee94625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee9464f20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 --2- 172.21.15.146:0/3917906411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9457f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 -- 172.21.15.146:0/3917906411 >> 172.21.15.146:0/3917906411 conn(0x55dee9453e60 msgr2=0x55dee9454240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 -- 172.21.15.146:0/3917906411 shutdown_connections 2024-04-24T08:26:50.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 -- 172.21.15.146:0/3917906411 wait complete. 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.770+0000 7f1d98261740 1 -- start start 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 1 Processor -- start 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9462500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee945f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee945fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55dee94659c0 con 0x55dee9457b40 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55dee9376810 con 0x55dee9462b30 2024-04-24T08:26:50.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dee9465b40 con 0x55dee9458460 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d98261740 10 client.0 _ll_register_callbacks cb 0x55dee9344140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9462500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9462500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38554/0 (socket says 172.21.15.146:38554) 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d8e800640 1 -- 172.21.15.146:0/1555645784 learned_addr learned my addr 172.21.15.146:0/1555645784 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d8f200640 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee945fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.771+0000 7f1d87e00640 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee945f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d8e800640 1 -- 172.21.15.146:0/1555645784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 msgr2=0x55dee945f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d8e800640 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee945f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d8e800640 1 -- 172.21.15.146:0/1555645784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 msgr2=0x55dee945fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d8e800640 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee945fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d8e800640 1 -- 172.21.15.146:0/1555645784 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dee93873f0 con 0x55dee9457b40 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d8e800640 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9462500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1d8009c880 tx=0x7f1d80098f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd882cc7984a6b7b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d73e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d73e00640 1 -- 172.21.15.146:0/1555645784 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8009d280 con 0x55dee9457b40 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d73e00640 1 -- 172.21.15.146:0/1555645784 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1d80018b30 con 0x55dee9457b40 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.772+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d78012ad0 con 0x55dee9457b40 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d73e00640 1 -- 172.21.15.146:0/1555645784 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1d800b6050 con 0x55dee9457b40 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 10 client.16365 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:50.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dee90cf3f0 con 0x55dee9457b40 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d73e00640 1 -- 172.21.15.146:0/1555645784 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d78012ad0 con 0x55dee9457b40 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d73e00640 1 -- 172.21.15.146:0/1555645784 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1d8009bfd0 con 0x55dee9457b40 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 10 client.16365 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 10 client.16365 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 20 client.16365 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee9472ce0 0x55dee94750d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 4 client.16365 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:50.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dee93cabf0 con 0x55dee9472ce0 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.773+0000 7f1d87e00640 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee9472ce0 0x55dee94750d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.774+0000 7f1d87e00640 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee9472ce0 0x55dee94750d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=63ab72a70b3c47c1 server_cookie=4f7f76bdb49ca36b in_seq=0 out_seq=0 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.774+0000 7f1d73e00640 10 client.16365 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d73e00640 1 -- 172.21.15.146:0/1555645784 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dee93cabf0 con 0x55dee9472ce0 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d73e00640 10 client.16365 handle_command_reply: tid=0 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 client.16365 shutdown 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 msgr2=0x55dee9462500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9462500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1d8009c880 tx=0x7f1d80098f20 comp rx=0 tx=0).stop 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 shutdown_connections 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee9472ce0 0x55dee94750d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee945f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee945fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 --2- 172.21.15.146:0/1555645784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee9462500 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d73e00640 0 client.16365 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 >> 172.21.15.146:0/1555645784 conn(0x55dee9453e60 msgr2=0x55dee9459c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 shutdown_connections 2024-04-24T08:26:50.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 1 -- 172.21.15.146:0/1555645784 wait complete. 2024-04-24T08:26:50.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.775+0000 7f1d98261740 20 client.16365 trim_cache size 0 max 16384 2024-04-24T08:26:50.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 Processor -- start 2024-04-24T08:26:50.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 -- start start 2024-04-24T08:26:50.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee9462030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee947a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1d901b2600 con 0x55dee9457b40 2024-04-24T08:26:50.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d901afed0 con 0x55dee9462b30 2024-04-24T08:26:50.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.790+0000 7f1d98261740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d901add20 con 0x55dee9458460 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38562/0 (socket says 172.21.15.146:38562) 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 -- 172.21.15.146:0/1434591335 learned_addr learned my addr 172.21.15.146:0/1434591335 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d87e00640 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee9462030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8f200640 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee947a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 -- 172.21.15.146:0/1434591335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 msgr2=0x55dee9462030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee9462030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 -- 172.21.15.146:0/1434591335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 msgr2=0x55dee947a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee947a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 -- 172.21.15.146:0/1434591335 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dee9475440 con 0x55dee9457b40 2024-04-24T08:26:50.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d87e00640 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee9462030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:50.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d8e800640 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946e880 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c6a0 tx=0x7f1d80018f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2054f03f4f7e53ac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.791+0000 7f1d7ea00640 1 -- 172.21.15.146:0/1434591335 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8009d280 con 0x55dee9457b40 2024-04-24T08:26:50.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d7ea00640 1 -- 172.21.15.146:0/1434591335 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d800d4230 con 0x55dee9457b40 2024-04-24T08:26:50.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d7ea00640 1 -- 172.21.15.146:0/1434591335 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80018b30 con 0x55dee9457b40 2024-04-24T08:26:50.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 -- 172.21.15.146:0/1434591335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 msgr2=0x55dee946e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946e880 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c6a0 tx=0x7f1d80018f10 comp rx=0 tx=0).stop 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 -- 172.21.15.146:0/1434591335 shutdown_connections 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9458460 0x55dee9462030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9462b30 0x55dee947a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 --2- 172.21.15.146:0/1434591335 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946e880 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 -- 172.21.15.146:0/1434591335 >> 172.21.15.146:0/1434591335 conn(0x55dee9453e60 msgr2=0x55dee945aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 -- 172.21.15.146:0/1434591335 shutdown_connections 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.792+0000 7f1d98261740 1 -- 172.21.15.146:0/1434591335 wait complete. 2024-04-24T08:26:50.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 -- start start 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 Processor -- start 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9458460 0x55dee945ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9462b30 0x55dee945cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1d901ae390 con 0x55dee9457b40 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1d901af800 con 0x55dee9458460 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d901aea60 con 0x55dee9462b30 2024-04-24T08:26:50.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.793+0000 7f1d98261740 10 client.0 _ll_register_callbacks cb 0x7f1d8000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38566/0 (socket says 172.21.15.146:38566) 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 -- 172.21.15.146:0/2887988899 learned_addr learned my addr 172.21.15.146:0/2887988899 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8f200640 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9462b30 0x55dee945cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d87e00640 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9458460 0x55dee945ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 -- 172.21.15.146:0/2887988899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9462b30 msgr2=0x55dee945cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9462b30 0x55dee945cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 -- 172.21.15.146:0/2887988899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9458460 msgr2=0x55dee945ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9458460 0x55dee945ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 -- 172.21.15.146:0/2887988899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dee91997d0 con 0x55dee9457b40 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d8e800640 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946b180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1d800a52c0 tx=0x7f1d800ce410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8681c4b14d2a30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.794+0000 7f1d73e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d73e00640 1 -- 172.21.15.146:0/2887988899 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80098bf0 con 0x55dee9457b40 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d73e00640 1 -- 172.21.15.146:0/2887988899 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f1d80010430 con 0x55dee9457b40 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d800d4230 con 0x55dee9457b40 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d73e00640 1 -- 172.21.15.146:0/2887988899 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1d8009cb80 con 0x55dee9457b40 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 10 client.16371 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dee94d9210 con 0x55dee9457b40 2024-04-24T08:26:50.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d73e00640 1 -- 172.21.15.146:0/2887988899 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d800d4230 con 0x55dee9457b40 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d73e00640 1 -- 172.21.15.146:0/2887988899 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1d800b1180 con 0x55dee9457b40 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 10 client.16371 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 10 client.16371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 20 client.16371 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee94d9680 0x55dee94dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 4 client.16371 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.795+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dee93c4a60 con 0x55dee94d9680 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.796+0000 7f1d87e00640 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee94d9680 0x55dee94dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:50.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.796+0000 7f1d73e00640 10 client.16371 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:50.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.796+0000 7f1d87e00640 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee94d9680 0x55dee94dba70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db852e289b21c484 server_cookie=3f9f4997e6b4d27f in_seq=0 out_seq=0 2024-04-24T08:26:50.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.796+0000 7f1d73e00640 1 -- 172.21.15.146:0/2887988899 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dee93c4a60 con 0x55dee94d9680 2024-04-24T08:26:50.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.796+0000 7f1d73e00640 10 client.16371 handle_command_reply: tid=0 2024-04-24T08:26:50.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.796+0000 7f1d98261740 1 client.16371 shutdown 2024-04-24T08:26:50.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 msgr2=0x55dee946b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946b180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1d800a52c0 tx=0x7f1d800ce410 comp rx=0 tx=0).stop 2024-04-24T08:26:50.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 shutdown_connections 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55dee94d9680 0x55dee94dba70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dee9462b30 0x55dee945cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55dee9458460 0x55dee945ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d73e00640 0 client.16371 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 --2- 172.21.15.146:0/2887988899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55dee9457b40 0x55dee946b180 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 >> 172.21.15.146:0/2887988899 conn(0x55dee9453e60 msgr2=0x55dee94526c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 shutdown_connections 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 1 -- 172.21.15.146:0/2887988899 wait complete. 2024-04-24T08:26:50.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.797+0000 7f1d98261740 20 client.16371 trim_cache size 0 max 16384 2024-04-24T08:26:50.808 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:26:50.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1d740a3d20 msgr2=0x7f1d740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1d740a3d20 0x7f1d740a61e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1d88006010 tx=0x7f1d88005d80 comp rx=0 tx=0).stop 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d901020f0 msgr2=0x7f1d901a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d901020f0 0x7f1d901a0640 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000e9f0 tx=0x7f1d8000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 shutdown_connections 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1d740a3d20 0x7f1d740a61e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d9019a7b0 0x7f1d9019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d901071c0 0x7f1d901a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 --2- 172.21.15.146:0/169101990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d901020f0 0x7f1d901a0640 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:50.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 >> 172.21.15.146:0/169101990 conn(0x7f1d900fd9d0 msgr2=0x7f1d900fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:50.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 shutdown_connections 2024-04-24T08:26:50.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:50.798+0000 7f1d94a00640 1 -- 172.21.15.146:0/169101990 wait complete. 2024-04-24T08:26:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:50 smithi177 ceph-mon[33046]: pgmap v172: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:50 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2019719362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:26:50.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:26:51.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 -- 172.21.15.146:0/3611166402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc0104320 msgr2=0x7f3cc0104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/3611166402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc0104320 0x7f3cc0104700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009900 tx=0x7f3cb002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 -- 172.21.15.146:0/3611166402 shutdown_connections 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/3611166402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cc0104cd0 0x7f3cc0108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/3611166402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc01093f0 0x7f3cc0110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/3611166402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc0104320 0x7f3cc0104700 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 -- 172.21.15.146:0/3611166402 >> 172.21.15.146:0/3611166402 conn(0x7f3cc00ffb40 msgr2=0x7f3cc0101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 -- 172.21.15.146:0/3611166402 shutdown_connections 2024-04-24T08:26:51.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.012+0000 7f3cbfe00640 1 -- 172.21.15.146:0/3611166402 wait complete. 2024-04-24T08:26:51.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.013+0000 7f3cbfe00640 1 Processor -- start 2024-04-24T08:26:51.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.013+0000 7f3cbfe00640 1 -- start start 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.013+0000 7f3cbfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc0104320 0x7f3cc01a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cc01093f0 0x7f3cc01a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc019a820 0x7f3cc019acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01144a0 con 0x7f3cc0104320 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbfe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc0114320 con 0x7f3cc019a820 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbfe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc0114620 con 0x7f3cc01093f0 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc0104320 0x7f3cc01a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc019a820 0x7f3cc019acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc019a820 0x7f3cc019acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41316/0 (socket says 172.21.15.146:41316) 2024-04-24T08:26:51.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 -- 172.21.15.146:0/369105704 learned_addr learned my addr 172.21.15.146:0/369105704 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:51.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cc01093f0 0x7f3cc01a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 -- 172.21.15.146:0/369105704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cc01093f0 msgr2=0x7f3cc01a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cc01093f0 0x7f3cc01a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 -- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc0104320 msgr2=0x7f3cc01a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc0104320 0x7f3cc01a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.014+0000 7f3cbf400640 1 -- 172.21.15.146:0/369105704 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc019b3b0 con 0x7f3cc019a820 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.015+0000 7f3cbf400640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc019a820 0x7f3cc019acd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3cb800efd0 tx=0x7f3cb800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.015+0000 7f3cb7400640 1 -- 172.21.15.146:0/369105704 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8003b70 con 0x7f3cc019a820 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.015+0000 7f3cb7400640 1 -- 172.21.15.146:0/369105704 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cb8003d10 con 0x7f3cc019a820 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.015+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc019b6a0 con 0x7f3cc019a820 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.015+0000 7f3cb7400640 1 -- 172.21.15.146:0/369105704 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb80109f0 con 0x7f3cc019a820 2024-04-24T08:26:51.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.015+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc01a7830 con 0x7f3cc019a820 2024-04-24T08:26:51.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.018+0000 7f3cc881c740 1 Processor -- start 2024-04-24T08:26:51.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.018+0000 7f3cb7400640 1 -- 172.21.15.146:0/369105704 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3cb8004020 con 0x7f3cc019a820 2024-04-24T08:26:51.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.018+0000 7f3cc881c740 1 -- start start 2024-04-24T08:26:51.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94db40 0x558d2a94df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a9585f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a958b30 0x558d2a95af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cc881c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558d2a8b31d0 con 0x558d2a958b30 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cc881c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558d2a859450 con 0x558d2a94e460 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cc881c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558d2a8af190 con 0x558d2a94db40 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cb7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94db40 0x558d2a94df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a9585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a9585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41322/0 (socket says 172.21.15.146:41322) 2024-04-24T08:26:51.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 -- 172.21.15.146:0/4249097016 learned_addr learned my addr 172.21.15.146:0/4249097016 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a958b30 0x558d2a95af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 -- 172.21.15.146:0/4249097016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94db40 msgr2=0x558d2a94df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94db40 0x558d2a94df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 -- 172.21.15.146:0/4249097016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a958b30 msgr2=0x558d2a95af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a958b30 0x558d2a95af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cbea00640 1 -- 172.21.15.146:0/4249097016 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d2a916310 con 0x558d2a94e460 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.019+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a958b30 0x558d2a95af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.020+0000 7f3cbea00640 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a9585f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009900 tx=0x7f3cb0005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6ee21dc9c2d5e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:51.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.020+0000 7f3cb7400640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ca40a3d20 0x7f3ca40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.020+0000 7f3cafe00640 1 -- 172.21.15.146:0/4249097016 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0040070 con 0x558d2a94e460 2024-04-24T08:26:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.020+0000 7f3cafe00640 1 -- 172.21.15.146:0/4249097016 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cb0031870 con 0x558d2a94e460 2024-04-24T08:26:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.020+0000 7f3cafe00640 1 -- 172.21.15.146:0/4249097016 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0035040 con 0x558d2a94e460 2024-04-24T08:26:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.020+0000 7f3cb7400640 1 -- 172.21.15.146:0/369105704 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3cb80d0c10 con 0x7f3cc019a820 2024-04-24T08:26:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.020+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ca40a3d20 0x7f3ca40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 -- 172.21.15.146:0/4249097016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 msgr2=0x558d2a9585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a9585f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009900 tx=0x7f3cb0005be0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ca40a3d20 0x7f3ca40a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x558d2a94aff0 tx=0x7f3ca8013040 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 -- 172.21.15.146:0/4249097016 shutdown_connections 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a958b30 0x558d2a95af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a9585f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 --2- 172.21.15.146:0/4249097016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94db40 0x558d2a94df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 -- 172.21.15.146:0/4249097016 >> 172.21.15.146:0/4249097016 conn(0x558d2a949e60 msgr2=0x558d2a94a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 -- 172.21.15.146:0/4249097016 shutdown_connections 2024-04-24T08:26:51.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 -- 172.21.15.146:0/4249097016 wait complete. 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.021+0000 7f3cc881c740 1 -- start start 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 1 Processor -- start 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a9584c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a9557b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a955d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558d2a95b960 con 0x558d2a94db40 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558d2a86c810 con 0x558d2a958b30 2024-04-24T08:26:51.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558d2a95bae0 con 0x558d2a94e460 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cc881c740 10 client.0 _ll_register_callbacks cb 0x558d2a83a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a9584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a9584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50102/0 (socket says 172.21.15.146:50102) 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cbea00640 1 -- 172.21.15.146:0/3796303735 learned_addr learned my addr 172.21.15.146:0/3796303735 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.022+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a9557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cbf400640 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a955d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cbea00640 1 -- 172.21.15.146:0/3796303735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 msgr2=0x558d2a9557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cbea00640 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a9557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cbea00640 1 -- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 msgr2=0x558d2a955d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cbea00640 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a955d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cbea00640 1 -- 172.21.15.146:0/3796303735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d2a87d3f0 con 0x558d2a94db40 2024-04-24T08:26:51.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cbea00640 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a9584c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0006010 tx=0x7f3cb002a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d765c709d8491453 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:51.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cacc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:51.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cacc00640 1 -- 172.21.15.146:0/3796303735 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb004c020 con 0x558d2a94db40 2024-04-24T08:26:51.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cacc00640 1 -- 172.21.15.146:0/3796303735 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3cb0004960 con 0x558d2a94db40 2024-04-24T08:26:51.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.023+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cb0031870 con 0x558d2a94db40 2024-04-24T08:26:51.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cacc00640 1 -- 172.21.15.146:0/3796303735 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3cb0057050 con 0x558d2a94db40 2024-04-24T08:26:51.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 10 client.16383 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:51.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558d2a5c53f0 con 0x558d2a94db40 2024-04-24T08:26:51.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cacc00640 1 -- 172.21.15.146:0/3796303735 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cb0031870 con 0x558d2a94db40 2024-04-24T08:26:51.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cacc00640 1 -- 172.21.15.146:0/3796303735 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3cb003cae0 con 0x558d2a94db40 2024-04-24T08:26:51.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 10 client.16383 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:51.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 10 client.16383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:51.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 20 client.16383 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:51.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a968aa0 0x558d2a96ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 4 client.16383 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:51.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.024+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d2a8c0bf0 con 0x558d2a968aa0 2024-04-24T08:26:51.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.025+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a968aa0 0x558d2a96ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.025+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a968aa0 0x558d2a96ae90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6105f180964e3c69 server_cookie=dd7d815ebf7e7625 in_seq=0 out_seq=0 2024-04-24T08:26:51.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.025+0000 7f3cacc00640 10 client.16383 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.026+0000 7f3cacc00640 1 -- 172.21.15.146:0/3796303735 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d2a8c0bf0 con 0x558d2a968aa0 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.026+0000 7f3cacc00640 10 client.16383 handle_command_reply: tid=0 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.026+0000 7f3cc881c740 1 client.16383 shutdown 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 msgr2=0x558d2a9584c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a9584c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0006010 tx=0x7f3cb002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 shutdown_connections 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a968aa0 0x558d2a96ae90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a9557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a955d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cacc00640 0 client.16383 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:51.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 --2- 172.21.15.146:0/3796303735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a9584c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 >> 172.21.15.146:0/3796303735 conn(0x558d2a949e60 msgr2=0x558d2a94fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:51.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 shutdown_connections 2024-04-24T08:26:51.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 1 -- 172.21.15.146:0/3796303735 wait complete. 2024-04-24T08:26:51.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.027+0000 7f3cc881c740 20 client.16383 trim_cache size 0 max 16384 2024-04-24T08:26:51.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 Processor -- start 2024-04-24T08:26:51.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 -- start start 2024-04-24T08:26:51.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a964850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a954080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a957ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01b2600 con 0x558d2a94db40 2024-04-24T08:26:51.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01afed0 con 0x558d2a958b30 2024-04-24T08:26:51.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.041+0000 7f3cc881c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01add20 con 0x558d2a94e460 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a964850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a954080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cbf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a957ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a954080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38738/0 (socket says 172.21.15.146:38738) 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 -- 172.21.15.146:0/1075466710 learned_addr learned my addr 172.21.15.146:0/1075466710 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 -- 172.21.15.146:0/1075466710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 msgr2=0x558d2a957ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/1075466710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a957ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 -- 172.21.15.146:0/1075466710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 msgr2=0x558d2a964850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/1075466710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a964850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 -- 172.21.15.146:0/1075466710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d2a9560a0 con 0x558d2a94e460 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.042+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/1075466710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a954080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3cc010aa10 tx=0x7f3ca8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f6d207f4d2f0205 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3caf400640 1 -- 172.21.15.146:0/1075466710 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8032070 con 0x558d2a94e460 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3caf400640 1 -- 172.21.15.146:0/1075466710 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ca8020ce0 con 0x558d2a94e460 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3caf400640 1 -- 172.21.15.146:0/1075466710 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800f640 con 0x558d2a94e460 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 -- 172.21.15.146:0/1075466710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 msgr2=0x558d2a954080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1075466710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a954080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3cc010aa10 tx=0x7f3ca8005f90 comp rx=0 tx=0).stop 2024-04-24T08:26:51.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 -- 172.21.15.146:0/1075466710 shutdown_connections 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1075466710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a94e460 0x558d2a954080 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1075466710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a958b30 0x558d2a957ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1075466710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a964850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 -- 172.21.15.146:0/1075466710 >> 172.21.15.146:0/1075466710 conn(0x558d2a949e60 msgr2=0x558d2a94a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 -- 172.21.15.146:0/1075466710 shutdown_connections 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.043+0000 7f3cc881c740 1 -- 172.21.15.146:0/1075466710 wait complete. 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 -- start start 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 Processor -- start 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a95a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a95a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a958b30 0x558d2a95aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01ae390 con 0x558d2a94db40 2024-04-24T08:26:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01af800 con 0x558d2a94e460 2024-04-24T08:26:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc01aea60 con 0x558d2a958b30 2024-04-24T08:26:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.044+0000 7f3cc881c740 10 client.0 _ll_register_callbacks cb 0x7f3ca80099f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a95a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cbea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a95a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50130/0 (socket says 172.21.15.146:50130) 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cbea00640 1 -- 172.21.15.146:0/1257357116 learned_addr learned my addr 172.21.15.146:0/1257357116 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a95a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cbf400640 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a958b30 0x558d2a95aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cb7e00640 1 -- 172.21.15.146:0/1257357116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a958b30 msgr2=0x558d2a95aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a958b30 0x558d2a95aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cb7e00640 1 -- 172.21.15.146:0/1257357116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 msgr2=0x558d2a95a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a95a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cb7e00640 1 -- 172.21.15.146:0/1257357116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d2a68f7d0 con 0x558d2a94e460 2024-04-24T08:26:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cb7e00640 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a95a590 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x558d2a956e30 tx=0x7f3ca8004480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f16a81b0c92ad9ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.045+0000 7f3cacc00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.046+0000 7f3cacc00640 1 -- 172.21.15.146:0/1257357116 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8032040 con 0x558d2a94e460 2024-04-24T08:26:51.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.046+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ca8020ce0 con 0x558d2a94e460 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.046+0000 7f3cacc00640 1 -- 172.21.15.146:0/1257357116 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f3ca802d070 con 0x558d2a94e460 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.046+0000 7f3cacc00640 1 -- 172.21.15.146:0/1257357116 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3ca802d270 con 0x558d2a94e460 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.046+0000 7f3cc881c740 10 client.26531 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.046+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558d2a9cedb0 con 0x558d2a94e460 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.046+0000 7f3cacc00640 1 -- 172.21.15.146:0/1257357116 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ca8020ce0 con 0x558d2a94e460 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cacc00640 1 -- 172.21.15.146:0/1257357116 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3ca8004880 con 0x558d2a94e460 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cc881c740 10 client.26531 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:51.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cc881c740 10 client.26531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cc881c740 20 client.26531 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a9cf450 0x558d2a9d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cc881c740 4 client.26531 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558d2a8baa60 con 0x558d2a9cf450 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cbea00640 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a9cf450 0x558d2a9d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cacc00640 10 client.26531 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.047+0000 7f3cbea00640 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a9cf450 0x558d2a9d1870 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df84155c331ff800 server_cookie=3b5ee51a38ab895c in_seq=0 out_seq=0 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cacc00640 1 -- 172.21.15.146:0/1257357116 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558d2a8baa60 con 0x558d2a9cf450 2024-04-24T08:26:51.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cacc00640 10 client.26531 handle_command_reply: tid=0 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 client.26531 shutdown 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 msgr2=0x558d2a95a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a95a590 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x558d2a956e30 tx=0x7f3ca8004480 comp rx=0 tx=0).stop 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 shutdown_connections 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558d2a9cf450 0x558d2a9d1870 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558d2a958b30 0x558d2a95aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558d2a94e460 0x558d2a95a590 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cacc00640 0 client.26531 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 --2- 172.21.15.146:0/1257357116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558d2a94db40 0x558d2a95a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 >> 172.21.15.146:0/1257357116 conn(0x558d2a949e60 msgr2=0x558d2a9486c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 shutdown_connections 2024-04-24T08:26:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 1 -- 172.21.15.146:0/1257357116 wait complete. 2024-04-24T08:26:51.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.048+0000 7f3cc881c740 20 client.26531 trim_cache size 0 max 16384 2024-04-24T08:26:51.058 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:26:51.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ca40a3d20 msgr2=0x7f3ca40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ca40a3d20 0x7f3ca40a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x558d2a94aff0 tx=0x7f3ca8013040 comp rx=0 tx=0).stop 2024-04-24T08:26:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc019a820 msgr2=0x7f3cc019acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc019a820 0x7f3cc019acd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3cb800efd0 tx=0x7f3cb800c6a0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 shutdown_connections 2024-04-24T08:26:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3ca40a3d20 0x7f3ca40a61e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cc01093f0 0x7f3cc01a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3cc019a820 0x7f3cc019acd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 --2- 172.21.15.146:0/369105704 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cc0104320 0x7f3cc01a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 >> 172.21.15.146:0/369105704 conn(0x7f3cc00ffb40 msgr2=0x7f3cc0101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 shutdown_connections 2024-04-24T08:26:51.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:51.049+0000 7f3cbfe00640 1 -- 172.21.15.146:0/369105704 wait complete. 2024-04-24T08:26:52.068 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:26:52.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:26:52.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:26:52.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 -- 172.21.15.146:0/1606285384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 msgr2=0x7f29681046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 --2- 172.21.15.146:0/1606285384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681046f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2958009900 tx=0x7f295802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 -- 172.21.15.146:0/1606285384 shutdown_connections 2024-04-24T08:26:52.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 --2- 172.21.15.146:0/1606285384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29681093e0 0x7f2968110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 --2- 172.21.15.146:0/1606285384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2968104cc0 0x7f2968108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 --2- 172.21.15.146:0/1606285384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681046f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 -- 172.21.15.146:0/1606285384 >> 172.21.15.146:0/1606285384 conn(0x7f29680ffb70 msgr2=0x7f2968101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 -- 172.21.15.146:0/1606285384 shutdown_connections 2024-04-24T08:26:52.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.293+0000 7f296d200640 1 -- 172.21.15.146:0/1606285384 wait complete. 2024-04-24T08:26:52.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.294+0000 7f296d200640 1 Processor -- start 2024-04-24T08:26:52.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.294+0000 7f296d200640 1 -- start start 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f296d200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f296d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29681093e0 0x7f29681a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f296d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f296819a830 0x7f296819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f296d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2968114470 con 0x7f296819a830 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f296d200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29681142f0 con 0x7f2968104310 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f296d200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29681145f0 con 0x7f29681093e0 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2966a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29681093e0 0x7f29681a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41378/0 (socket says 172.21.15.146:41378) 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 -- 172.21.15.146:0/3298573890 learned_addr learned my addr 172.21.15.146:0/3298573890 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f296819a830 0x7f296819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 -- 172.21.15.146:0/3298573890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29681093e0 msgr2=0x7f29681a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29681093e0 0x7f29681a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 -- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f296819a830 msgr2=0x7f296819ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f296819a830 0x7f296819ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967400640 1 -- 172.21.15.146:0/3298573890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296819b3c0 con 0x7f2968104310 2024-04-24T08:26:52.300 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2966a00640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29681093e0 0x7f29681a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:52.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.295+0000 7f2967e00640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f296819a830 0x7f296819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:52.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.296+0000 7f2967400640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681a0670 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2958006010 tx=0x7f2958004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.296+0000 7f295fe00640 1 -- 172.21.15.146:0/3298573890 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295803f030 con 0x7f2968104310 2024-04-24T08:26:52.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.296+0000 7f295fe00640 1 -- 172.21.15.146:0/3298573890 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2958004440 con 0x7f2968104310 2024-04-24T08:26:52.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.296+0000 7f295fe00640 1 -- 172.21.15.146:0/3298573890 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958035070 con 0x7f2968104310 2024-04-24T08:26:52.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.296+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f296819b650 con 0x7f2968104310 2024-04-24T08:26:52.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.296+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29681a77b0 con 0x7f2968104310 2024-04-24T08:26:52.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.299+0000 7f2971785740 1 Processor -- start 2024-04-24T08:26:52.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2971785740 1 -- start start 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f295fe00640 1 -- 172.21.15.146:0/3298573890 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f2958042030 con 0x7f2968104310 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2971785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fa270 0x5643000fa650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2971785740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fab90 0x564300104d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2971785740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564300105260 0x564300107650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2971785740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56430005fe70 con 0x564300105260 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2971785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564300006380 con 0x5643000fa270 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2971785740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56430005be30 con 0x5643000fab90 2024-04-24T08:26:52.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fa270 0x5643000fa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fa270 0x5643000fa650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41382/0 (socket says 172.21.15.146:41382) 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 -- 172.21.15.146:0/2484785211 learned_addr learned my addr 172.21.15.146:0/2484785211 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fab90 0x564300104d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2966a00640 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564300105260 0x564300107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 -- 172.21.15.146:0/2484785211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fab90 msgr2=0x564300104d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fab90 0x564300104d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 -- 172.21.15.146:0/2484785211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564300105260 msgr2=0x564300107650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564300105260 0x564300107650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 -- 172.21.15.146:0/2484785211 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642ffe3a7e0 con 0x5643000fa270 2024-04-24T08:26:52.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967e00640 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fa270 0x5643000fa650 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2960011470 tx=0x7f2960011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a0bd83b17151b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f295fe00640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f29480a3d20 0x7f29480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f295fe00640 1 -- 172.21.15.146:0/3298573890 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f29580f0a70 con 0x7f2968104310 2024-04-24T08:26:52.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f295cc00640 1 -- 172.21.15.146:0/2484785211 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296001e070 con 0x5643000fa270 2024-04-24T08:26:52.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f295cc00640 1 -- 172.21.15.146:0/2484785211 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f296000fa20 con 0x5643000fa270 2024-04-24T08:26:52.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f295cc00640 1 -- 172.21.15.146:0/2484785211 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960019b80 con 0x5643000fa270 2024-04-24T08:26:52.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.301+0000 7f2967400640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f29480a3d20 0x7f29480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 -- 172.21.15.146:0/2484785211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fa270 msgr2=0x5643000fa650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fa270 0x5643000fa650 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2960011470 tx=0x7f2960011930 comp rx=0 tx=0).stop 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2967400640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f29480a3d20 0x7f29480a61e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x5643000fb830 tx=0x7f29580f44a0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 -- 172.21.15.146:0/2484785211 shutdown_connections 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fab90 0x564300104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564300105260 0x564300107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 --2- 172.21.15.146:0/2484785211 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fa270 0x5643000fa650 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 -- 172.21.15.146:0/2484785211 >> 172.21.15.146:0/2484785211 conn(0x5643000f6590 msgr2=0x5643000f6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 -- 172.21.15.146:0/2484785211 shutdown_connections 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.302+0000 7f2971785740 1 -- 172.21.15.146:0/2484785211 wait complete. 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 -- start start 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 Processor -- start 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fa270 0x564300102210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fab90 0x564300102750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564300105260 0x564300102c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564300108050 con 0x5643000fa270 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564300019660 con 0x5643000fab90 2024-04-24T08:26:52.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5643001081d0 con 0x564300105260 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2971785740 10 client.0 _ll_register_callbacks cb 0x564300007aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2966a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fab90 0x564300102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fa270 0x564300102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2966a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fab90 0x564300102750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41396/0 (socket says 172.21.15.146:41396) 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2966a00640 1 -- 172.21.15.146:0/3455363013 learned_addr learned my addr 172.21.15.146:0/3455363013 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564300105260 0x564300102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 -- 172.21.15.146:0/3455363013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564300105260 msgr2=0x564300102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564300105260 0x564300102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 -- 172.21.15.146:0/3455363013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fab90 msgr2=0x564300102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fab90 0x564300102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 -- 172.21.15.146:0/3455363013 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642ffe9af70 con 0x5643000fa270 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2967e00640 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fa270 0x564300102210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2960000c00 tx=0x7f29600196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d6db046a2ef8f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.304+0000 7f2955600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.306+0000 7f2955600640 1 -- 172.21.15.146:0/3455363013 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296001e070 con 0x5643000fa270 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.306+0000 7f2955600640 1 -- 172.21.15.146:0/3455363013 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2960019900 con 0x5643000fa270 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.306+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f296000fa20 con 0x5643000fa270 2024-04-24T08:26:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.306+0000 7f2955600640 1 -- 172.21.15.146:0/3455363013 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2960019b00 con 0x5643000fa270 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.306+0000 7f2971785740 10 client.16401 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.306+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5642ffd32170 con 0x5643000fa270 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.306+0000 7f2955600640 1 -- 172.21.15.146:0/3455363013 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f296000fa20 con 0x5643000fa270 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2955600640 1 -- 172.21.15.146:0/3455363013 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2960004a20 con 0x5643000fa270 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2971785740 10 client.16401 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2971785740 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2971785740 20 client.16401 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:52.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2971785740 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564300115520 0x564300117910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2971785740 4 client.16401 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56430006d890 con 0x564300115520 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2966a00640 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564300115520 0x564300117910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2966a00640 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564300115520 0x564300117910 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5781ba36ae075b0 server_cookie=d8e38bf52268846e in_seq=0 out_seq=0 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.307+0000 7f2955600640 10 client.16401 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2955600640 1 -- 172.21.15.146:0/3455363013 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56430006d890 con 0x564300115520 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2955600640 10 client.16401 handle_command_reply: tid=0 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 client.16401 shutdown 2024-04-24T08:26:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fa270 msgr2=0x564300102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fa270 0x564300102210 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2960000c00 tx=0x7f29600196c0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 shutdown_connections 2024-04-24T08:26:52.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564300115520 0x564300117910 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564300105260 0x564300102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643000fab90 0x564300102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2955600640 0 client.16401 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 --2- 172.21.15.146:0/3455363013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fa270 0x564300102210 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 >> 172.21.15.146:0/3455363013 conn(0x5643000f6590 msgr2=0x5643000fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 shutdown_connections 2024-04-24T08:26:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 1 -- 172.21.15.146:0/3455363013 wait complete. 2024-04-24T08:26:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.309+0000 7f2971785740 20 client.16401 trim_cache size 0 max 16384 2024-04-24T08:26:52.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.327+0000 7f2971785740 1 Processor -- start 2024-04-24T08:26:52.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.327+0000 7f2971785740 1 -- start start 2024-04-24T08:26:52.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2971785740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x564300110f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2971785740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430011d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2971785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430011d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2971785740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f29681b26b0 con 0x5643000fab90 2024-04-24T08:26:52.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2971785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29681aff80 con 0x564300105260 2024-04-24T08:26:52.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2971785740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29681addd0 con 0x5643000fa270 2024-04-24T08:26:52.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x564300110f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x564300110f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38784/0 (socket says 172.21.15.146:38784) 2024-04-24T08:26:52.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 -- 172.21.15.146:0/1538123359 learned_addr learned my addr 172.21.15.146:0/1538123359 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430011d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2966a00640 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430011d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 -- 172.21.15.146:0/1538123359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 msgr2=0x56430011d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430011d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 -- 172.21.15.146:0/1538123359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 msgr2=0x56430011d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.328+0000 7f2967e00640 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430011d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.330+0000 7f2967e00640 1 -- 172.21.15.146:0/1538123359 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564300100760 con 0x5643000fa270 2024-04-24T08:26:52.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.330+0000 7f2967e00640 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x564300110f40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x56430011db00 tx=0x7f296000fee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9abdafdfcbd6e105 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.330+0000 7f2957e00640 1 -- 172.21.15.146:0/1538123359 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296001e070 con 0x5643000fa270 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.330+0000 7f2957e00640 1 -- 172.21.15.146:0/1538123359 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f296001f040 con 0x5643000fa270 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.330+0000 7f2957e00640 1 -- 172.21.15.146:0/1538123359 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960026910 con 0x5643000fa270 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- 172.21.15.146:0/1538123359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 msgr2=0x564300110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x564300110f40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x56430011db00 tx=0x7f296000fee0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- 172.21.15.146:0/1538123359 shutdown_connections 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430011d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430011d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 --2- 172.21.15.146:0/1538123359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x564300110f40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- 172.21.15.146:0/1538123359 >> 172.21.15.146:0/1538123359 conn(0x5643000f6590 msgr2=0x5643000f56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- 172.21.15.146:0/1538123359 shutdown_connections 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- 172.21.15.146:0/1538123359 wait complete. 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- start start 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 Processor -- start 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x56430010d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430010d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430010a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f29681ae440 con 0x5643000fab90 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29681af8b0 con 0x564300105260 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29681aeb10 con 0x5643000fa270 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2971785740 10 client.0 _ll_register_callbacks cb 0x7f296001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2967e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x56430010d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2967e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x56430010d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38786/0 (socket says 172.21.15.146:38786) 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2967e00640 1 -- 172.21.15.146:0/4120638719 learned_addr learned my addr 172.21.15.146:0/4120638719 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2966a00640 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430010d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.331+0000 7f2967e00640 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430010a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2967e00640 1 -- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 msgr2=0x56430010a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2967e00640 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430010a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2967e00640 1 -- 172.21.15.146:0/4120638719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 msgr2=0x56430010d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2967e00640 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430010d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2967e00640 1 -- 172.21.15.146:0/4120638719 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56430010ade0 con 0x5643000fa270 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2967e00640 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x56430010d450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2960004ec0 tx=0x7f2960005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ae172c48308641 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2955600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:26:52.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2955600640 1 -- 172.21.15.146:0/4120638719 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296001e040 con 0x5643000fa270 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f296001f040 con 0x5643000fa270 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2955600640 1 -- 172.21.15.146:0/4120638719 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f2960026b00 con 0x5643000fa270 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2955600640 1 -- 172.21.15.146:0/4120638719 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f29600035f0 con 0x5643000fa270 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2971785740 10 client.26536 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.333+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56430017c520 con 0x5643000fa270 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2955600640 1 -- 172.21.15.146:0/4120638719 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f296001f040 con 0x5643000fa270 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2955600640 1 -- 172.21.15.146:0/4120638719 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f296003c5b0 con 0x5643000fa270 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2971785740 10 client.26536 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:52.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2971785740 10 client.26536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2971785740 20 client.26536 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2971785740 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56430017ca60 0x56430017ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2971785740 4 client.26536 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:26:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564300067700 con 0x56430017ca60 2024-04-24T08:26:52.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.334+0000 7f2966a00640 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56430017ca60 0x56430017ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2966a00640 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56430017ca60 0x56430017ee50 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30f5514eaf8561b5 server_cookie=1ee158863d70cbb5 in_seq=0 out_seq=0 2024-04-24T08:26:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2955600640 10 client.26536 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2955600640 1 -- 172.21.15.146:0/4120638719 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564300067700 con 0x56430017ca60 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2955600640 10 client.26536 handle_command_reply: tid=0 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 client.26536 shutdown 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 msgr2=0x56430010d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x56430010d450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2960004ec0 tx=0x7f2960005970 comp rx=0 tx=0).stop 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 shutdown_connections 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56430017ca60 0x56430017ee50 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564300105260 0x56430010a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643000fab90 0x56430010d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 --2- 172.21.15.146:0/4120638719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643000fa270 0x56430010d450 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2955600640 0 client.26536 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 >> 172.21.15.146:0/4120638719 conn(0x5643000f6590 msgr2=0x5643000fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 shutdown_connections 2024-04-24T08:26:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.335+0000 7f2971785740 1 -- 172.21.15.146:0/4120638719 wait complete. 2024-04-24T08:26:52.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f2971785740 20 client.26536 trim_cache size 0 max 16384 2024-04-24T08:26:52.344 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:26:52.344 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:26:52.345 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "ca83eefc-e76d-4996-b47f-8a0e50b89128", 2024-04-24T08:26:52.345 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:26:52.345 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:26:52.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f29480a3d20 msgr2=0x7f29480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f29480a3d20 0x7f29480a61e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x5643000fb830 tx=0x7f29580f44a0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 msgr2=0x7f29681a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681a0670 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2958006010 tx=0x7f2958004040 comp rx=0 tx=0).stop 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 shutdown_connections 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f29480a3d20 0x7f29480a61e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29681093e0 0x7f29681a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f296819a830 0x7f296819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 --2- 172.21.15.146:0/3298573890 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2968104310 0x7f29681a0670 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.337+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 >> 172.21.15.146:0/3298573890 conn(0x7f29680ffb70 msgr2=0x7f2968100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.338+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 shutdown_connections 2024-04-24T08:26:52.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.338+0000 7f296d200640 1 -- 172.21.15.146:0/3298573890 wait complete. 2024-04-24T08:26:52.360 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:26:52.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:26:52.423 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:52 smithi146 ceph-mon[27805]: pgmap v173: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:52.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 -- 172.21.15.146:0/1345375157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 msgr2=0x7f5b581024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 --2- 172.21.15.146:0/1345375157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b581024f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009900 tx=0x7f5b4802a400 comp rx=0 tx=0).stop 2024-04-24T08:26:52.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 -- 172.21.15.146:0/1345375157 shutdown_connections 2024-04-24T08:26:52.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 --2- 172.21.15.146:0/1345375157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58102ac0 0x7f5b58106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 --2- 172.21.15.146:0/1345375157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b581071e0 0x7f5b5810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 --2- 172.21.15.146:0/1345375157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b581024f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 -- 172.21.15.146:0/1345375157 >> 172.21.15.146:0/1345375157 conn(0x7f5b580fd9d0 msgr2=0x7f5b580ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 -- 172.21.15.146:0/1345375157 shutdown_connections 2024-04-24T08:26:52.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.620+0000 7f5b57e00640 1 -- 172.21.15.146:0/1345375157 wait complete. 2024-04-24T08:26:52.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.621+0000 7f5b57e00640 1 Processor -- start 2024-04-24T08:26:52.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b57e00640 1 -- start start 2024-04-24T08:26:52.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b58074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b581071e0 0x7f5b58074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b57e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b58075fa0 0x7f5b58074d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b57e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58112340 con 0x7f5b58075fa0 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b57e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581121c0 con 0x7f5b58102110 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b57e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581124c0 con 0x7f5b581071e0 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b58074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.622+0000 7f5b56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b581071e0 0x7f5b58074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b58074300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41444/0 (socket says 172.21.15.146:41444) 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56a00640 1 -- 172.21.15.146:0/954656297 learned_addr learned my addr 172.21.15.146:0/954656297 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b57400640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b58075fa0 0x7f5b58074d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56a00640 1 -- 172.21.15.146:0/954656297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b581071e0 msgr2=0x7f5b58074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56a00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b581071e0 0x7f5b58074840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56a00640 1 -- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b58075fa0 msgr2=0x7f5b58074d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56a00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b58075fa0 0x7f5b58074d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56a00640 1 -- 172.21.15.146:0/954656297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5819ca50 con 0x7f5b58102110 2024-04-24T08:26:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b56000640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b581071e0 0x7f5b58074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.623+0000 7f5b57400640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b58075fa0 0x7f5b58074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:52.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.624+0000 7f5b56a00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b58074300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5b48006010 tx=0x7f5b48003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.624+0000 7f5b4f400640 1 -- 172.21.15.146:0/954656297 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4803f070 con 0x7f5b58102110 2024-04-24T08:26:52.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.624+0000 7f5b4f400640 1 -- 172.21.15.146:0/954656297 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b48031c90 con 0x7f5b58102110 2024-04-24T08:26:52.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.624+0000 7f5b4f400640 1 -- 172.21.15.146:0/954656297 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48035030 con 0x7f5b58102110 2024-04-24T08:26:52.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.624+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5819cce0 con 0x7f5b58102110 2024-04-24T08:26:52.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.624+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b5819d180 con 0x7f5b58102110 2024-04-24T08:26:52.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.629+0000 7f5b607cb740 1 Processor -- start 2024-04-24T08:26:52.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b607cb740 1 -- start start 2024-04-24T08:26:52.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c12a30 0x555fc7c12e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c1d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c1da20 0x555fc7c1fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b4f400640 1 -- 172.21.15.146:0/954656297 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f5b48042070 con 0x7f5b58102110 2024-04-24T08:26:52.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b607cb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555fc7b781f0 con 0x555fc7c13350 2024-04-24T08:26:52.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b607cb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555fc7b1e350 con 0x555fc7c12a30 2024-04-24T08:26:52.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.630+0000 7f5b607cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fc7b741b0 con 0x555fc7c1da20 2024-04-24T08:26:52.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b57400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c12a30 0x555fc7c12e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c1da20 0x555fc7c1fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b57400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c12a30 0x555fc7c12e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41456/0 (socket says 172.21.15.146:41456) 2024-04-24T08:26:52.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b57400640 1 -- 172.21.15.146:0/2822456220 learned_addr learned my addr 172.21.15.146:0/2822456220 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c1da20 0x555fc7c1fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38812/0 (socket says 172.21.15.146:38812) 2024-04-24T08:26:52.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56a00640 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c1d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 -- 172.21.15.146:0/2822456220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c12a30 msgr2=0x555fc7c12e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c12a30 0x555fc7c12e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 -- 172.21.15.146:0/2822456220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 msgr2=0x555fc7c1d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c1d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 -- 172.21.15.146:0/2822456220 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc7b72f50 con 0x555fc7c1da20 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b57400640 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c12a30 0x555fc7c12e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.631+0000 7f5b56000640 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c1da20 0x555fc7c1fe10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b58075b50 tx=0x7f5b4400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74f2a196f6d4bada server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.632+0000 7f5b43e00640 1 -- 172.21.15.146:0/2822456220 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4401d070 con 0x555fc7c1da20 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.632+0000 7f5b43e00640 1 -- 172.21.15.146:0/2822456220 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44010d20 con 0x555fc7c1da20 2024-04-24T08:26:52.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.632+0000 7f5b43e00640 1 -- 172.21.15.146:0/2822456220 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44016340 con 0x555fc7c1da20 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.632+0000 7f5b4f400640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5b3c0a3da0 0x7f5b3c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b4f400640 1 -- 172.21.15.146:0/954656297 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5b480f0f90 con 0x7f5b58102110 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 -- 172.21.15.146:0/2822456220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c1da20 msgr2=0x555fc7c1fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c1da20 0x555fc7c1fe10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5b58075b50 tx=0x7f5b4400f790 comp rx=0 tx=0).stop 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b57400640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5b3c0a3da0 0x7f5b3c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 -- 172.21.15.146:0/2822456220 shutdown_connections 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c1da20 0x555fc7c1fe10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c1d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2822456220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c12a30 0x555fc7c12e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 -- 172.21.15.146:0/2822456220 >> 172.21.15.146:0/2822456220 conn(0x555fc7c0ed50 msgr2=0x555fc7c0f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.633+0000 7f5b607cb740 1 -- 172.21.15.146:0/2822456220 shutdown_connections 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.634+0000 7f5b607cb740 1 -- 172.21.15.146:0/2822456220 wait complete. 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.634+0000 7f5b57400640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5b3c0a3da0 0x7f5b3c0a6260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000a580 tx=0x7f5b5000d040 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.634+0000 7f5b607cb740 1 -- start start 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.634+0000 7f5b607cb740 1 Processor -- start 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c1a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c13350 0x555fc7c1aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c1da20 0x555fc7c1b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b607cb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555fc7c20810 con 0x555fc7c1da20 2024-04-24T08:26:52.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b607cb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555fc7b31690 con 0x555fc7c13350 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b607cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fc7c20990 con 0x555fc7c12a30 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b607cb740 10 client.0 _ll_register_callbacks cb 0x555fc7b6d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b56000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c13350 0x555fc7c1aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b56000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c13350 0x555fc7c1aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:41462/0 (socket says 172.21.15.146:41462) 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b56000640 1 -- 172.21.15.146:0/2063396036 learned_addr learned my addr 172.21.15.146:0/2063396036 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.635+0000 7f5b57400640 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c1da20 0x555fc7c1b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b56a00640 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c1a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b56000640 1 -- 172.21.15.146:0/2063396036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 msgr2=0x555fc7c1a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b56000640 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c1a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b56000640 1 -- 172.21.15.146:0/2063396036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c1da20 msgr2=0x555fc7c1b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b56000640 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c1da20 0x555fc7c1b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b56000640 1 -- 172.21.15.146:0/2063396036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc7b30920 con 0x555fc7c13350 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b56000640 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c13350 0x555fc7c1aea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5b44010010 tx=0x7f5b440186d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285f4d87f039ae06 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b40c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b40c00640 1 -- 172.21.15.146:0/2063396036 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4401d050 con 0x555fc7c13350 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b44010d20 con 0x555fc7c13350 2024-04-24T08:26:52.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.636+0000 7f5b40c00640 1 -- 172.21.15.146:0/2063396036 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5b44010850 con 0x555fc7c13350 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.637+0000 7f5b40c00640 1 -- 172.21.15.146:0/2063396036 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5b44031050 con 0x555fc7c13350 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.637+0000 7f5b607cb740 10 client.26561 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.637+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555fc79547d0 con 0x555fc7c13350 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.637+0000 7f5b40c00640 1 -- 172.21.15.146:0/2063396036 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b44010d20 con 0x555fc7c13350 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.637+0000 7f5b40c00640 1 -- 172.21.15.146:0/2063396036 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5b440047e0 con 0x555fc7c13350 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b607cb740 10 client.26561 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b607cb740 10 client.26561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b607cb740 20 client.26561 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:52.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c2dca0 0x555fc7c30090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b607cb740 4 client.26561 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:26:52.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fc7b85c10 con 0x555fc7c2dca0 2024-04-24T08:26:52.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b56a00640 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c2dca0 0x555fc7c30090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.639+0000 7f5b40c00640 10 client.26561 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.638+0000 7f5b56a00640 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c2dca0 0x555fc7c30090 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a22a64cb91d7f8db server_cookie=545510b1682304e9 in_seq=0 out_seq=0 2024-04-24T08:26:52.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.640+0000 7f5b40c00640 1 -- 172.21.15.146:0/2063396036 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fc7b85c10 con 0x555fc7c2dca0 2024-04-24T08:26:52.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.640+0000 7f5b40c00640 10 client.26561 handle_command_reply: tid=0 2024-04-24T08:26:52.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.640+0000 7f5b607cb740 1 client.26561 shutdown 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.640+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c13350 msgr2=0x555fc7c1aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.640+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c13350 0x555fc7c1aea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5b44010010 tx=0x7f5b440186d0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.640+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 shutdown_connections 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.640+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c2dca0 0x555fc7c30090 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c1da20 0x555fc7c1b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c13350 0x555fc7c1aea0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2063396036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c1a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 >> 172.21.15.146:0/2063396036 conn(0x555fc7c0ed50 msgr2=0x555fc7c0dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b40c00640 0 client.26561 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 shutdown_connections 2024-04-24T08:26:52.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b607cb740 1 -- 172.21.15.146:0/2063396036 wait complete. 2024-04-24T08:26:52.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.641+0000 7f5b607cb740 20 client.26561 trim_cache size 0 max 16384 2024-04-24T08:26:52.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 Processor -- start 2024-04-24T08:26:52.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 -- start start 2024-04-24T08:26:52.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c12a30 0x555fc7c0e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c13350 0x555fc7c151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c35720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581bf290 con 0x555fc7c12a30 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581bcb60 con 0x555fc7c1da20 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.660+0000 7f5b607cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581a3780 con 0x555fc7c13350 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c12a30 0x555fc7c0e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c13350 0x555fc7c151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c12a30 0x555fc7c0e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50230/0 (socket says 172.21.15.146:50230) 2024-04-24T08:26:52.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b57400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c35720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56a00640 1 -- 172.21.15.146:0/2285142609 learned_addr learned my addr 172.21.15.146:0/2285142609 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56000640 1 -- 172.21.15.146:0/2285142609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 msgr2=0x555fc7c35720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56000640 1 --2- 172.21.15.146:0/2285142609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c35720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56000640 1 -- 172.21.15.146:0/2285142609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c12a30 msgr2=0x555fc7c0e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56000640 1 --2- 172.21.15.146:0/2285142609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c12a30 0x555fc7c0e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56000640 1 -- 172.21.15.146:0/2285142609 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc788a3f0 con 0x555fc7c13350 2024-04-24T08:26:52.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.661+0000 7f5b56000640 1 --2- 172.21.15.146:0/2285142609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c13350 0x555fc7c151d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b44004da0 tx=0x7f5b44010f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e53e04a6805248e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b43400640 1 -- 172.21.15.146:0/2285142609 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44039070 con 0x555fc7c13350 2024-04-24T08:26:52.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b43400640 1 -- 172.21.15.146:0/2285142609 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b4401e040 con 0x555fc7c13350 2024-04-24T08:26:52.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b43400640 1 -- 172.21.15.146:0/2285142609 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4401d070 con 0x555fc7c13350 2024-04-24T08:26:52.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b607cb740 1 -- 172.21.15.146:0/2285142609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c13350 msgr2=0x555fc7c151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2285142609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c13350 0x555fc7c151d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b44004da0 tx=0x7f5b44010f90 comp rx=0 tx=0).stop 2024-04-24T08:26:52.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b607cb740 1 -- 172.21.15.146:0/2285142609 shutdown_connections 2024-04-24T08:26:52.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2285142609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c13350 0x555fc7c151d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2285142609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c35720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b607cb740 1 --2- 172.21.15.146:0/2285142609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c12a30 0x555fc7c0e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.662+0000 7f5b607cb740 1 -- 172.21.15.146:0/2285142609 >> 172.21.15.146:0/2285142609 conn(0x555fc7c0ed50 msgr2=0x555fc7c1b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.663+0000 7f5b607cb740 1 -- 172.21.15.146:0/2285142609 shutdown_connections 2024-04-24T08:26:52.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.663+0000 7f5b607cb740 1 -- 172.21.15.146:0/2285142609 wait complete. 2024-04-24T08:26:52.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.663+0000 7f5b607cb740 1 -- start start 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.663+0000 7f5b607cb740 1 Processor -- start 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.663+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c25e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.663+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c171d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b607cb740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c17710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b607cb740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581a3dd0 con 0x555fc7c13350 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b607cb740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581bc490 con 0x555fc7c1da20 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b607cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581a44a0 con 0x555fc7c12a30 2024-04-24T08:26:52.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b607cb740 10 client.0 _ll_register_callbacks cb 0x7f5b4401b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b56000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c171d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b57400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c17710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b56000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c171d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50246/0 (socket says 172.21.15.146:50246) 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b56000640 1 -- 172.21.15.146:0/1721018960 learned_addr learned my addr 172.21.15.146:0/1721018960 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b56a00640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c25e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b57400640 1 -- 172.21.15.146:0/1721018960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 msgr2=0x555fc7c25e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b57400640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c25e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b57400640 1 -- 172.21.15.146:0/1721018960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 msgr2=0x555fc7c171d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:26:52.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b57400640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c171d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b57400640 1 -- 172.21.15.146:0/1721018960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc784a0f0 con 0x555fc7c1da20 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b56000640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c171d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.664+0000 7f5b56a00640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c25e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b57400640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c17710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x555fc7c1bcf0 tx=0x7f5b50004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47ea8b10cef6f4a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b40c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b40c00640 1 -- 172.21.15.146:0/1721018960 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50029030 con 0x555fc7c1da20 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b4401e040 con 0x555fc7c1da20 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b40c00640 1 -- 172.21.15.146:0/1721018960 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5b50024680 con 0x555fc7c1da20 2024-04-24T08:26:52.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b40c00640 1 -- 172.21.15.146:0/1721018960 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5b50033050 con 0x555fc7c1da20 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b607cb740 10 client.26567 fetch_fsmap learned FSMap version 26 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555fc7b3d810 con 0x555fc7c1da20 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.665+0000 7f5b40c00640 1 -- 172.21.15.146:0/1721018960 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b4401e040 con 0x555fc7c1da20 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b40c00640 1 -- 172.21.15.146:0/1721018960 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5b500259e0 con 0x555fc7c1da20 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b607cb740 10 client.26567 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b607cb740 10 client.26567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b607cb740 20 client.26567 populate_metadata read hostname 'smithi146' 2024-04-24T08:26:52.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b607cb740 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c94240 0x555fc7c96630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b607cb740 4 client.26567 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555fc7b7fa80 con 0x555fc7c94240 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b56a00640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c94240 0x555fc7c96630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b40c00640 10 client.26567 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.666+0000 7f5b56a00640 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c94240 0x555fc7c96630 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=548cea658caf0f2a server_cookie=a35d1228d934adf7 in_seq=0 out_seq=0 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b40c00640 1 -- 172.21.15.146:0/1721018960 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555fc7b7fa80 con 0x555fc7c94240 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b40c00640 10 client.26567 handle_command_reply: tid=0 2024-04-24T08:26:52.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 client.26567 shutdown 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 msgr2=0x555fc7c17710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c17710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x555fc7c1bcf0 tx=0x7f5b50004a40 comp rx=0 tx=0).stop 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 shutdown_connections 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555fc7c94240 0x555fc7c96630 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555fc7c1da20 0x555fc7c17710 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555fc7c13350 0x555fc7c171d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 --2- 172.21.15.146:0/1721018960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fc7c12a30 0x555fc7c25e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 >> 172.21.15.146:0/1721018960 conn(0x555fc7c0ed50 msgr2=0x555fc7c1c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b40c00640 0 client.26567 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:26:52.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 shutdown_connections 2024-04-24T08:26:52.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 1 -- 172.21.15.146:0/1721018960 wait complete. 2024-04-24T08:26:52.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.667+0000 7f5b607cb740 20 client.26567 trim_cache size 0 max 16384 2024-04-24T08:26:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:26:52.677 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:26:52.677 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:26:52.678 INFO:teuthology.orchestra.run.smithi146.stdout: "ca83eefc-e76d-4996-b47f-8a0e50b89128": { 2024-04-24T08:26:52.678 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:26:52.678 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "ca83eefc-e76d-4996-b47f-8a0e50b89128", 2024-04-24T08:26:52.678 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:26:52.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:26:52.678 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:26:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:26:52.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5b3c0a3da0 msgr2=0x7f5b3c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5b3c0a3da0 0x7f5b3c0a6260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000a580 tx=0x7f5b5000d040 comp rx=0 tx=0).stop 2024-04-24T08:26:52.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 msgr2=0x7f5b58074300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:26:52.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b58074300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5b48006010 tx=0x7f5b48003ba0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 shutdown_connections 2024-04-24T08:26:52.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5b3c0a3da0 0x7f5b3c0a6260 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b581071e0 0x7f5b58074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b58075fa0 0x7f5b58074d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 --2- 172.21.15.146:0/954656297 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b58102110 0x7f5b58074300 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:26:52.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 >> 172.21.15.146:0/954656297 conn(0x7f5b580fd9d0 msgr2=0x7f5b580fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:26:52.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 shutdown_connections 2024-04-24T08:26:52.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:26:52.668+0000 7f5b57e00640 1 -- 172.21.15.146:0/954656297 wait complete. 2024-04-24T08:26:52.686 INFO:tasks.cephfs.filesystem:scrub status for tag:ca83eefc-e76d-4996-b47f-8a0e50b89128 - {'path': '/', 'tag': 'ca83eefc-e76d-4996-b47f-8a0e50b89128', 'options': 'recursive,force'} 2024-04-24T08:26:52.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:52 smithi153 ceph-mon[32746]: pgmap v173: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:52 smithi177 ceph-mon[33046]: pgmap v173: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:53.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:26:53.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:26:53.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:26:53.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:26:53.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:53 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:26:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:26:53.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:26:53.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:26:53.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:53 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:26:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:26:53.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:26:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:26:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:53.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:53 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:26:54.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:54 smithi153 ceph-mon[32746]: pgmap v174: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:54.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:54 smithi146 ceph-mon[27805]: pgmap v174: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:54 smithi177 ceph-mon[33046]: pgmap v174: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:26:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:56 smithi153 ceph-mon[32746]: pgmap v175: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:56 smithi146 ceph-mon[27805]: pgmap v175: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:56 smithi177 ceph-mon[33046]: pgmap v175: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:26:58.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:26:58 smithi153 ceph-mon[32746]: pgmap v176: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:26:58.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:26:58 smithi146 ceph-mon[27805]: pgmap v176: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:26:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:26:58 smithi177 ceph-mon[33046]: pgmap v176: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-04-24T08:27:00.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:00 smithi153 ceph-mon[32746]: pgmap v177: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:27:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:00 smithi146 ceph-mon[27805]: pgmap v177: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:27:00.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:00 smithi177 ceph-mon[33046]: pgmap v177: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:27:02.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:02 smithi153 ceph-mon[32746]: pgmap v178: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:27:02.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:02 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:27:02.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:02 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:27:02.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:02 smithi146 ceph-mon[27805]: pgmap v178: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:27:02.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:02 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:27:02.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:02 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:27:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:02 smithi177 ceph-mon[33046]: pgmap v178: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-24T08:27:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:02 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:27:02.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:02 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:27:04.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:04 smithi153 ceph-mon[32746]: pgmap v179: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:04.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:04 smithi146 ceph-mon[27805]: pgmap v179: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:04 smithi177 ceph-mon[33046]: pgmap v179: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:06 smithi153 ceph-mon[32746]: pgmap v180: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:27:06.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:06 smithi146 ceph-mon[27805]: pgmap v180: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:27:06.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:06 smithi177 ceph-mon[33046]: pgmap v180: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:27:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:08 smithi153 ceph-mon[32746]: pgmap v181: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:08.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:08 smithi146 ceph-mon[27805]: pgmap v181: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:08 smithi177 ceph-mon[33046]: pgmap v181: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:10.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:10 smithi153 ceph-mon[32746]: pgmap v182: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:27:10.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:10 smithi146 ceph-mon[27805]: pgmap v182: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:27:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:10 smithi177 ceph-mon[33046]: pgmap v182: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:27:12.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:12 smithi153 ceph-mon[32746]: pgmap v183: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:27:12.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:12 smithi146 ceph-mon[27805]: pgmap v183: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:27:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:12 smithi177 ceph-mon[33046]: pgmap v183: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:27:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:14 smithi153 ceph-mon[32746]: pgmap v184: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:14 smithi146 ceph-mon[27805]: pgmap v184: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:14 smithi177 ceph-mon[33046]: pgmap v184: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:27:16.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:16 smithi146 ceph-mon[27805]: pgmap v185: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:27:16.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:16 smithi177 ceph-mon[33046]: pgmap v185: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:27:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:16 smithi153 ceph-mon[32746]: pgmap v185: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:27:18.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:18 smithi146 ceph-mon[27805]: pgmap v186: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:27:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:18 smithi177 ceph-mon[33046]: pgmap v186: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:27:18.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:18 smithi153 ceph-mon[32746]: pgmap v186: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-24T08:27:20.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:20 smithi146 ceph-mon[27805]: pgmap v187: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:27:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:20 smithi177 ceph-mon[33046]: pgmap v187: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:27:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:20 smithi153 ceph-mon[32746]: pgmap v187: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:27:22.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:27:22.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:27:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:22 smithi146 ceph-mon[27805]: pgmap v188: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:27:22.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:22 smithi177 ceph-mon[33046]: pgmap v188: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:27:22.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 -- 172.21.15.146:0/3526169682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c074cf0 msgr2=0x7fd52c10e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 --2- 172.21.15.146:0/3526169682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c074cf0 0x7fd52c10e380 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd520009900 tx=0x7fd52002a480 comp rx=0 tx=0).stop 2024-04-24T08:27:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 -- 172.21.15.146:0/3526169682 shutdown_connections 2024-04-24T08:27:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 --2- 172.21.15.146:0/3526169682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd52c10e8c0 0x7fd52c110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 --2- 172.21.15.146:0/3526169682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c074cf0 0x7fd52c10e380 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 --2- 172.21.15.146:0/3526169682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd52c0743d0 0x7fd52c0747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 -- 172.21.15.146:0/3526169682 >> 172.21.15.146:0/3526169682 conn(0x7fd52c0fdb10 msgr2=0x7fd52c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 -- 172.21.15.146:0/3526169682 shutdown_connections 2024-04-24T08:27:22.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.900+0000 7fd533800640 1 -- 172.21.15.146:0/3526169682 wait complete. 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.901+0000 7fd533800640 1 Processor -- start 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.901+0000 7fd533800640 1 -- start start 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd533800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c0743d0 0x7fd52c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd533800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd52c074cf0 0x7fd52c1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd533800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd52c10e8c0 0x7fd52c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd533800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c114620 con 0x7fd52c074cf0 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd533800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1144a0 con 0x7fd52c0743d0 2024-04-24T08:27:22.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd533800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1147a0 con 0x7fd52c10e8c0 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd52be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd52c074cf0 0x7fd52c1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd530e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c0743d0 0x7fd52c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd52be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd52c074cf0 0x7fd52c1a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47062/0 (socket says 172.21.15.146:47062) 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd52be00640 1 -- 172.21.15.146:0/3280035455 learned_addr learned my addr 172.21.15.146:0/3280035455 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.902+0000 7fd531800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd52c10e8c0 0x7fd52c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd531800640 1 -- 172.21.15.146:0/3280035455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c0743d0 msgr2=0x7fd52c1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd531800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c0743d0 0x7fd52c1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd531800640 1 -- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd52c074cf0 msgr2=0x7fd52c1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd531800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd52c074cf0 0x7fd52c1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd531800640 1 -- 172.21.15.146:0/3280035455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c19b180 con 0x7fd52c10e8c0 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd530e00640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c0743d0 0x7fd52c1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd531800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd52c10e8c0 0x7fd52c19aa00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd52c075320 tx=0x7fd52400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd529600640 1 -- 172.21.15.146:0/3280035455 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52400cdf0 con 0x7fd52c10e8c0 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd529600640 1 -- 172.21.15.146:0/3280035455 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd524005020 con 0x7fd52c10e8c0 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd529600640 1 -- 172.21.15.146:0/3280035455 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524010670 con 0x7fd52c10e8c0 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c19b470 con 0x7fd52c10e8c0 2024-04-24T08:27:22.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.903+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c1a7710 con 0x7fd52c10e8c0 2024-04-24T08:27:22.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.906+0000 7fd5365b9740 1 Processor -- start 2024-04-24T08:27:22.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd529600640 1 -- 172.21.15.146:0/3280035455 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fd5240054d0 con 0x7fd52c10e8c0 2024-04-24T08:27:22.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd5365b9740 1 -- start start 2024-04-24T08:27:22.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f21519cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f2151a7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd5365b9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f2151021f0 con 0x55f2151a7850 2024-04-24T08:27:22.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd5365b9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f2150a8350 con 0x55f21519c860 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd5365b9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f2150fe1b0 con 0x55f21519d180 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd530e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f2151a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.907+0000 7fd52be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f21519cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd530e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f2151a7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53086/0 (socket says 172.21.15.146:53086) 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd530e00640 1 -- 172.21.15.146:0/709102268 learned_addr learned my addr 172.21.15.146:0/709102268 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd52be00640 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd52be00640 1 -- 172.21.15.146:0/709102268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 msgr2=0x55f2151a7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd52be00640 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f2151a7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd52be00640 1 -- 172.21.15.146:0/709102268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 msgr2=0x55f2151a9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd52be00640 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd52be00640 1 -- 172.21.15.146:0/709102268 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2150fcf50 con 0x55f21519c860 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd530e00640 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f2151a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd52be00640 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f21519cc40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd520009230 tx=0x7fd520009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b6fe8d59280e8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd529600640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd5140a3d20 0x7fd5140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd51e000640 1 -- 172.21.15.146:0/709102268 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd520035070 con 0x55f21519c860 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd51e000640 1 -- 172.21.15.146:0/709102268 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd520032e40 con 0x55f21519c860 2024-04-24T08:27:22.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.908+0000 7fd51e000640 1 -- 172.21.15.146:0/709102268 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd520004760 con 0x55f21519c860 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd529600640 1 -- 172.21.15.146:0/3280035455 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fd5240d0e10 con 0x7fd52c10e8c0 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd530e00640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd5140a3d20 0x7fd5140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 -- 172.21.15.146:0/709102268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 msgr2=0x55f21519cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f21519cc40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd520009230 tx=0x7fd520009260 comp rx=0 tx=0).stop 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 -- 172.21.15.146:0/709102268 shutdown_connections 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f2151a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 --2- 172.21.15.146:0/709102268 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f21519cc40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 -- 172.21.15.146:0/709102268 >> 172.21.15.146:0/709102268 conn(0x55f215198b80 msgr2=0x55f215198f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:22.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 -- 172.21.15.146:0/709102268 shutdown_connections 2024-04-24T08:27:22.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.909+0000 7fd5365b9740 1 -- 172.21.15.146:0/709102268 wait complete. 2024-04-24T08:27:22.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.910+0000 7fd530e00640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd5140a3d20 0x7fd5140a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55f21519de20 tx=0x7fd518005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:22.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.910+0000 7fd5365b9740 1 -- start start 2024-04-24T08:27:22.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.910+0000 7fd5365b9740 1 Processor -- start 2024-04-24T08:27:22.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519c860 0x55f2151a4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519d180 0x55f2151a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd5365b9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f2151aa580 con 0x55f2151a7850 2024-04-24T08:27:22.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd5365b9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f2150bb690 con 0x55f21519d180 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd5365b9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f2151aa700 con 0x55f21519c860 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd5365b9740 10 client.0 _ll_register_callbacks cb 0x55f2150f7650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd531800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd530e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519c860 0x55f2151a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd52be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519d180 0x55f2151a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd531800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47082/0 (socket says 172.21.15.146:47082) 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd531800640 1 -- 172.21.15.146:0/1988061940 learned_addr learned my addr 172.21.15.146:0/1988061940 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:22.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd530e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519c860 0x55f2151a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53090/0 (socket says 172.21.15.146:53090) 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd52be00640 1 -- 172.21.15.146:0/1988061940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519c860 msgr2=0x55f2151a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd52be00640 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519c860 0x55f2151a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd52be00640 1 -- 172.21.15.146:0/1988061940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 msgr2=0x55f2151a52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd52be00640 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd52be00640 1 -- 172.21.15.146:0/1988061940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2150ba920 con 0x55f21519d180 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.911+0000 7fd531800640 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd52be00640 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519d180 0x55f2151a4da0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd520000c00 tx=0x7fd52002ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b54a608976fabd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:22.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd512a00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd512a00640 1 -- 172.21.15.146:0/1988061940 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52004e070 con 0x55f21519d180 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f21510fc10 con 0x55f21519d180 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd512a00640 1 -- 172.21.15.146:0/1988061940 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fd520004220 con 0x55f21519d180 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd512a00640 1 -- 172.21.15.146:0/1988061940 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd52005a050 con 0x55f21519d180 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd5365b9740 10 client.26579 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.912+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f214ede7d0 con 0x55f21519d180 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd512a00640 1 -- 172.21.15.146:0/1988061940 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f21510fc10 con 0x55f21519d180 2024-04-24T08:27:22.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd512a00640 1 -- 172.21.15.146:0/1988061940 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd52003d450 con 0x55f21519d180 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd5365b9740 10 client.26579 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd5365b9740 10 client.26579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd5365b9740 20 client.26579 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd5365b9740 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2151b7a30 0x55f2151b9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd5365b9740 4 client.26579 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f215109a80 con 0x55f2151b7a30 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.913+0000 7fd530e00640 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2151b7a30 0x55f2151b9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.914+0000 7fd530e00640 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2151b7a30 0x55f2151b9e20 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7703e781669bff8 server_cookie=eb5a06fd999d8153 in_seq=0 out_seq=0 2024-04-24T08:27:22.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.914+0000 7fd512a00640 10 client.26579 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd512a00640 1 -- 172.21.15.146:0/1988061940 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f215109a80 con 0x55f2151b7a30 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd512a00640 10 client.26579 handle_command_reply: tid=0 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 client.26579 shutdown 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519d180 msgr2=0x55f2151a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519d180 0x55f2151a4da0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd520000c00 tx=0x7fd52002ab80 comp rx=0 tx=0).stop 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 shutdown_connections 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f2151b7a30 0x55f2151b9e20 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151a52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519d180 0x55f2151a4da0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 --2- 172.21.15.146:0/1988061940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519c860 0x55f2151a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 >> 172.21.15.146:0/1988061940 conn(0x55f215198b80 msgr2=0x55f21519e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:22.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd512a00640 0 client.26579 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:22.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 shutdown_connections 2024-04-24T08:27:22.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.915+0000 7fd5365b9740 1 -- 172.21.15.146:0/1988061940 wait complete. 2024-04-24T08:27:22.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.916+0000 7fd5365b9740 20 client.26579 trim_cache size 0 max 16384 2024-04-24T08:27:22.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.934+0000 7fd5365b9740 1 Processor -- start 2024-04-24T08:27:22.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.934+0000 7fd5365b9740 1 -- start start 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f215197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151bca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd5365b9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1b2780 con 0x55f2151a7850 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd5365b9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1b0050 con 0x55f21519c860 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd5365b9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1adea0 con 0x55f21519d180 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151a39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54278/0 (socket says 172.21.15.146:54278) 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 -- 172.21.15.146:0/871286249 learned_addr learned my addr 172.21.15.146:0/871286249 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd531800640 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151bca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd52be00640 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f215197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 -- 172.21.15.146:0/871286249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 msgr2=0x55f215197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f215197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 -- 172.21.15.146:0/871286249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 msgr2=0x55f2151bca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151bca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.935+0000 7fd530e00640 1 -- 172.21.15.146:0/871286249 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f214e143f0 con 0x55f21519c860 2024-04-24T08:27:22.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.936+0000 7fd530e00640 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151a39c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd518009c00 tx=0x7fd51801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75677168def14406 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.936+0000 7fd51d600640 1 -- 172.21.15.146:0/871286249 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518031070 con 0x55f21519c860 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.936+0000 7fd51d600640 1 -- 172.21.15.146:0/871286249 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51801d5d0 con 0x55f21519c860 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.936+0000 7fd51d600640 1 -- 172.21.15.146:0/871286249 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5180296a0 con 0x55f21519c860 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 -- 172.21.15.146:0/871286249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 msgr2=0x55f2151a39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151a39c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd518009c00 tx=0x7fd51801b000 comp rx=0 tx=0).stop 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 -- 172.21.15.146:0/871286249 shutdown_connections 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f21519d180 0x55f215197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f2151a7850 0x55f2151bca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 --2- 172.21.15.146:0/871286249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151a39c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 -- 172.21.15.146:0/871286249 >> 172.21.15.146:0/871286249 conn(0x55f215198b80 msgr2=0x55f215197390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 -- 172.21.15.146:0/871286249 shutdown_connections 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 -- 172.21.15.146:0/871286249 wait complete. 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.937+0000 7fd5365b9740 1 -- start start 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 1 Processor -- start 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151bc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f21519d180 0x55f2151bcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2151a7850 0x55f2151b9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1ae510 con 0x55f21519d180 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1af980 con 0x55f21519c860 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c1aebe0 con 0x55f2151a7850 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd5365b9740 10 client.0 _ll_register_callbacks cb 0x7fd520009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd52be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f21519d180 0x55f2151bcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd530e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd531800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2151a7850 0x55f2151b9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd52be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f21519d180 0x55f2151bcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47092/0 (socket says 172.21.15.146:47092) 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd52be00640 1 -- 172.21.15.146:0/738076667 learned_addr learned my addr 172.21.15.146:0/738076667 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.938+0000 7fd530e00640 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151bc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54284/0 (socket says 172.21.15.146:54284) 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd531800640 1 -- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 msgr2=0x55f2151bc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd531800640 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151bc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd531800640 1 -- 172.21.15.146:0/738076667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f21519d180 msgr2=0x55f2151bcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd531800640 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f21519d180 0x55f2151bcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd531800640 1 -- 172.21.15.146:0/738076667 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f214dd40f0 con 0x55f2151a7850 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd52be00640 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f21519d180 0x55f2151bcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:22.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd531800640 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2151a7850 0x55f2151b9bc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd5240a5500 tx=0x7fd52409d600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f17175f6da353b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd512a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd512a00640 1 -- 172.21.15.146:0/738076667 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5240b5070 con 0x55f2151a7850 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd51801d5d0 con 0x55f2151a7850 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd512a00640 1 -- 172.21.15.146:0/738076667 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fd52407bbe0 con 0x55f2151a7850 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd512a00640 1 -- 172.21.15.146:0/738076667 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd5240a5a30 con 0x55f2151a7850 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.939+0000 7fd5365b9740 10 client.26569 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f2150c7810 con 0x55f2151a7850 2024-04-24T08:27:22.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd512a00640 1 -- 172.21.15.146:0/738076667 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51801d5d0 con 0x55f2151a7850 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd512a00640 1 -- 172.21.15.146:0/738076667 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd52409c2a0 con 0x55f2151a7850 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd5365b9740 10 client.26569 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd5365b9740 10 client.26569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd5365b9740 20 client.26569 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd5365b9740 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f21521daf0 0x55f21521ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd5365b9740 4 client.26569 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f215085360 con 0x55f21521daf0 2024-04-24T08:27:22.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.940+0000 7fd530e00640 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f21521daf0 0x55f21521ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd530e00640 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f21521daf0 0x55f21521ff10 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53a0114e152146ec server_cookie=9100335004532567 in_seq=0 out_seq=0 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd512a00640 10 client.26569 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd512a00640 1 -- 172.21.15.146:0/738076667 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f215085360 con 0x55f21521daf0 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd512a00640 10 client.26569 handle_command_reply: tid=0 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 client.26569 shutdown 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2151a7850 msgr2=0x55f2151b9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2151a7850 0x55f2151b9bc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd5240a5500 tx=0x7fd52409d600 comp rx=0 tx=0).stop 2024-04-24T08:27:22.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 shutdown_connections 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f21521daf0 0x55f21521ff10 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2151a7850 0x55f2151b9bc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f21519d180 0x55f2151bcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd512a00640 0 client.26569 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 --2- 172.21.15.146:0/738076667 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f21519c860 0x55f2151bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.941+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 >> 172.21.15.146:0/738076667 conn(0x55f215198b80 msgr2=0x55f2151a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 shutdown_connections 2024-04-24T08:27:22.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd5365b9740 1 -- 172.21.15.146:0/738076667 wait complete. 2024-04-24T08:27:22.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd5365b9740 20 client.26569 trim_cache size 0 max 16384 2024-04-24T08:27:22.952 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:27:22.952 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:27:22.952 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:27:22.952 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:27:22.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd5140a3d20 msgr2=0x7fd5140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd5140a3d20 0x7fd5140a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55f21519de20 tx=0x7fd518005f90 comp rx=0 tx=0).stop 2024-04-24T08:27:22.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd52c10e8c0 msgr2=0x7fd52c19aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:22.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd52c10e8c0 0x7fd52c19aa00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd52c075320 tx=0x7fd52400ef10 comp rx=0 tx=0).stop 2024-04-24T08:27:22.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 shutdown_connections 2024-04-24T08:27:22.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd5140a3d20 0x7fd5140a61e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd52c10e8c0 0x7fd52c19aa00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd52c074cf0 0x7fd52c1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 --2- 172.21.15.146:0/3280035455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd52c0743d0 0x7fd52c1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:22.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.942+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 >> 172.21.15.146:0/3280035455 conn(0x7fd52c0fdb10 msgr2=0x7fd52c0ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:22.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.943+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 shutdown_connections 2024-04-24T08:27:22.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:22.943+0000 7fd533800640 1 -- 172.21.15.146:0/3280035455 wait complete. 2024-04-24T08:27:22.961 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:27:22.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:27:22.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:22 smithi153 ceph-mon[32746]: pgmap v188: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:27:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 -- 172.21.15.146:0/386188470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 msgr2=0x7f083410ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 --2- 172.21.15.146:0/386188470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f083410ecf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f082c009be0 tx=0x7f082c02a6c0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 -- 172.21.15.146:0/386188470 shutdown_connections 2024-04-24T08:27:23.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 --2- 172.21.15.146:0/386188470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f083410ecf0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 --2- 172.21.15.146:0/386188470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0834102a00 0x7f0834106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 --2- 172.21.15.146:0/386188470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0834102050 0x7f0834102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 -- 172.21.15.146:0/386188470 >> 172.21.15.146:0/386188470 conn(0x7f08340fd930 msgr2=0x7f08340ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 -- 172.21.15.146:0/386188470 shutdown_connections 2024-04-24T08:27:23.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.154+0000 7f0838e00640 1 -- 172.21.15.146:0/386188470 wait complete. 2024-04-24T08:27:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 Processor -- start 2024-04-24T08:27:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 -- start start 2024-04-24T08:27:23.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0834102050 0x7f08341a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f08341a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f083419a700 0x7f083419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f08341121c0 con 0x7f0834102050 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0834112040 con 0x7f0834107120 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.155+0000 7f0838e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0834112340 con 0x7f083419a700 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f08341a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f0833400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0834102050 0x7f08341a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f08341a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54300/0 (socket says 172.21.15.146:54300) 2024-04-24T08:27:23.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 -- 172.21.15.146:0/1602674998 learned_addr learned my addr 172.21.15.146:0/1602674998 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f0833e00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f083419a700 0x7f083419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 -- 172.21.15.146:0/1602674998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f083419a700 msgr2=0x7f083419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f083419a700 0x7f083419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 -- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0834102050 msgr2=0x7f08341a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0834102050 0x7f08341a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 -- 172.21.15.146:0/1602674998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083419b300 con 0x7f0834107120 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f082aa00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f08341a0b80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f081c00e9f0 tx=0x7f081c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f0830c00640 1 -- 172.21.15.146:0/1602674998 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c00ccc0 con 0x7f0834107120 2024-04-24T08:27:23.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f0830c00640 1 -- 172.21.15.146:0/1602674998 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f081c010910 con 0x7f0834107120 2024-04-24T08:27:23.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.156+0000 7f0830c00640 1 -- 172.21.15.146:0/1602674998 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c018960 con 0x7f0834107120 2024-04-24T08:27:23.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.157+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08341a73e0 con 0x7f0834107120 2024-04-24T08:27:23.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.157+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08341a7840 con 0x7f0834107120 2024-04-24T08:27:23.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.158+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0834103520 con 0x7f0834107120 2024-04-24T08:27:23.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.160+0000 7f0830c00640 1 -- 172.21.15.146:0/1602674998 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f081c010ac0 con 0x7f0834107120 2024-04-24T08:27:23.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.161+0000 7f0830c00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08180a3da0 0x7f08180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.161+0000 7f0830c00640 1 -- 172.21.15.146:0/1602674998 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f081c0d0f80 con 0x7f0834107120 2024-04-24T08:27:23.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.162+0000 7f0833400640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08180a3da0 0x7f08180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.162+0000 7f0833400640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08180a3da0 0x7f08180a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0824009bd0 tx=0x7f0824005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.164+0000 7f0830c00640 1 -- 172.21.15.146:0/1602674998 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f081c09d5d0 con 0x7f0834107120 2024-04-24T08:27:23.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.408+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0834069190 con 0x7f0834107120 2024-04-24T08:27:23.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.409+0000 7f0830c00640 1 -- 172.21.15.146:0/1602674998 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f0834069190 con 0x7f0834107120 2024-04-24T08:27:23.411 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:27:23.411 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:27:23.411 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:27:23.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08180a3da0 msgr2=0x7f08180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08180a3da0 0x7f08180a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0824009bd0 tx=0x7f0824005f90 comp rx=0 tx=0).stop 2024-04-24T08:27:23.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 msgr2=0x7f08341a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f08341a0b80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f081c00e9f0 tx=0x7f081c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 shutdown_connections 2024-04-24T08:27:23.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f08180a3da0 0x7f08180a6260 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f083419a700 0x7f083419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0834107120 0x7f08341a0b80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 --2- 172.21.15.146:0/1602674998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0834102050 0x7f08341a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 >> 172.21.15.146:0/1602674998 conn(0x7f08340fd930 msgr2=0x7f083410d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.412+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 shutdown_connections 2024-04-24T08:27:23.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.413+0000 7f0838e00640 1 -- 172.21.15.146:0/1602674998 wait complete. 2024-04-24T08:27:23.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:27:23.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 -- 172.21.15.146:0/3257580554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076e50 msgr2=0x7faf98112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 --2- 172.21.15.146:0/3257580554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076e50 0x7faf98112800 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faf84009900 tx=0x7faf8402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 -- 172.21.15.146:0/3257580554 shutdown_connections 2024-04-24T08:27:23.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 --2- 172.21.15.146:0/3257580554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98112d40 0x7faf98115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 --2- 172.21.15.146:0/3257580554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076e50 0x7faf98112800 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 --2- 172.21.15.146:0/3257580554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf98076530 0x7faf98076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 -- 172.21.15.146:0/3257580554 >> 172.21.15.146:0/3257580554 conn(0x7faf980ffb70 msgr2=0x7faf98101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 -- 172.21.15.146:0/3257580554 shutdown_connections 2024-04-24T08:27:23.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.641+0000 7faf9ce00640 1 -- 172.21.15.146:0/3257580554 wait complete. 2024-04-24T08:27:23.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 Processor -- start 2024-04-24T08:27:23.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 -- start start 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076530 0x7faf9819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98076e50 0x7faf9819f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf98112d40 0x7faf981a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faf98118900 con 0x7faf98076530 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf98118780 con 0x7faf98076e50 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.642+0000 7faf9ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf98118a80 con 0x7faf98112d40 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf97400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076530 0x7faf9819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf96a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98076e50 0x7faf9819f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf97400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076530 0x7faf9819ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47120/0 (socket says 172.21.15.146:47120) 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf96a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98076e50 0x7faf9819f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54320/0 (socket says 172.21.15.146:54320) 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf97400640 1 -- 172.21.15.146:0/1080120563 learned_addr learned my addr 172.21.15.146:0/1080120563 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf97e00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf98112d40 0x7faf981a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf96a00640 1 -- 172.21.15.146:0/1080120563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf98112d40 msgr2=0x7faf981a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf96a00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf98112d40 0x7faf981a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf96a00640 1 -- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076530 msgr2=0x7faf9819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf96a00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076530 0x7faf9819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.643+0000 7faf96a00640 1 -- 172.21.15.146:0/1080120563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf981a3df0 con 0x7faf98076e50 2024-04-24T08:27:23.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.644+0000 7faf96a00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98076e50 0x7faf9819f340 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faf8402a910 tx=0x7faf84003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.644+0000 7faf8fe00640 1 -- 172.21.15.146:0/1080120563 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8403f070 con 0x7faf98076e50 2024-04-24T08:27:23.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.644+0000 7faf8fe00640 1 -- 172.21.15.146:0/1080120563 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf84031c20 con 0x7faf98076e50 2024-04-24T08:27:23.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.644+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf981a4080 con 0x7faf98076e50 2024-04-24T08:27:23.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.644+0000 7faf8fe00640 1 -- 172.21.15.146:0/1080120563 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84035030 con 0x7faf98076e50 2024-04-24T08:27:23.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.644+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf981ab970 con 0x7faf98076e50 2024-04-24T08:27:23.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.647+0000 7fafa1383740 1 Processor -- start 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.647+0000 7faf8fe00640 1 -- 172.21.15.146:0/1080120563 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7faf84042070 con 0x7faf98076e50 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7fafa1383740 1 -- start start 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7fafa1383740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433edb40 0x562e433edf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7fafa1383740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7fafa1383740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433faf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7fafa1383740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x562e433531d0 con 0x562e433f8b30 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7fafa1383740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562e432f9450 con 0x562e433edb40 2024-04-24T08:27:23.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7fafa1383740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562e4334f190 con 0x562e433ee460 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7faf97e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433edb40 0x562e433edf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7faf97400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7faf97e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433edb40 0x562e433edf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54324/0 (socket says 172.21.15.146:54324) 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7faf97e00640 1 -- 172.21.15.146:0/1360039157 learned_addr learned my addr 172.21.15.146:0/1360039157 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.648+0000 7faf96a00640 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433faf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf97e00640 1 -- 172.21.15.146:0/1360039157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 msgr2=0x562e433f85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf97e00640 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433f85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf97e00640 1 -- 172.21.15.146:0/1360039157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 msgr2=0x562e433faf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf97e00640 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433faf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf97e00640 1 -- 172.21.15.146:0/1360039157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e433b6310 con 0x562e433edb40 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf97e00640 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433edb40 0x562e433edf20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7faf90009f30 tx=0x7faf900118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb46acda4ce59a8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf8cc00640 1 -- 172.21.15.146:0/1360039157 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90004d20 con 0x562e433edb40 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf8cc00640 1 -- 172.21.15.146:0/1360039157 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf90004050 con 0x562e433edb40 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf8cc00640 1 -- 172.21.15.146:0/1360039157 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90005260 con 0x562e433edb40 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf8fe00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faf7c0a3d20 0x7faf7c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf8fe00640 1 -- 172.21.15.146:0/1080120563 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faf840f0d60 con 0x7faf98076e50 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.649+0000 7faf97400640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faf7c0a3d20 0x7faf7c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 -- 172.21.15.146:0/1360039157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433edb40 msgr2=0x562e433edf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433edb40 0x562e433edf20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7faf90009f30 tx=0x7faf900118d0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 -- 172.21.15.146:0/1360039157 shutdown_connections 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433faf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 --2- 172.21.15.146:0/1360039157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433edb40 0x562e433edf20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 -- 172.21.15.146:0/1360039157 >> 172.21.15.146:0/1360039157 conn(0x562e433e9e60 msgr2=0x562e433ea240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 -- 172.21.15.146:0/1360039157 shutdown_connections 2024-04-24T08:27:23.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7fafa1383740 1 -- 172.21.15.146:0/1360039157 wait complete. 2024-04-24T08:27:23.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.650+0000 7faf97400640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faf7c0a3d20 0x7faf7c0a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faf88009640 tx=0x7faf88005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 -- start start 2024-04-24T08:27:23.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 Processor -- start 2024-04-24T08:27:23.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433ee460 0x562e433f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433f6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x562e433fb9c0 con 0x562e433ee460 2024-04-24T08:27:23.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x562e4330c810 con 0x562e433f8b30 2024-04-24T08:27:23.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562e433fbb40 con 0x562e433edb40 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.651+0000 7fafa1383740 10 client.0 _ll_register_callbacks cb 0x562e432da140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433ee460 0x562e433f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433ee460 0x562e433f58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47134/0 (socket says 172.21.15.146:47134) 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 -- 172.21.15.146:0/1260198977 learned_addr learned my addr 172.21.15.146:0/1260198977 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf97e00640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf97400640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 -- 172.21.15.146:0/1260198977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 msgr2=0x562e433f85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 -- 172.21.15.146:0/1260198977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 msgr2=0x562e433f6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433f6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 -- 172.21.15.146:0/1260198977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e4331d3f0 con 0x562e433ee460 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf97e00640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433f6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf96a00640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433ee460 0x562e433f58d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faf840bccf0 tx=0x7faf8403b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e0907888dac5be server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.652+0000 7faf81600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7faf81600640 1 -- 172.21.15.146:0/1260198977 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84049070 con 0x562e433ee460 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7faf81600640 1 -- 172.21.15.146:0/1260198977 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faf840b93d0 con 0x562e433ee460 2024-04-24T08:27:23.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf90004050 con 0x562e433ee460 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7faf81600640 1 -- 172.21.15.146:0/1260198977 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf840bcd60 con 0x562e433ee460 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 10 client.16458 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562e430653f0 con 0x562e433ee460 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7faf81600640 1 -- 172.21.15.146:0/1260198977 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf90004050 con 0x562e433ee460 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7faf81600640 1 -- 172.21.15.146:0/1260198977 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7faf840dd0b0 con 0x562e433ee460 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 10 client.16458 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 10 client.16458 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:27:23.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 20 client.16458 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e43408ce0 0x562e4340b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 4 client.16458 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.653+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e43360bf0 con 0x562e43408ce0 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.654+0000 7faf97400640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e43408ce0 0x562e4340b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.654+0000 7faf97400640 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e43408ce0 0x562e4340b150 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ed230ad97ac94a server_cookie=5d52b11eca8b7649 in_seq=0 out_seq=0 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.654+0000 7faf81600640 10 client.16458 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7faf81600640 1 -- 172.21.15.146:0/1260198977 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e43360bf0 con 0x562e43408ce0 2024-04-24T08:27:23.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7faf81600640 10 client.16458 handle_command_reply: tid=0 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 client.16458 shutdown 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433ee460 msgr2=0x562e433f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433ee460 0x562e433f58d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faf840bccf0 tx=0x7faf8403b720 comp rx=0 tx=0).stop 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 shutdown_connections 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e43408ce0 0x562e4340b150 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433f6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433ee460 0x562e433f58d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 --2- 172.21.15.146:0/1260198977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7faf81600640 0 client.16458 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:23.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.655+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 >> 172.21.15.146:0/1260198977 conn(0x562e433e9e60 msgr2=0x562e433efc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.656+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 shutdown_connections 2024-04-24T08:27:23.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.656+0000 7fafa1383740 1 -- 172.21.15.146:0/1260198977 wait complete. 2024-04-24T08:27:23.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.656+0000 7fafa1383740 20 client.16458 trim_cache size 0 max 16384 2024-04-24T08:27:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.670+0000 7fafa1383740 1 Processor -- start 2024-04-24T08:27:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7fafa1383740 1 -- start start 2024-04-24T08:27:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7fafa1383740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433edb40 0x562e433f8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7fafa1383740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7fafa1383740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433e9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7fafa1383740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faf981b69b0 con 0x562e433edb40 2024-04-24T08:27:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7fafa1383740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf981b4280 con 0x562e433f8b30 2024-04-24T08:27:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7fafa1383740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf981b20d0 con 0x562e433ee460 2024-04-24T08:27:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7faf97400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433edb40 0x562e433f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7faf97400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433edb40 0x562e433f8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47144/0 (socket says 172.21.15.146:47144) 2024-04-24T08:27:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7faf97400640 1 -- 172.21.15.146:0/2331564186 learned_addr learned my addr 172.21.15.146:0/2331564186 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7faf97e00640 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.671+0000 7faf96a00640 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf97400640 1 -- 172.21.15.146:0/2331564186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 msgr2=0x562e433e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf97400640 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433e8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf97400640 1 -- 172.21.15.146:0/2331564186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 msgr2=0x562e433e9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf97400640 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433e9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf97400640 1 -- 172.21.15.146:0/2331564186 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e4312f7d0 con 0x562e433edb40 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf97400640 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433edb40 0x562e433f8110 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x562e4340ba00 tx=0x7faf8801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450ee03e130dd95b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf83e00640 1 -- 172.21.15.146:0/2331564186 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8802b4f0 con 0x562e433edb40 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf83e00640 1 -- 172.21.15.146:0/2331564186 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf88005500 con 0x562e433edb40 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.672+0000 7faf83e00640 1 -- 172.21.15.146:0/2331564186 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88004ce0 con 0x562e433edb40 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 -- 172.21.15.146:0/2331564186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433edb40 msgr2=0x562e433f8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433edb40 0x562e433f8110 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x562e4340ba00 tx=0x7faf8801b000 comp rx=0 tx=0).stop 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 -- 172.21.15.146:0/2331564186 shutdown_connections 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433ee460 0x562e433e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433f8b30 0x562e433e9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 --2- 172.21.15.146:0/2331564186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433edb40 0x562e433f8110 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 -- 172.21.15.146:0/2331564186 >> 172.21.15.146:0/2331564186 conn(0x562e433e9e60 msgr2=0x562e433e86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 -- 172.21.15.146:0/2331564186 shutdown_connections 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 -- 172.21.15.146:0/2331564186 wait complete. 2024-04-24T08:27:23.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.673+0000 7fafa1383740 1 -- start start 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 1 Processor -- start 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433ee460 0x562e433fa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433ff350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7faf981b2740 con 0x562e433f8b30 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faf981b3bb0 con 0x562e433ee460 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf981b2e10 con 0x562e433edb40 2024-04-24T08:27:23.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7fafa1383740 10 client.0 _ll_register_callbacks cb 0x7faf84009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7faf97e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433ff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.674+0000 7faf97400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433ff350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47156/0 (socket says 172.21.15.146:47156) 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97e00640 1 -- 172.21.15.146:0/1008715460 learned_addr learned my addr 172.21.15.146:0/1008715460 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf96a00640 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433ee460 0x562e433fa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97400640 1 -- 172.21.15.146:0/1008715460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433ee460 msgr2=0x562e433fa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97400640 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433ee460 0x562e433fa220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97400640 1 -- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 msgr2=0x562e433ff350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97400640 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433ff350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97400640 1 -- 172.21.15.146:0/1008715460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e433ff890 con 0x562e433edb40 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97e00640 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf97400640 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f9ce0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faf880097e0 tx=0x7faf880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f34a5c8960e392d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.675+0000 7faf81600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7faf81600640 1 -- 172.21.15.146:0/1008715460 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8802b4f0 con 0x562e433edb40 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf88005500 con 0x562e433edb40 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7faf81600640 1 -- 172.21.15.146:0/1008715460 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7faf8802b690 con 0x562e433edb40 2024-04-24T08:27:23.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7faf81600640 1 -- 172.21.15.146:0/1008715460 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf88043050 con 0x562e433edb40 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7fafa1383740 10 client.26593 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562e4346f990 con 0x562e433edb40 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7faf81600640 1 -- 172.21.15.146:0/1008715460 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf88005500 con 0x562e433edb40 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7faf81600640 1 -- 172.21.15.146:0/1008715460 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7faf88029440 con 0x562e433edb40 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7fafa1383740 10 client.26593 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7fafa1383740 10 client.26593 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.676+0000 7fafa1383740 20 client.26593 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:23.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.677+0000 7fafa1383740 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e434700b0 0x562e434724a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.677+0000 7fafa1383740 4 client.26593 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:27:23.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.677+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e4335aa60 con 0x562e434700b0 2024-04-24T08:27:23.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.677+0000 7faf96a00640 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e434700b0 0x562e434724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.677+0000 7faf96a00640 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e434700b0 0x562e434724a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d37c610c47a0dc89 server_cookie=2f35400f652d26b4 in_seq=0 out_seq=0 2024-04-24T08:27:23.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.677+0000 7faf81600640 10 client.26593 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:23.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7faf81600640 1 -- 172.21.15.146:0/1008715460 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e4335aa60 con 0x562e434700b0 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7faf81600640 10 client.26593 handle_command_reply: tid=0 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 client.26593 shutdown 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 msgr2=0x562e433f9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f9ce0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faf880097e0 tx=0x7faf880046c0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 shutdown_connections 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x562e434700b0 0x562e434724a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x562e433f8b30 0x562e433ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x562e433ee460 0x562e433fa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 --2- 172.21.15.146:0/1008715460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e433edb40 0x562e433f9ce0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7faf81600640 0 client.26593 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:23.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 >> 172.21.15.146:0/1008715460 conn(0x562e433e9e60 msgr2=0x562e433efb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 shutdown_connections 2024-04-24T08:27:23.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 1 -- 172.21.15.146:0/1008715460 wait complete. 2024-04-24T08:27:23.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.678+0000 7fafa1383740 20 client.26593 trim_cache size 0 max 16384 2024-04-24T08:27:23.688 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:27:23.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faf7c0a3d20 msgr2=0x7faf7c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faf7c0a3d20 0x7faf7c0a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7faf88009640 tx=0x7faf88005f90 comp rx=0 tx=0).stop 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98076e50 msgr2=0x7faf9819f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98076e50 0x7faf9819f340 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faf8402a910 tx=0x7faf84003b20 comp rx=0 tx=0).stop 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 shutdown_connections 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7faf7c0a3d20 0x7faf7c0a61e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf98112d40 0x7faf981a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faf98076e50 0x7faf9819f340 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 --2- 172.21.15.146:0/1080120563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf98076530 0x7faf9819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 >> 172.21.15.146:0/1080120563 conn(0x7faf980ffb70 msgr2=0x7faf981139b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 shutdown_connections 2024-04-24T08:27:23.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.679+0000 7faf9ce00640 1 -- 172.21.15.146:0/1080120563 wait complete. 2024-04-24T08:27:23.694 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:23 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1602674998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:27:23.697 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:27:23.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:23 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1602674998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:27:23.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 -- 172.21.15.146:0/3254573026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 msgr2=0x7f9874108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 --2- 172.21.15.146:0/3254573026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f9874108a00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f985c009930 tx=0x7f985c02a420 comp rx=0 tx=0).stop 2024-04-24T08:27:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 -- 172.21.15.146:0/3254573026 shutdown_connections 2024-04-24T08:27:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 --2- 172.21.15.146:0/3254573026 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9874108f40 0x7f98741153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 --2- 172.21.15.146:0/3254573026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f9874108a00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 --2- 172.21.15.146:0/3254573026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f987410d500 0x7f987410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 -- 172.21.15.146:0/3254573026 >> 172.21.15.146:0/3254573026 conn(0x7f9874077e20 msgr2=0x7f9874106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.892+0000 7f9873200640 1 -- 172.21.15.146:0/3254573026 shutdown_connections 2024-04-24T08:27:23.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.893+0000 7f9873200640 1 -- 172.21.15.146:0/3254573026 wait complete. 2024-04-24T08:27:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.893+0000 7f9873200640 1 Processor -- start 2024-04-24T08:27:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9873200640 1 -- start start 2024-04-24T08:27:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9873200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f98741a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9873200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9874108f40 0x7f98741a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9873200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f987410d500 0x7f987419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9873200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9874118900 con 0x7f9874108580 2024-04-24T08:27:23.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9873200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9874118780 con 0x7f9874108f40 2024-04-24T08:27:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9873200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9874118a80 con 0x7f987410d500 2024-04-24T08:27:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9871e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f98741a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9871e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f98741a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47178/0 (socket says 172.21.15.146:47178) 2024-04-24T08:27:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9871e00640 1 -- 172.21.15.146:0/4052548703 learned_addr learned my addr 172.21.15.146:0/4052548703 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9872800640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f987410d500 0x7f987419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.894+0000 7f9869400640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9874108f40 0x7f98741a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f9871e00640 1 -- 172.21.15.146:0/4052548703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f987410d500 msgr2=0x7f987419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f9871e00640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f987410d500 0x7f987419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f9871e00640 1 -- 172.21.15.146:0/4052548703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9874108f40 msgr2=0x7f98741a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f9871e00640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9874108f40 0x7f98741a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f9871e00640 1 -- 172.21.15.146:0/4052548703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987419f440 con 0x7f9874108580 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f9871e00640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f98741a4a70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f986400e9c0 tx=0x7f986400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f986b400640 1 -- 172.21.15.146:0/4052548703 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986400cd80 con 0x7f9874108580 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987419f730 con 0x7f9874108580 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f986b400640 1 -- 172.21.15.146:0/4052548703 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98640035f0 con 0x7f9874108580 2024-04-24T08:27:23.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.895+0000 7f986b400640 1 -- 172.21.15.146:0/4052548703 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864010690 con 0x7f9874108580 2024-04-24T08:27:23.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.896+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98741ab9f0 con 0x7f9874108580 2024-04-24T08:27:23.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 Processor -- start 2024-04-24T08:27:23.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 -- start start 2024-04-24T08:27:23.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f986b400640 1 -- 172.21.15.146:0/4052548703 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f98640037a0 con 0x7f9874108580 2024-04-24T08:27:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b5b40 0x55fad72b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b6460 0x55fad72c05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72c0b30 0x55fad72c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fad721b1d0 con 0x55fad72c0b30 2024-04-24T08:27:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fad71c1450 con 0x55fad72b5b40 2024-04-24T08:27:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.899+0000 7f987b86c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fad7217190 con 0x55fad72b6460 2024-04-24T08:27:23.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9869400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72c0b30 0x55fad72c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b5b40 0x55fad72b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9871e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b6460 0x55fad72c05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9869400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72c0b30 0x55fad72c2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47188/0 (socket says 172.21.15.146:47188) 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b5b40 0x55fad72b5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54380/0 (socket says 172.21.15.146:54380) 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9869400640 1 -- 172.21.15.146:0/2109828318 learned_addr learned my addr 172.21.15.146:0/2109828318 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 -- 172.21.15.146:0/2109828318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b6460 msgr2=0x55fad72c05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b6460 0x55fad72c05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 -- 172.21.15.146:0/2109828318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72c0b30 msgr2=0x55fad72c2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72c0b30 0x55fad72c2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 -- 172.21.15.146:0/2109828318 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fad727e310 con 0x55fad72b5b40 2024-04-24T08:27:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9869400640 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72c0b30 0x55fad72c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.900+0000 7f9872800640 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b5b40 0x55fad72b5f20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f986c009ef0 tx=0x7f986c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=362d00653036c8b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.901+0000 7f986b400640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98580a3d20 0x7f98580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.901+0000 7f9863000640 1 -- 172.21.15.146:0/2109828318 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c004d20 con 0x55fad72b5b40 2024-04-24T08:27:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.901+0000 7f986b400640 1 -- 172.21.15.146:0/4052548703 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f98640d0c10 con 0x7f9874108580 2024-04-24T08:27:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.901+0000 7f9871e00640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98580a3d20 0x7f98580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.901+0000 7f9863000640 1 -- 172.21.15.146:0/2109828318 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986c004030 con 0x55fad72b5b40 2024-04-24T08:27:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.901+0000 7f9863000640 1 -- 172.21.15.146:0/2109828318 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c005280 con 0x55fad72b5b40 2024-04-24T08:27:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 -- 172.21.15.146:0/2109828318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b5b40 msgr2=0x55fad72b5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f9871e00640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98580a3d20 0x7f98580a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55fad72b7100 tx=0x7f98640d46c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b5b40 0x55fad72b5f20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f986c009ef0 tx=0x7f986c011810 comp rx=0 tx=0).stop 2024-04-24T08:27:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 -- 172.21.15.146:0/2109828318 shutdown_connections 2024-04-24T08:27:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b6460 0x55fad72c05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72c0b30 0x55fad72c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 --2- 172.21.15.146:0/2109828318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b5b40 0x55fad72b5f20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 -- 172.21.15.146:0/2109828318 >> 172.21.15.146:0/2109828318 conn(0x55fad72b1e60 msgr2=0x55fad72b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 -- 172.21.15.146:0/2109828318 shutdown_connections 2024-04-24T08:27:23.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 -- 172.21.15.146:0/2109828318 wait complete. 2024-04-24T08:27:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 -- start start 2024-04-24T08:27:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.902+0000 7f987b86c740 1 Processor -- start 2024-04-24T08:27:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f987b86c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b5b40 0x55fad72c0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f987b86c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b6460 0x55fad72bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f987b86c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72c0b30 0x55fad72bdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f987b86c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fad72c39c0 con 0x55fad72b5b40 2024-04-24T08:27:23.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f987b86c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fad71d4810 con 0x55fad72b6460 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f987b86c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fad72c3b40 con 0x55fad72c0b30 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f987b86c740 10 client.0 _ll_register_callbacks cb 0x55fad71a2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f9872800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b5b40 0x55fad72c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f9872800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b5b40 0x55fad72c0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47196/0 (socket says 172.21.15.146:47196) 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f9872800640 1 -- 172.21.15.146:0/950351279 learned_addr learned my addr 172.21.15.146:0/950351279 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.903+0000 7f9869400640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b6460 0x55fad72bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9872800640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72c0b30 0x55fad72bdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9872800640 1 -- 172.21.15.146:0/950351279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72c0b30 msgr2=0x55fad72bdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9872800640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72c0b30 0x55fad72bdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9872800640 1 -- 172.21.15.146:0/950351279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b6460 msgr2=0x55fad72bd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9872800640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b6460 0x55fad72bd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9872800640 1 -- 172.21.15.146:0/950351279 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fad71e53f0 con 0x55fad72b5b40 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9869400640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b6460 0x55fad72bd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9872800640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b5b40 0x55fad72c0580 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f986c005c40 tx=0x7f986c003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4521ad33375e7740 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9857400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:23.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9857400640 1 -- 172.21.15.146:0/950351279 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c02d060 con 0x55fad72b5b40 2024-04-24T08:27:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.904+0000 7f9857400640 1 -- 172.21.15.146:0/950351279 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f986c004c50 con 0x55fad72b5b40 2024-04-24T08:27:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986c004030 con 0x55fad72b5b40 2024-04-24T08:27:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f9857400640 1 -- 172.21.15.146:0/950351279 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f986c004e50 con 0x55fad72b5b40 2024-04-24T08:27:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 10 client.16476 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fad6f2d3f0 con 0x55fad72b5b40 2024-04-24T08:27:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f9857400640 1 -- 172.21.15.146:0/950351279 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986c004030 con 0x55fad72b5b40 2024-04-24T08:27:23.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f9857400640 1 -- 172.21.15.146:0/950351279 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f986c024070 con 0x55fad72b5b40 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 10 client.16476 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 10 client.16476 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 20 client.16476 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad72d0cd0 0x55fad72d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 4 client.16476 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.905+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fad7228bf0 con 0x55fad72d0cd0 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.906+0000 7f9869400640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad72d0cd0 0x55fad72d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.906+0000 7f9869400640 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad72d0cd0 0x55fad72d30c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d3fe40036243b574 server_cookie=9fe9f18470390d60 in_seq=0 out_seq=0 2024-04-24T08:27:23.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.906+0000 7f9857400640 10 client.16476 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f9857400640 1 -- 172.21.15.146:0/950351279 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fad7228bf0 con 0x55fad72d0cd0 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f9857400640 10 client.16476 handle_command_reply: tid=0 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 client.16476 shutdown 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b5b40 msgr2=0x55fad72c0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b5b40 0x55fad72c0580 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f986c005c40 tx=0x7f986c003f80 comp rx=0 tx=0).stop 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 shutdown_connections 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad72d0cd0 0x55fad72d30c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72c0b30 0x55fad72bdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72b6460 0x55fad72bd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 --2- 172.21.15.146:0/950351279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b5b40 0x55fad72c0580 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 >> 172.21.15.146:0/950351279 conn(0x55fad72b1e60 msgr2=0x55fad72b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.907+0000 7f9857400640 0 client.16476 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:23.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.908+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 shutdown_connections 2024-04-24T08:27:23.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.908+0000 7f987b86c740 1 -- 172.21.15.146:0/950351279 wait complete. 2024-04-24T08:27:23.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.908+0000 7f987b86c740 20 client.16476 trim_cache size 0 max 16384 2024-04-24T08:27:23.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.922+0000 7f987b86c740 1 Processor -- start 2024-04-24T08:27:23.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.922+0000 7f987b86c740 1 -- start start 2024-04-24T08:27:23.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f987b86c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f987b86c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f987b86c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f987b86c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b6980 con 0x55fad72b6460 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f987b86c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b4250 con 0x55fad72c0b30 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f987b86c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b20a0 con 0x55fad72b5b40 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9869400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53240/0 (socket says 172.21.15.146:53240) 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 -- 172.21.15.146:0/1313511804 learned_addr learned my addr 172.21.15.146:0/1313511804 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9869400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47200/0 (socket says 172.21.15.146:47200) 2024-04-24T08:27:23.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 -- 172.21.15.146:0/1313511804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 msgr2=0x55fad72c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72c00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 -- 172.21.15.146:0/1313511804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 msgr2=0x55fad72bc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.923+0000 7f9872800640 1 -- 172.21.15.146:0/1313511804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fad6ff77d0 con 0x55fad72b5b40 2024-04-24T08:27:23.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.924+0000 7f9872800640 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72cc8b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f986c009ef0 tx=0x7f986c005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ece767f2e5192a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.924+0000 7f9862600640 1 -- 172.21.15.146:0/1313511804 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c003ff0 con 0x55fad72b5b40 2024-04-24T08:27:23.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.924+0000 7f9862600640 1 -- 172.21.15.146:0/1313511804 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986c00fd70 con 0x55fad72b5b40 2024-04-24T08:27:23.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.924+0000 7f9862600640 1 -- 172.21.15.146:0/1313511804 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c0289e0 con 0x55fad72b5b40 2024-04-24T08:27:23.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 -- 172.21.15.146:0/1313511804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 msgr2=0x55fad72cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72cc8b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f986c009ef0 tx=0x7f986c005bd0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 -- 172.21.15.146:0/1313511804 shutdown_connections 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72c00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 --2- 172.21.15.146:0/1313511804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72cc8b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 -- 172.21.15.146:0/1313511804 >> 172.21.15.146:0/1313511804 conn(0x55fad72b1e60 msgr2=0x55fad72bc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 -- 172.21.15.146:0/1313511804 shutdown_connections 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 -- 172.21.15.146:0/1313511804 wait complete. 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 -- start start 2024-04-24T08:27:23.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.925+0000 7f987b86c740 1 Processor -- start 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f987b86c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72c9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f987b86c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f987b86c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72bb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f987b86c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b2710 con 0x55fad72b6460 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f987b86c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b3b80 con 0x55fad72c0b30 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f987b86c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b2de0 con 0x55fad72b5b40 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f987b86c740 10 client.0 _ll_register_callbacks cb 0x7f9864005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:23.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9869400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72c9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53248/0 (socket says 172.21.15.146:53248) 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 -- 172.21.15.146:0/1195591756 learned_addr learned my addr 172.21.15.146:0/1195591756 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72bb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 -- 172.21.15.146:0/1195591756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 msgr2=0x55fad72bb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72bb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 -- 172.21.15.146:0/1195591756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 msgr2=0x55fad72bab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.926+0000 7f9872800640 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f9872800640 1 -- 172.21.15.146:0/1195591756 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fad72bb740 con 0x55fad72b5b40 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f9872800640 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72c9090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f986c002230 tx=0x7f986c01b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b50525a275c1cf58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f9857400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f9857400640 1 -- 172.21.15.146:0/1195591756 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c009480 con 0x55fad72b5b40 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986c00fd70 con 0x55fad72b5b40 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f9857400640 1 -- 172.21.15.146:0/1195591756 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f986c03d430 con 0x55fad72b5b40 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f9857400640 1 -- 172.21.15.146:0/1195591756 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f986c028760 con 0x55fad72b5b40 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f987b86c740 10 client.26611 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:23.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fad7337050 con 0x55fad72b5b40 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.927+0000 7f9857400640 1 -- 172.21.15.146:0/1195591756 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986c00fd70 con 0x55fad72b5b40 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f9857400640 1 -- 172.21.15.146:0/1195591756 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f986c024070 con 0x55fad72b5b40 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f987b86c740 10 client.26611 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f987b86c740 10 client.26611 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f987b86c740 20 client.26611 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f987b86c740 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad73375b0 0x55fad73399a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f987b86c740 4 client.26611 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:27:23.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fad7222a60 con 0x55fad73375b0 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.928+0000 7f9869400640 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad73375b0 0x55fad73399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f9869400640 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad73375b0 0x55fad73399a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=87cad01c0920d2ed server_cookie=1ca209d6b1e03c2c in_seq=0 out_seq=0 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f9857400640 10 client.26611 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f9857400640 1 -- 172.21.15.146:0/1195591756 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fad7222a60 con 0x55fad73375b0 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f9857400640 10 client.26611 handle_command_reply: tid=0 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 client.26611 shutdown 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 msgr2=0x55fad72c9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72c9090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f986c002230 tx=0x7f986c01b040 comp rx=0 tx=0).stop 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 shutdown_connections 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55fad73375b0 0x55fad73399a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fad72c0b30 0x55fad72bb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fad72b6460 0x55fad72bab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f9857400640 0 client.26611 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 --2- 172.21.15.146:0/1195591756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fad72b5b40 0x55fad72c9090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 >> 172.21.15.146:0/1195591756 conn(0x55fad72b1e60 msgr2=0x55fad72bc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 shutdown_connections 2024-04-24T08:27:23.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 1 -- 172.21.15.146:0/1195591756 wait complete. 2024-04-24T08:27:23.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.929+0000 7f987b86c740 20 client.26611 trim_cache size 0 max 16384 2024-04-24T08:27:23.939 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:27:23.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98580a3d20 msgr2=0x7f98580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98580a3d20 0x7f98580a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55fad72b7100 tx=0x7f98640d46c0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 msgr2=0x7f98741a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:23.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f98741a4a70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f986400e9c0 tx=0x7f986400ee80 comp rx=0 tx=0).stop 2024-04-24T08:27:23.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 shutdown_connections 2024-04-24T08:27:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98580a3d20 0x7f98580a61e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f987410d500 0x7f987419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9874108f40 0x7f98741a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 --2- 172.21.15.146:0/4052548703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9874108580 0x7f98741a4a70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 >> 172.21.15.146:0/4052548703 conn(0x7f9874077e20 msgr2=0x7f98740783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 shutdown_connections 2024-04-24T08:27:23.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:23.930+0000 7f9873200640 1 -- 172.21.15.146:0/4052548703 wait complete. 2024-04-24T08:27:23.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:27:23.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:23 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1602674998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:27:24.139 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 -- 172.21.15.146:0/2057356308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d141039e0 msgr2=0x7f9d1410bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2057356308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d141039e0 0x7f9d1410bec0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009860 tx=0x7f9cfc02a3d0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 -- 172.21.15.146:0/2057356308 shutdown_connections 2024-04-24T08:27:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2057356308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d1410c4e0 0x7f9d1410e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2057356308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d141039e0 0x7f9d1410bec0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.140 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2057356308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 0x7f9d141034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 -- 172.21.15.146:0/2057356308 >> 172.21.15.146:0/2057356308 conn(0x7f9d140fc920 msgr2=0x7f9d140fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.138+0000 7f9d13e00640 1 -- 172.21.15.146:0/2057356308 shutdown_connections 2024-04-24T08:27:24.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.139+0000 7f9d13e00640 1 -- 172.21.15.146:0/2057356308 wait complete. 2024-04-24T08:27:24.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 Processor -- start 2024-04-24T08:27:24.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 -- start start 2024-04-24T08:27:24.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 0x7f9d1419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d141039e0 0x7f9d1419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1410c4e0 0x7f9d14198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d141122f0 con 0x7f9d141039e0 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d14112170 con 0x7f9d1410c4e0 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.140+0000 7f9d13e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9d14112470 con 0x7f9d141030c0 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d0a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d141039e0 0x7f9d1419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 0x7f9d1419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 0x7f9d1419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53272/0 (socket says 172.21.15.146:53272) 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1410c4e0 0x7f9d14198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 -- 172.21.15.146:0/2120335666 learned_addr learned my addr 172.21.15.146:0/2120335666 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:24.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 -- 172.21.15.146:0/2120335666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1410c4e0 msgr2=0x7f9d14198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1410c4e0 0x7f9d14198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 -- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d141039e0 msgr2=0x7f9d1419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d141039e0 0x7f9d1419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.141+0000 7f9d12a00640 1 -- 172.21.15.146:0/2120335666 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d14198df0 con 0x7f9d141030c0 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.142+0000 7f9d12a00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 0x7f9d1419e470 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400e9c0 tx=0x7f9d0400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.142+0000 7f9d18a00640 1 -- 172.21.15.146:0/2120335666 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0400cde0 con 0x7f9d141030c0 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.142+0000 7f9d18a00640 1 -- 172.21.15.146:0/2120335666 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d040035f0 con 0x7f9d141030c0 2024-04-24T08:27:24.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.142+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d141990e0 con 0x7f9d141030c0 2024-04-24T08:27:24.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.142+0000 7f9d18a00640 1 -- 172.21.15.146:0/2120335666 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04010690 con 0x7f9d141030c0 2024-04-24T08:27:24.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.142+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d141a5330 con 0x7f9d141030c0 2024-04-24T08:27:24.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.145+0000 7f9d1cc7f740 1 Processor -- start 2024-04-24T08:27:24.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.145+0000 7f9d18a00640 1 -- 172.21.15.146:0/2120335666 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f9d040037a0 con 0x7f9d141030c0 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.145+0000 7f9d1cc7f740 1 -- start start 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc0df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc185f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc1af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fcdfb731d0 con 0x55fcdfc18b30 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcdfb19450 con 0x55fcdfc0e460 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fcdfb6f190 con 0x55fcdfc0db40 2024-04-24T08:27:24.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc0df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc185f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d0a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc1af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc0df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53286/0 (socket says 172.21.15.146:53286) 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.146+0000 7f9d13400640 1 -- 172.21.15.146:0/677523754 learned_addr learned my addr 172.21.15.146:0/677523754 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d13400640 1 -- 172.21.15.146:0/677523754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 msgr2=0x55fcdfc185f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d13400640 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc185f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d13400640 1 -- 172.21.15.146:0/677523754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 msgr2=0x55fcdfc1af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d13400640 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc1af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d13400640 1 -- 172.21.15.146:0/677523754 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcdfbd6310 con 0x55fcdfc0db40 2024-04-24T08:27:24.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d12a00640 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc185f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:24.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d13400640 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc0df20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009f50 tx=0x7f9d0c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eaa9e8cacaf59 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:24.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d03e00640 1 -- 172.21.15.146:0/677523754 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c004d20 con 0x55fcdfc0db40 2024-04-24T08:27:24.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d03e00640 1 -- 172.21.15.146:0/677523754 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c004050 con 0x55fcdfc0db40 2024-04-24T08:27:24.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d03e00640 1 -- 172.21.15.146:0/677523754 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c005280 con 0x55fcdfc0db40 2024-04-24T08:27:24.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.147+0000 7f9d18a00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9cf80a3d20 0x7f9cf80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d18a00640 1 -- 172.21.15.146:0/2120335666 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9d040d0b70 con 0x7f9d141030c0 2024-04-24T08:27:24.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d12a00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9cf80a3d20 0x7f9cf80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/677523754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 msgr2=0x55fcdfc0df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc0df20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009f50 tx=0x7f9d0c0118d0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/677523754 shutdown_connections 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc1af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d12a00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9cf80a3d20 0x7f9cf80a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55fcdfc0f100 tx=0x7f9d040d4620 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc185f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/677523754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc0df20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/677523754 >> 172.21.15.146:0/677523754 conn(0x55fcdfc09e60 msgr2=0x55fcdfc0a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:24.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.148+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/677523754 shutdown_connections 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.149+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/677523754 wait complete. 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.149+0000 7f9d1cc7f740 1 -- start start 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.149+0000 7f9d1cc7f740 1 Processor -- start 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0db40 0x55fcdfc185a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc15890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc15fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fcdfc1b9c0 con 0x55fcdfc0db40 2024-04-24T08:27:24.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fcdfb2c810 con 0x55fcdfc0e460 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fcdfc1bb40 con 0x55fcdfc18b30 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d1cc7f740 10 client.0 _ll_register_callbacks cb 0x55fcdfafa140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d13400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0db40 0x55fcdfc185a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d0a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc15890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d13400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0db40 0x55fcdfc185a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47246/0 (socket says 172.21.15.146:47246) 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d13400640 1 -- 172.21.15.146:0/4037065701 learned_addr learned my addr 172.21.15.146:0/4037065701 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d0a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc15890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54436/0 (socket says 172.21.15.146:54436) 2024-04-24T08:27:24.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d13400640 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc15fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d0a000640 1 -- 172.21.15.146:0/4037065701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 msgr2=0x55fcdfc15fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d0a000640 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc15fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d0a000640 1 -- 172.21.15.146:0/4037065701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0db40 msgr2=0x55fcdfc185a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d0a000640 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0db40 0x55fcdfc185a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.150+0000 7f9d0a000640 1 -- 172.21.15.146:0/4037065701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcdfb3d3f0 con 0x55fcdfc0e460 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.151+0000 7f9d0a000640 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc15890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009830 tx=0x7f9cfc0305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6acac980a3f6ab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.151+0000 7f9d00c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:27:24.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.151+0000 7f9d00c00640 1 -- 172.21.15.146:0/4037065701 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc009450 con 0x55fcdfc0e460 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.151+0000 7f9d00c00640 1 -- 172.21.15.146:0/4037065701 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9cfc004050 con 0x55fcdfc0e460 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.151+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d0c004050 con 0x55fcdfc0e460 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.151+0000 7f9d00c00640 1 -- 172.21.15.146:0/4037065701 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9cfc004250 con 0x55fcdfc0e460 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 10 client.26639 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fcdf8853f0 con 0x55fcdfc0e460 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d00c00640 1 -- 172.21.15.146:0/4037065701 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c004050 con 0x55fcdfc0e460 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d00c00640 1 -- 172.21.15.146:0/4037065701 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9cfc004990 con 0x55fcdfc0e460 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 10 client.26639 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:24.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 10 client.26639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 20 client.26639 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc28c90 0x55fcdfc2b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 4 client.26639 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.152+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcdfb80bf0 con 0x55fcdfc28c90 2024-04-24T08:27:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.153+0000 7f9d13400640 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc28c90 0x55fcdfc2b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.153+0000 7f9d13400640 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc28c90 0x55fcdfc2b0c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4652046ccbfa1b97 server_cookie=1506ae207f094e1d in_seq=0 out_seq=0 2024-04-24T08:27:24.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.153+0000 7f9d00c00640 10 client.26639 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:24.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d00c00640 1 -- 172.21.15.146:0/4037065701 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcdfb80bf0 con 0x55fcdfc28c90 2024-04-24T08:27:24.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d00c00640 10 client.26639 handle_command_reply: tid=0 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 client.26639 shutdown 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 msgr2=0x55fcdfc15890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc15890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009830 tx=0x7f9cfc0305d0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 shutdown_connections 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc28c90 0x55fcdfc2b0c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc15fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc15890 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/4037065701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0db40 0x55fcdfc185a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 >> 172.21.15.146:0/4037065701 conn(0x55fcdfc09e60 msgr2=0x55fcdfc0fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d00c00640 0 client.26639 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.154+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 shutdown_connections 2024-04-24T08:27:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.155+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/4037065701 wait complete. 2024-04-24T08:27:24.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.155+0000 7f9d1cc7f740 20 client.26639 trim_cache size 0 max 16384 2024-04-24T08:27:24.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.169+0000 7f9d1cc7f740 1 Processor -- start 2024-04-24T08:27:24.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.169+0000 7f9d1cc7f740 1 -- start start 2024-04-24T08:27:24.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0db40 0x55fcdfc102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0e460 0x55fcdfc149f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc08dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d141b0490 con 0x55fcdfc0e460 2024-04-24T08:27:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d141add60 con 0x55fcdfc0db40 2024-04-24T08:27:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9d141abbb0 con 0x55fcdfc18b30 2024-04-24T08:27:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0e460 0x55fcdfc149f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0db40 0x55fcdfc102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0e460 0x55fcdfc149f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47252/0 (socket says 172.21.15.146:47252) 2024-04-24T08:27:24.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 -- 172.21.15.146:0/3659744622 learned_addr learned my addr 172.21.15.146:0/3659744622 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d13400640 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc08dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 -- 172.21.15.146:0/3659744622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 msgr2=0x55fcdfc08dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc08dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 -- 172.21.15.146:0/3659744622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0db40 msgr2=0x55fcdfc102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0db40 0x55fcdfc102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.170+0000 7f9d0a000640 1 -- 172.21.15.146:0/3659744622 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcdf94f7d0 con 0x55fcdfc0e460 2024-04-24T08:27:24.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.171+0000 7f9d0a000640 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0e460 0x55fcdfc149f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc000c00 tx=0x7f9cfc030f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa181a25a05cd3a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:24.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.171+0000 7f9d03400640 1 -- 172.21.15.146:0/3659744622 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc005850 con 0x55fcdfc0e460 2024-04-24T08:27:24.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.171+0000 7f9d03400640 1 -- 172.21.15.146:0/3659744622 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cfc0043f0 con 0x55fcdfc0e460 2024-04-24T08:27:24.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.171+0000 7f9d03400640 1 -- 172.21.15.146:0/3659744622 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc005200 con 0x55fcdfc0e460 2024-04-24T08:27:24.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/3659744622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0e460 msgr2=0x55fcdfc149f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0e460 0x55fcdfc149f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc000c00 tx=0x7f9cfc030f40 comp rx=0 tx=0).stop 2024-04-24T08:27:24.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/3659744622 shutdown_connections 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc18b30 0x55fcdfc08dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc0e460 0x55fcdfc149f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/3659744622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0db40 0x55fcdfc102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/3659744622 >> 172.21.15.146:0/3659744622 conn(0x55fcdfc09e60 msgr2=0x55fcdfc086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/3659744622 shutdown_connections 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/3659744622 wait complete. 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 -- start start 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.172+0000 7f9d1cc7f740 1 Processor -- start 2024-04-24T08:27:24.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc134d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc20d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d1cc7f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc19b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d141ac220 con 0x55fcdfc18b30 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d141ad690 con 0x55fcdfc0e460 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d1cc7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9d141ac8f0 con 0x55fcdfc0db40 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d1cc7f740 10 client.0 _ll_register_callbacks cb 0x7f9d04005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc134d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53316/0 (socket says 172.21.15.146:53316) 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d0a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc20d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d13400640 1 -- 172.21.15.146:0/2508615095 learned_addr learned my addr 172.21.15.146:0/2508615095 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.173+0000 7f9d13400640 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc19b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d0a000640 1 -- 172.21.15.146:0/2508615095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 msgr2=0x55fcdfc134d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d0a000640 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc134d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d0a000640 1 -- 172.21.15.146:0/2508615095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 msgr2=0x55fcdfc19b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d0a000640 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc19b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d0a000640 1 -- 172.21.15.146:0/2508615095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcdfc21630 con 0x55fcdfc0e460 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d0a000640 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc20d90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc02a8f0 tx=0x7f9cfc005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ee5274b715ac8bb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d00c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d00c00640 1 -- 172.21.15.146:0/2508615095 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc05e020 con 0x55fcdfc0e460 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d00c00640 1 -- 172.21.15.146:0/2508615095 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f9cfc0048d0 con 0x55fcdfc0e460 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.174+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cfc0043f0 con 0x55fcdfc0e460 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d00c00640 1 -- 172.21.15.146:0/2508615095 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9cfc069050 con 0x55fcdfc0e460 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 10 client.26645 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fcdfc93c10 con 0x55fcdfc0e460 2024-04-24T08:27:24.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d00c00640 1 -- 172.21.15.146:0/2508615095 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cfc0043f0 con 0x55fcdfc0e460 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d00c00640 1 -- 172.21.15.146:0/2508615095 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9cfc003990 con 0x55fcdfc0e460 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 10 client.26645 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 10 client.26645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 20 client.26645 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc94240 0x55fcdfc96660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 4 client.26645 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcdfc09cb0 con 0x55fcdfc94240 2024-04-24T08:27:24.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.175+0000 7f9d13400640 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc94240 0x55fcdfc96660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d13400640 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc94240 0x55fcdfc96660 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9486134f8c300f server_cookie=f921f47be3eff187 in_seq=0 out_seq=0 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d00c00640 10 client.26645 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d00c00640 1 -- 172.21.15.146:0/2508615095 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcdfc09cb0 con 0x55fcdfc94240 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d00c00640 10 client.26645 handle_command_reply: tid=0 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 client.26645 shutdown 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 msgr2=0x55fcdfc20d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc20d90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc02a8f0 tx=0x7f9cfc005bd0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 shutdown_connections 2024-04-24T08:27:24.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fcdfc94240 0x55fcdfc96660 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fcdfc18b30 0x55fcdfc19b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fcdfc0e460 0x55fcdfc20d90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 --2- 172.21.15.146:0/2508615095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fcdfc0db40 0x55fcdfc134d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d00c00640 0 client.26645 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:24.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 >> 172.21.15.146:0/2508615095 conn(0x55fcdfc09e60 msgr2=0x55fcdfc086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:24.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.176+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 shutdown_connections 2024-04-24T08:27:24.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d1cc7f740 1 -- 172.21.15.146:0/2508615095 wait complete. 2024-04-24T08:27:24.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d1cc7f740 20 client.26645 trim_cache size 0 max 16384 2024-04-24T08:27:24.186 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:27:24.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9cf80a3d20 msgr2=0x7f9cf80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9cf80a3d20 0x7f9cf80a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55fcdfc0f100 tx=0x7f9d040d4620 comp rx=0 tx=0).stop 2024-04-24T08:27:24.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 msgr2=0x7f9d1419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:24.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 0x7f9d1419e470 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400e9c0 tx=0x7f9d0400ee80 comp rx=0 tx=0).stop 2024-04-24T08:27:24.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 shutdown_connections 2024-04-24T08:27:24.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9cf80a3d20 0x7f9cf80a61e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d1410c4e0 0x7f9d14198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d141039e0 0x7f9d1419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 --2- 172.21.15.146:0/2120335666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d141030c0 0x7f9d1419e470 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:24.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 >> 172.21.15.146:0/2120335666 conn(0x7f9d140fc920 msgr2=0x7f9d140fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:24.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 shutdown_connections 2024-04-24T08:27:24.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:24.177+0000 7f9d13e00640 1 -- 172.21.15.146:0/2120335666 wait complete. 2024-04-24T08:27:24.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:24 smithi177 ceph-mon[33046]: pgmap v189: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:27:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:24 smithi153 ceph-mon[32746]: pgmap v189: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:27:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:24 smithi146 ceph-mon[27805]: pgmap v189: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-24T08:27:25.196 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:27:25.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:27:25.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:27:25.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 -- 172.21.15.146:0/1218877131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 msgr2=0x7f9064100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 --2- 172.21.15.146:0/1218877131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f9064100c50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9054009860 tx=0x7f905402a340 comp rx=0 tx=0).stop 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 -- 172.21.15.146:0/1218877131 shutdown_connections 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 --2- 172.21.15.146:0/1218877131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f9064100c50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 --2- 172.21.15.146:0/1218877131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9064101190 0x7f906410eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 --2- 172.21.15.146:0/1218877131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9064107030 0x7f9064107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 -- 172.21.15.146:0/1218877131 >> 172.21.15.146:0/1218877131 conn(0x7f9064076120 msgr2=0x7f9064076530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.419+0000 7f906a200640 1 -- 172.21.15.146:0/1218877131 shutdown_connections 2024-04-24T08:27:25.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.420+0000 7f906a200640 1 -- 172.21.15.146:0/1218877131 wait complete. 2024-04-24T08:27:25.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.420+0000 7f906a200640 1 Processor -- start 2024-04-24T08:27:25.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f906a200640 1 -- start start 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f906a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f90641a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f906a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9064101190 0x7f90641a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f906a200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9064107030 0x7f906419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f906a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9064112270 con 0x7f9064101190 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f906a200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90641120f0 con 0x7f9064107030 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f906a200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f90641123f0 con 0x7f90641007d0 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f9063e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9064101190 0x7f90641a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f9068e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f90641a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f9068e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f90641a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53332/0 (socket says 172.21.15.146:53332) 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f9068e00640 1 -- 172.21.15.146:0/3784309029 learned_addr learned my addr 172.21.15.146:0/3784309029 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.421+0000 7f9069800640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9064107030 0x7f906419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9068e00640 1 -- 172.21.15.146:0/3784309029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9064107030 msgr2=0x7f906419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9068e00640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9064107030 0x7f906419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9068e00640 1 -- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9064101190 msgr2=0x7f90641a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9068e00640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9064101190 0x7f90641a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9068e00640 1 -- 172.21.15.146:0/3784309029 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906419b060 con 0x7f90641007d0 2024-04-24T08:27:25.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9068e00640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f90641a06d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f905800e9c0 tx=0x7f905800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9061600640 1 -- 172.21.15.146:0/3784309029 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905800cd80 con 0x7f90641007d0 2024-04-24T08:27:25.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9061600640 1 -- 172.21.15.146:0/3784309029 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90580035f0 con 0x7f90641007d0 2024-04-24T08:27:25.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f9061600640 1 -- 172.21.15.146:0/3784309029 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058010690 con 0x7f90641007d0 2024-04-24T08:27:25.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.422+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906419b350 con 0x7f90641007d0 2024-04-24T08:27:25.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.423+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90641a7520 con 0x7f90641007d0 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 Processor -- start 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 -- start start 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f9061600640 1 -- 172.21.15.146:0/3784309029 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f90580037a0 con 0x7f90641007d0 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6b90 0x55fe1b200d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b203650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fe1b15be70 con 0x55fe1b1f6b90 2024-04-24T08:27:25.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fe1b102380 con 0x55fe1b201260 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f906e602740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fe1b157e30 con 0x55fe1b1f6270 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f9068e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6b90 0x55fe1b200d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.426+0000 7f9069800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53348/0 (socket says 172.21.15.146:53348) 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 -- 172.21.15.146:0/1538163871 learned_addr learned my addr 172.21.15.146:0/1538163871 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9063e00640 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b203650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 -- 172.21.15.146:0/1538163871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 msgr2=0x55fe1b203650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b203650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 -- 172.21.15.146:0/1538163871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6b90 msgr2=0x55fe1b200d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6b90 0x55fe1b200d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 -- 172.21.15.146:0/1538163871 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe1af367e0 con 0x55fe1b1f6270 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9069800640 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f6650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e580 tx=0x7f905c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d91dda870991541 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.427+0000 7f9052000640 1 -- 172.21.15.146:0/1538163871 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c003970 con 0x55fe1b1f6270 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.428+0000 7f9052000640 1 -- 172.21.15.146:0/1538163871 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905c00d830 con 0x55fe1b1f6270 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.428+0000 7f9052000640 1 -- 172.21.15.146:0/1538163871 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c0045a0 con 0x55fe1b1f6270 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.428+0000 7f9061600640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f904c0a3d20 0x7f904c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.428+0000 7f9061600640 1 -- 172.21.15.146:0/3784309029 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f90580d0ad0 con 0x7f90641007d0 2024-04-24T08:27:25.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.428+0000 7f9068e00640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f904c0a3d20 0x7f904c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.428+0000 7f906e602740 1 -- 172.21.15.146:0/1538163871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6270 msgr2=0x55fe1b1f6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.428+0000 7f906e602740 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f6650 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e580 tx=0x7f905c0095c0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 -- 172.21.15.146:0/1538163871 shutdown_connections 2024-04-24T08:27:25.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b203650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6b90 0x55fe1b200d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 --2- 172.21.15.146:0/1538163871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f6650 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 -- 172.21.15.146:0/1538163871 >> 172.21.15.146:0/1538163871 conn(0x55fe1b1f2590 msgr2=0x55fe1b1f2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 -- 172.21.15.146:0/1538163871 shutdown_connections 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 -- 172.21.15.146:0/1538163871 wait complete. 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f9068e00640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f904c0a3d20 0x7f904c0a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f90580ce2e0 tx=0x7f90580d4620 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 -- start start 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.429+0000 7f906e602740 1 Processor -- start 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f906e602740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1fe220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f906e602740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fe760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f906e602740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b201260 0x55fe1b1feca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f906e602740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fe1b204050 con 0x55fe1b1f6270 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f906e602740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fe1b115660 con 0x55fe1b1f6b90 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f906e602740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fe1b2041d0 con 0x55fe1b201260 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f906e602740 10 client.0 _ll_register_callbacks cb 0x55fe1b103aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f9069800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f9069800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1fe220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47286/0 (socket says 172.21.15.146:47286) 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f9069800640 1 -- 172.21.15.146:0/51659836 learned_addr learned my addr 172.21.15.146:0/51659836 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f9063e00640 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.430+0000 7f9069800640 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b201260 0x55fe1b1feca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9069800640 1 -- 172.21.15.146:0/51659836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b201260 msgr2=0x55fe1b1feca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9069800640 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b201260 0x55fe1b1feca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9069800640 1 -- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b1f6b90 msgr2=0x55fe1b1fe760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9069800640 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fe760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9069800640 1 -- 172.21.15.146:0/51659836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1af96f70 con 0x55fe1b1f6270 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9069800640 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1fe220 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f905c0097f0 tx=0x7f905c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92396d8827a8a2d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9046a00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9046a00640 1 -- 172.21.15.146:0/51659836 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c00de70 con 0x55fe1b1f6270 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f9046a00640 1 -- 172.21.15.146:0/51659836 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f905c01c040 con 0x55fe1b1f6270 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.431+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f905c00d830 con 0x55fe1b1f6270 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f9046a00640 1 -- 172.21.15.146:0/51659836 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f905c034050 con 0x55fe1b1f6270 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 10 client.16512 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fe1ae2e170 con 0x55fe1b1f6270 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f9046a00640 1 -- 172.21.15.146:0/51659836 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905c00d830 con 0x55fe1b1f6270 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f9046a00640 1 -- 172.21.15.146:0/51659836 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f905c00b040 con 0x55fe1b1f6270 2024-04-24T08:27:25.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 10 client.16512 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 20 client.16512 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b211550 0x55fe1b213940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 4 client.16512 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.432+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe1b169890 con 0x55fe1b211550 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.433+0000 7f9063e00640 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b211550 0x55fe1b213940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.433+0000 7f9063e00640 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b211550 0x55fe1b213940 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39e9695aae62c402 server_cookie=52e0ba1cf433fefe in_seq=0 out_seq=0 2024-04-24T08:27:25.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.433+0000 7f9046a00640 10 client.16512 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.434+0000 7f9046a00640 1 -- 172.21.15.146:0/51659836 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe1b169890 con 0x55fe1b211550 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.434+0000 7f9046a00640 10 client.16512 handle_command_reply: tid=0 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.434+0000 7f906e602740 1 client.16512 shutdown 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.434+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 msgr2=0x55fe1b1fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.434+0000 7f906e602740 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1fe220 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f905c0097f0 tx=0x7f905c009780 comp rx=0 tx=0).stop 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 shutdown_connections 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b211550 0x55fe1b213940 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b201260 0x55fe1b1feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 --2- 172.21.15.146:0/51659836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1fe220 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f9046a00640 0 client.16512 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 >> 172.21.15.146:0/51659836 conn(0x55fe1b1f2590 msgr2=0x55fe1b1f8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 shutdown_connections 2024-04-24T08:27:25.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 1 -- 172.21.15.146:0/51659836 wait complete. 2024-04-24T08:27:25.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.435+0000 7f906e602740 20 client.16512 trim_cache size 0 max 16384 2024-04-24T08:27:25.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 Processor -- start 2024-04-24T08:27:25.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 -- start start 2024-04-24T08:27:25.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b2190b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f90641b2720 con 0x55fe1b1f6270 2024-04-24T08:27:25.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90641afff0 con 0x55fe1b201260 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f906e602740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f90641ade40 con 0x55fe1b1f6b90 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.455+0000 7f9069800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47300/0 (socket says 172.21.15.146:47300) 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 -- 172.21.15.146:0/2826046807 learned_addr learned my addr 172.21.15.146:0/2826046807 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9063e00640 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b2190b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 -- 172.21.15.146:0/2826046807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 msgr2=0x55fe1b1fd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 -- 172.21.15.146:0/2826046807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 msgr2=0x55fe1b2190b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b2190b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 -- 172.21.15.146:0/2826046807 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe1b1ede80 con 0x55fe1b1f6270 2024-04-24T08:27:25.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9069800640 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f8320 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f905c000f60 tx=0x7f905c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976fc48d77e5a048 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9051600640 1 -- 172.21.15.146:0/2826046807 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c00de50 con 0x55fe1b1f6270 2024-04-24T08:27:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9051600640 1 -- 172.21.15.146:0/2826046807 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905c0175b0 con 0x55fe1b1f6270 2024-04-24T08:27:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.456+0000 7f9051600640 1 -- 172.21.15.146:0/2826046807 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c004110 con 0x55fe1b1f6270 2024-04-24T08:27:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 -- 172.21.15.146:0/2826046807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 msgr2=0x55fe1b1f8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f8320 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f905c000f60 tx=0x7f905c0095c0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 -- 172.21.15.146:0/2826046807 shutdown_connections 2024-04-24T08:27:25.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b1fd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b2190b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 --2- 172.21.15.146:0/2826046807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b1f8320 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 -- 172.21.15.146:0/2826046807 >> 172.21.15.146:0/2826046807 conn(0x55fe1b1f2590 msgr2=0x55fe1b1f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 -- 172.21.15.146:0/2826046807 shutdown_connections 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.457+0000 7f906e602740 1 -- 172.21.15.146:0/2826046807 wait complete. 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.458+0000 7f906e602740 1 -- start start 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.458+0000 7f906e602740 1 Processor -- start 2024-04-24T08:27:25.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.458+0000 7f906e602740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b2794b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.458+0000 7f906e602740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b2799f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f906e602740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b205680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f906e602740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f90641ae4b0 con 0x55fe1b1f6270 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f906e602740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90641af920 con 0x55fe1b201260 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f906e602740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f90641aeb80 con 0x55fe1b1f6b90 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f906e602740 10 client.0 _ll_register_callbacks cb 0x7f90580d44f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b2794b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b2794b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47316/0 (socket says 172.21.15.146:47316) 2024-04-24T08:27:25.465 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 -- 172.21.15.146:0/3929511409 learned_addr learned my addr 172.21.15.146:0/3929511409 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9063e00640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b2799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b205680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 -- 172.21.15.146:0/3929511409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 msgr2=0x55fe1b2799f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b2799f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 -- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 msgr2=0x55fe1b205680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b205680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 -- 172.21.15.146:0/3929511409 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1b205d70 con 0x55fe1b1f6270 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9063e00640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b2799f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9069800640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b2794b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f905c009ec0 tx=0x7f905c003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d8df19f2dd34de server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.459+0000 7f9046a00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f9046a00640 1 -- 172.21.15.146:0/3929511409 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c017760 con 0x55fe1b1f6270 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f9046a00640 1 -- 172.21.15.146:0/3929511409 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f905c02c040 con 0x55fe1b1f6270 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f905c0175b0 con 0x55fe1b1f6270 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f9046a00640 1 -- 172.21.15.146:0/3929511409 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f905c017a30 con 0x55fe1b1f6270 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f906e602740 10 client.16518 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:25.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fe1b280c40 con 0x55fe1b1f6270 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f9046a00640 1 -- 172.21.15.146:0/3929511409 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905c0175b0 con 0x55fe1b1f6270 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.460+0000 7f9046a00640 1 -- 172.21.15.146:0/3929511409 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f905c00b050 con 0x55fe1b1f6270 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f906e602740 10 client.16518 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f906e602740 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f906e602740 20 client.16518 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f906e602740 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b281110 0x55fe1b283500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f906e602740 4 client.16518 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fe1b163700 con 0x55fe1b281110 2024-04-24T08:27:25.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f9063e00640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b281110 0x55fe1b283500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f9063e00640 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b281110 0x55fe1b283500 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3942c21bb6a12e21 server_cookie=2853111229bfd1dd in_seq=0 out_seq=0 2024-04-24T08:27:25.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.461+0000 7f9046a00640 10 client.16518 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f9046a00640 1 -- 172.21.15.146:0/3929511409 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fe1b163700 con 0x55fe1b281110 2024-04-24T08:27:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f9046a00640 10 client.16518 handle_command_reply: tid=0 2024-04-24T08:27:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 client.16518 shutdown 2024-04-24T08:27:25.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 msgr2=0x55fe1b2794b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b2794b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f905c009ec0 tx=0x7f905c003d40 comp rx=0 tx=0).stop 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 shutdown_connections 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fe1b281110 0x55fe1b283500 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe1b1f6b90 0x55fe1b2799f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f9046a00640 0 client.16518 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fe1b201260 0x55fe1b205680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 --2- 172.21.15.146:0/3929511409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fe1b1f6270 0x55fe1b2794b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 >> 172.21.15.146:0/3929511409 conn(0x55fe1b1f2590 msgr2=0x55fe1b1f16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 shutdown_connections 2024-04-24T08:27:25.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 1 -- 172.21.15.146:0/3929511409 wait complete. 2024-04-24T08:27:25.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.463+0000 7f906e602740 20 client.16518 trim_cache size 0 max 16384 2024-04-24T08:27:25.472 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:27:25.472 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:27:25.472 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "3de33791-b204-44fb-b416-e700a673290d", 2024-04-24T08:27:25.472 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:27:25.472 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:27:25.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f904c0a3d20 msgr2=0x7f904c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f904c0a3d20 0x7f904c0a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f90580ce2e0 tx=0x7f90580d4620 comp rx=0 tx=0).stop 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 msgr2=0x7f90641a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f90641a06d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f905800e9c0 tx=0x7f905800ee80 comp rx=0 tx=0).stop 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 shutdown_connections 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f904c0a3d20 0x7f904c0a61e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9064107030 0x7f906419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9064101190 0x7f90641a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 --2- 172.21.15.146:0/3784309029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90641007d0 0x7f90641a06d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 >> 172.21.15.146:0/3784309029 conn(0x7f9064076120 msgr2=0x7f9064101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 shutdown_connections 2024-04-24T08:27:25.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.464+0000 7f906a200640 1 -- 172.21.15.146:0/3784309029 wait complete. 2024-04-24T08:27:25.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:27:25.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:27:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.733+0000 7f6e4a800640 1 -- 172.21.15.146:0/1606712032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108610 msgr2=0x7f6e44108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.733+0000 7f6e4a800640 1 --2- 172.21.15.146:0/1606712032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108610 0x7f6e44108a90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009930 tx=0x7f6e3002a440 comp rx=0 tx=0).stop 2024-04-24T08:27:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.734+0000 7f6e4a800640 1 -- 172.21.15.146:0/1606712032 shutdown_connections 2024-04-24T08:27:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.734+0000 7f6e4a800640 1 --2- 172.21.15.146:0/1606712032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108610 0x7f6e44108a90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.734+0000 7f6e4a800640 1 --2- 172.21.15.146:0/1606712032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e44108fd0 0x7f6e44115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.734+0000 7f6e4a800640 1 --2- 172.21.15.146:0/1606712032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e4410e660 0x7f6e4410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.735 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.734+0000 7f6e4a800640 1 -- 172.21.15.146:0/1606712032 >> 172.21.15.146:0/1606712032 conn(0x7f6e44076540 msgr2=0x7f6e44078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.734+0000 7f6e4a800640 1 -- 172.21.15.146:0/1606712032 shutdown_connections 2024-04-24T08:27:25.736 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.734+0000 7f6e4a800640 1 -- 172.21.15.146:0/1606712032 wait complete. 2024-04-24T08:27:25.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.736+0000 7f6e4a800640 1 Processor -- start 2024-04-24T08:27:25.737 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.736+0000 7f6e4a800640 1 -- start start 2024-04-24T08:27:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e4a800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e44108610 0x7f6e4419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e4a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108fd0 0x7f6e4419f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e4a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e4410e660 0x7f6e441a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e4a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6e441189f0 con 0x7f6e4410e660 2024-04-24T08:27:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e4a800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e44118870 con 0x7f6e44108610 2024-04-24T08:27:25.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e4a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e44118b70 con 0x7f6e44108fd0 2024-04-24T08:27:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e43400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e44108610 0x7f6e4419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e43400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e44108610 0x7f6e4419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54534/0 (socket says 172.21.15.146:54534) 2024-04-24T08:27:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e43400640 1 -- 172.21.15.146:0/2223766628 learned_addr learned my addr 172.21.15.146:0/2223766628 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e42a00640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108fd0 0x7f6e4419f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.737+0000 7f6e43e00640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e4410e660 0x7f6e441a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.738+0000 7f6e43400640 1 -- 172.21.15.146:0/2223766628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108fd0 msgr2=0x7f6e4419f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.738+0000 7f6e43400640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108fd0 0x7f6e4419f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.738+0000 7f6e43400640 1 -- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e4410e660 msgr2=0x7f6e441a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.738+0000 7f6e43400640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e4410e660 0x7f6e441a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.738+0000 7f6e43400640 1 -- 172.21.15.146:0/2223766628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e441a3e60 con 0x7f6e44108610 2024-04-24T08:27:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.738+0000 7f6e43400640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e44108610 0x7f6e4419ee70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400e9c0 tx=0x7f6e3400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.739+0000 7f6e3be00640 1 -- 172.21.15.146:0/2223766628 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3400cd80 con 0x7f6e44108610 2024-04-24T08:27:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.739+0000 7f6e3be00640 1 -- 172.21.15.146:0/2223766628 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e340035f0 con 0x7f6e44108610 2024-04-24T08:27:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.739+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e441a4150 con 0x7f6e44108610 2024-04-24T08:27:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.739+0000 7f6e3be00640 1 -- 172.21.15.146:0/2223766628 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e34010690 con 0x7f6e44108610 2024-04-24T08:27:25.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.740+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e441ab9e0 con 0x7f6e44108610 2024-04-24T08:27:25.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.744+0000 7f6e3be00640 1 -- 172.21.15.146:0/2223766628 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f6e340037a0 con 0x7f6e44108610 2024-04-24T08:27:25.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.746+0000 7f6e4d76e740 1 Processor -- start 2024-04-24T08:27:25.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.747+0000 7f6e3be00640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6e280a3e70 0x7f6e280a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.747+0000 7f6e3be00640 1 -- 172.21.15.146:0/2223766628 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6e340d0c00 con 0x7f6e44108610 2024-04-24T08:27:25.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.747+0000 7f6e4d76e740 1 -- start start 2024-04-24T08:27:25.749 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.748+0000 7f6e43e00640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6e280a3e70 0x7f6e280a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf927c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf932310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e43400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf927c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e43e00640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6e280a3e70 0x7f6e280a6330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c005fd0 tx=0x7f6e3c009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf934c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf932310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5597cf88d1f0 con 0x5597cf932850 2024-04-24T08:27:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5597cf833350 con 0x5597cf927860 2024-04-24T08:27:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5597cf8891b0 con 0x5597cf928180 2024-04-24T08:27:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e42a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf932310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53414/0 (socket says 172.21.15.146:53414) 2024-04-24T08:27:25.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.749+0000 7f6e42a00640 1 -- 172.21.15.146:0/1614088258 learned_addr learned my addr 172.21.15.146:0/1614088258 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43400640 1 -- 172.21.15.146:0/1614088258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 msgr2=0x5597cf932310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43400640 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf932310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43400640 1 -- 172.21.15.146:0/1614088258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 msgr2=0x5597cf934c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43e00640 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf934c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43400640 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf934c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43400640 1 -- 172.21.15.146:0/1614088258 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597cf887f50 con 0x5597cf927860 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43e00640 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf934c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e43400640 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf927c40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6e340ce360 tx=0x7f6e3409cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f92eb81aefba98f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.750+0000 7f6e38c00640 1 -- 172.21.15.146:0/1614088258 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340b5020 con 0x5597cf927860 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.751+0000 7f6e38c00640 1 -- 172.21.15.146:0/1614088258 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e340a5820 con 0x5597cf927860 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.751+0000 7f6e38c00640 1 -- 172.21.15.146:0/1614088258 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340ce770 con 0x5597cf927860 2024-04-24T08:27:25.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.751+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1614088258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 msgr2=0x5597cf927c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.751+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf927c40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6e340ce360 tx=0x7f6e3409cab0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1614088258 shutdown_connections 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf932310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf934c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1614088258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf927c40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1614088258 >> 172.21.15.146:0/1614088258 conn(0x5597cf923b80 msgr2=0x5597cf923f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1614088258 shutdown_connections 2024-04-24T08:27:25.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1614088258 wait complete. 2024-04-24T08:27:25.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.752+0000 7f6e4d76e740 1 -- start start 2024-04-24T08:27:25.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.753+0000 7f6e4d76e740 1 Processor -- start 2024-04-24T08:27:25.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf927860 0x5597cf92f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf928180 0x5597cf92fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf932850 0x5597cf930300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5597cf9355e0 con 0x5597cf928180 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5597cf846690 con 0x5597cf932850 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5597cf935760 con 0x5597cf927860 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e4d76e740 10 client.0 _ll_register_callbacks cb 0x5597cf882650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf927860 0x5597cf92f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf927860 0x5597cf92f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:53426/0 (socket says 172.21.15.146:53426) 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 -- 172.21.15.146:0/1034452748 learned_addr learned my addr 172.21.15.146:0/1034452748 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e42a00640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf928180 0x5597cf92fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43e00640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf932850 0x5597cf930300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 -- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf932850 msgr2=0x5597cf930300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf932850 0x5597cf930300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 -- 172.21.15.146:0/1034452748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf928180 msgr2=0x5597cf92fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf928180 0x5597cf92fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43400640 1 -- 172.21.15.146:0/1034452748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597cf845920 con 0x5597cf927860 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e42a00640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf928180 0x5597cf92fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:27:25.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.754+0000 7f6e43e00640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf932850 0x5597cf930300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e43400640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf927860 0x5597cf92f880 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6e3409ae60 tx=0x7f6e340ce5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fe9d1461fae56bd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e2d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e2d600640 1 -- 172.21.15.146:0/1034452748 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340b5070 con 0x5597cf927860 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e2d600640 1 -- 172.21.15.146:0/1034452748 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6e340bd0d0 con 0x5597cf927860 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e340a5820 con 0x5597cf927860 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e2d600640 1 -- 172.21.15.146:0/1034452748 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6e3409c000 con 0x5597cf927860 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e4d76e740 10 client.26656 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5597cf6697d0 con 0x5597cf927860 2024-04-24T08:27:25.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.755+0000 7f6e2d600640 1 -- 172.21.15.146:0/1034452748 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e340a5820 con 0x5597cf927860 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e2d600640 1 -- 172.21.15.146:0/1034452748 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6e3409ba70 con 0x5597cf927860 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e4d76e740 10 client.26656 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e4d76e740 10 client.26656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e4d76e740 20 client.26656 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf942a00 0x5597cf944df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e4d76e740 4 client.26656 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597cf89ac10 con 0x5597cf942a00 2024-04-24T08:27:25.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.756+0000 7f6e42a00640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf942a00 0x5597cf944df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.757+0000 7f6e42a00640 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf942a00 0x5597cf944df0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30e4a4072207b479 server_cookie=30ef671ea6f91fcf in_seq=0 out_seq=0 2024-04-24T08:27:25.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.757+0000 7f6e2d600640 10 client.26656 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.758+0000 7f6e2d600640 1 -- 172.21.15.146:0/1034452748 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597cf89ac10 con 0x5597cf942a00 2024-04-24T08:27:25.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.758+0000 7f6e2d600640 10 client.26656 handle_command_reply: tid=0 2024-04-24T08:27:25.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.758+0000 7f6e4d76e740 1 client.26656 shutdown 2024-04-24T08:27:25.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.758+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf927860 msgr2=0x5597cf92f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.758+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf927860 0x5597cf92f880 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6e3409ae60 tx=0x7f6e340ce5e0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 shutdown_connections 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf942a00 0x5597cf944df0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf932850 0x5597cf930300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf928180 0x5597cf92fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1034452748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf927860 0x5597cf92f880 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 >> 172.21.15.146:0/1034452748 conn(0x5597cf923b80 msgr2=0x5597cf929980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e2d600640 0 client.26656 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 shutdown_connections 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1034452748 wait complete. 2024-04-24T08:27:25.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.759+0000 7f6e4d76e740 20 client.26656 trim_cache size 0 max 16384 2024-04-24T08:27:25.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.780+0000 7f6e4d76e740 1 Processor -- start 2024-04-24T08:27:25.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.780+0000 7f6e4d76e740 1 -- start start 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf927860 0x5597cf93e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf928180 0x5597cf947a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf932850 0x5597cf947f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6e441b6c10 con 0x5597cf927860 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e441b44e0 con 0x5597cf928180 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e441b2330 con 0x5597cf932850 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf928180 0x5597cf947a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e43400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf927860 0x5597cf93e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf928180 0x5597cf947a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54564/0 (socket says 172.21.15.146:54564) 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 -- 172.21.15.146:0/1264402802 learned_addr learned my addr 172.21.15.146:0/1264402802 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e43e00640 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf932850 0x5597cf947f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 -- 172.21.15.146:0/1264402802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf932850 msgr2=0x5597cf947f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf932850 0x5597cf947f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 -- 172.21.15.146:0/1264402802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf927860 msgr2=0x5597cf93e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf927860 0x5597cf93e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.781+0000 7f6e42a00640 1 -- 172.21.15.146:0/1264402802 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597cf59f3f0 con 0x5597cf928180 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.782+0000 7f6e42a00640 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf928180 0x5597cf947a30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009930 tx=0x7f6e30005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=228ad7e7089e449 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.782+0000 7f6e2fe00640 1 -- 172.21.15.146:0/1264402802 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30035040 con 0x5597cf928180 2024-04-24T08:27:25.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.782+0000 7f6e2fe00640 1 -- 172.21.15.146:0/1264402802 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e30005520 con 0x5597cf928180 2024-04-24T08:27:25.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.782+0000 7f6e2fe00640 1 -- 172.21.15.146:0/1264402802 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e300433e0 con 0x5597cf928180 2024-04-24T08:27:25.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1264402802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf928180 msgr2=0x5597cf947a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf928180 0x5597cf947a30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009930 tx=0x7f6e30005a40 comp rx=0 tx=0).stop 2024-04-24T08:27:25.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1264402802 shutdown_connections 2024-04-24T08:27:25.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf932850 0x5597cf947f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf928180 0x5597cf947a30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/1264402802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf927860 0x5597cf93e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1264402802 >> 172.21.15.146:0/1264402802 conn(0x5597cf923b80 msgr2=0x5597cf93d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1264402802 shutdown_connections 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 -- 172.21.15.146:0/1264402802 wait complete. 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.783+0000 7f6e4d76e740 1 -- start start 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 1 Processor -- start 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf947c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf945060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf9455a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6e441b29a0 con 0x5597cf932850 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6e441b3e10 con 0x5597cf927860 2024-04-24T08:27:25.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e441b3070 con 0x5597cf928180 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e4d76e740 10 client.0 _ll_register_callbacks cb 0x7f6e3c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e43e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf9455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e43400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf947c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e43400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf947c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:54578/0 (socket says 172.21.15.146:54578) 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.784+0000 7f6e43400640 1 -- 172.21.15.146:0/3890136982 learned_addr learned my addr 172.21.15.146:0/3890136982 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e42a00640 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf945060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e43e00640 1 -- 172.21.15.146:0/3890136982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 msgr2=0x5597cf945060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e43e00640 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf945060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e43e00640 1 -- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 msgr2=0x5597cf947c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:25.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e43e00640 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf947c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e43e00640 1 -- 172.21.15.146:0/3890136982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597cf55f0f0 con 0x5597cf932850 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e43400640 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf947c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e43e00640 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf9455a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x5597cf928e90 tx=0x7f6e3c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1930e1014f0b6e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e2d600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.785+0000 7f6e2d600640 1 -- 172.21.15.146:0/3890136982 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c01fb90 con 0x5597cf932850 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e2d600640 1 -- 172.21.15.146:0/3890136982 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f6e3c01fd30 con 0x5597cf932850 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e30005520 con 0x5597cf932850 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e2d600640 1 -- 172.21.15.146:0/3890136982 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6e3c003970 con 0x5597cf932850 2024-04-24T08:27:25.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 10 client.16533 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5597cf852810 con 0x5597cf932850 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e2d600640 1 -- 172.21.15.146:0/3890136982 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e30005520 con 0x5597cf932850 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e2d600640 1 -- 172.21.15.146:0/3890136982 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6e3c029420 con 0x5597cf932850 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 10 client.16533 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 20 client.16533 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf9a8f00 0x5597cf9ab320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 4 client.16533 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:27:25.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.786+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5597cf894a80 con 0x5597cf9a8f00 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e43400640 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf9a8f00 0x5597cf9ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e43400640 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf9a8f00 0x5597cf9ab320 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ee6ec4dd9acfc5e server_cookie=b5d96542cfc893 in_seq=0 out_seq=0 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e2d600640 10 client.16533 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e2d600640 1 -- 172.21.15.146:0/3890136982 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5597cf894a80 con 0x5597cf9a8f00 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e2d600640 10 client.16533 handle_command_reply: tid=0 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e4d76e740 1 client.16533 shutdown 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 msgr2=0x5597cf9455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.787+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf9455a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x5597cf928e90 tx=0x7f6e3c004ab0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 shutdown_connections 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5597cf9a8f00 0x5597cf9ab320 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597cf928180 0x5597cf945060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5597cf932850 0x5597cf9455a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e2d600640 0 client.16533 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 --2- 172.21.15.146:0/3890136982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5597cf927860 0x5597cf947c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 >> 172.21.15.146:0/3890136982 conn(0x5597cf923b80 msgr2=0x5597cf922c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 shutdown_connections 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 1 -- 172.21.15.146:0/3890136982 wait complete. 2024-04-24T08:27:25.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.788+0000 7f6e4d76e740 20 client.16533 trim_cache size 0 max 16384 2024-04-24T08:27:25.796 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:27:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:27:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:27:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "3de33791-b204-44fb-b416-e700a673290d": { 2024-04-24T08:27:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:27:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "3de33791-b204-44fb-b416-e700a673290d", 2024-04-24T08:27:25.796 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:27:25.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:27:25.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:27:25.797 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6e280a3e70 msgr2=0x7f6e280a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6e280a3e70 0x7f6e280a6330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c005fd0 tx=0x7f6e3c009500 comp rx=0 tx=0).stop 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e44108610 msgr2=0x7f6e4419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e44108610 0x7f6e4419ee70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400e9c0 tx=0x7f6e3400ee80 comp rx=0 tx=0).stop 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 shutdown_connections 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6e280a3e70 0x7f6e280a6330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e44108fd0 0x7f6e4419f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e4410e660 0x7f6e441a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 --2- 172.21.15.146:0/2223766628 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6e44108610 0x7f6e4419ee70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:25.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 >> 172.21.15.146:0/2223766628 conn(0x7f6e44076540 msgr2=0x7f6e44077cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:25.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 shutdown_connections 2024-04-24T08:27:25.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:25.789+0000 7f6e4a800640 1 -- 172.21.15.146:0/2223766628 wait complete. 2024-04-24T08:27:25.808 INFO:tasks.cephfs.filesystem:scrub status for tag:3de33791-b204-44fb-b416-e700a673290d - {'path': '/', 'tag': '3de33791-b204-44fb-b416-e700a673290d', 'options': 'recursive,force'} 2024-04-24T08:27:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: pgmap v190: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:27:26.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:26 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: pgmap v190: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:27:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:27:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:27:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:27:26.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:27:26.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:26 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: pgmap v190: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:27:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:27:26.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:27:26.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:27:26.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:27:26.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:26.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:26 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:28 smithi177 ceph-mon[33046]: pgmap v191: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:28 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:27:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:28 smithi153 ceph-mon[32746]: pgmap v191: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:28 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:27:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:28 smithi146 ceph-mon[27805]: pgmap v191: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:28.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:28 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:27:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:29 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:27:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:29 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:27:29.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:29 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:27:29.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:29 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:27:30.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:29 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:27:30.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:29 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:27:30.055 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:29 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:27:30.055 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:29 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:27:30.055 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:29 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:27:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:30 smithi177 ceph-mon[33046]: pgmap v192: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:30 smithi153 ceph-mon[32746]: pgmap v192: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:30.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:30 smithi146 ceph-mon[27805]: pgmap v192: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:32 smithi177 ceph-mon[33046]: pgmap v193: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:32 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:27:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:32 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:27:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:32 smithi153 ceph-mon[32746]: pgmap v193: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:32 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:27:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:32 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:27:32.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:32 smithi146 ceph-mon[27805]: pgmap v193: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:32.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:32 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:27:32.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:32 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:27:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:34 smithi177 ceph-mon[33046]: pgmap v194: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:34.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:34 smithi153 ceph-mon[32746]: pgmap v194: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:34 smithi146 ceph-mon[27805]: pgmap v194: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:36 smithi177 ceph-mon[33046]: pgmap v195: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:36.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:36 smithi153 ceph-mon[32746]: pgmap v195: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:36.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:36 smithi146 ceph-mon[27805]: pgmap v195: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:27:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:38 smithi177 ceph-mon[33046]: pgmap v196: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:38.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:38 smithi153 ceph-mon[32746]: pgmap v196: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:38.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:38 smithi146 ceph-mon[27805]: pgmap v196: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:27:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:40 smithi177 ceph-mon[33046]: pgmap v197: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:40 smithi153 ceph-mon[32746]: pgmap v197: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:40.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:40 smithi146 ceph-mon[27805]: pgmap v197: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:42 smithi177 ceph-mon[33046]: pgmap v198: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:42.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:42 smithi153 ceph-mon[32746]: pgmap v198: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:42.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:42 smithi146 ceph-mon[27805]: pgmap v198: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:44 smithi177 ceph-mon[33046]: pgmap v199: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:44.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:44 smithi153 ceph-mon[32746]: pgmap v199: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:44.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:44 smithi146 ceph-mon[27805]: pgmap v199: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:46 smithi177 ceph-mon[33046]: pgmap v200: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:46.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:46 smithi153 ceph-mon[32746]: pgmap v200: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:46.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:46 smithi146 ceph-mon[27805]: pgmap v200: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:48 smithi177 ceph-mon[33046]: pgmap v201: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:48.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:48 smithi153 ceph-mon[32746]: pgmap v201: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:48.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:48 smithi146 ceph-mon[27805]: pgmap v201: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:50 smithi177 ceph-mon[33046]: pgmap v202: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:50.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:50 smithi153 ceph-mon[32746]: pgmap v202: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:50.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:50 smithi146 ceph-mon[27805]: pgmap v202: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:52 smithi153 ceph-mon[32746]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:52.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:52 smithi146 ceph-mon[27805]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:53.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:52 smithi177 ceph-mon[33046]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:54.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:54 smithi153 ceph-mon[32746]: pgmap v204: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:54.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:54 smithi146 ceph-mon[27805]: pgmap v204: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:55.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:54 smithi177 ceph-mon[33046]: pgmap v204: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:55.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:27:55.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:27:56.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 -- 172.21.15.146:0/963484571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a8109580 msgr2=0x7f08a8111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 --2- 172.21.15.146:0/963484571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a8109580 0x7f08a8111150 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009900 tx=0x7f08a002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 -- 172.21.15.146:0/963484571 shutdown_connections 2024-04-24T08:27:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 --2- 172.21.15.146:0/963484571 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a8109580 0x7f08a8111150 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 --2- 172.21.15.146:0/963484571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a8104e60 0x7f08a8108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 --2- 172.21.15.146:0/963484571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 0x7f08a8104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 -- 172.21.15.146:0/963484571 >> 172.21.15.146:0/963484571 conn(0x7f08a80ffd90 msgr2=0x7f08a81021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 -- 172.21.15.146:0/963484571 shutdown_connections 2024-04-24T08:27:56.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.023+0000 7f08aee00640 1 -- 172.21.15.146:0/963484571 wait complete. 2024-04-24T08:27:56.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.024+0000 7f08aee00640 1 Processor -- start 2024-04-24T08:27:56.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.024+0000 7f08aee00640 1 -- start start 2024-04-24T08:27:56.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08aee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 0x7f08a81a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08aee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a8109580 0x7f08a81a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08aee00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a819a970 0x7f08a819ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08aee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f08a8114690 con 0x7f08a8109580 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08aee00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08a8114510 con 0x7f08a819a970 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08aee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08a8114810 con 0x7f08a81044b0 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 0x7f08a81a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f089f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a8109580 0x7f08a81a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08ace00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a819a970 0x7f08a819ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 0x7f08a81a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38638/0 (socket says 172.21.15.146:38638) 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08a7e00640 1 -- 172.21.15.146:0/247786411 learned_addr learned my addr 172.21.15.146:0/247786411 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08ace00640 1 -- 172.21.15.146:0/247786411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 msgr2=0x7f08a81a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08ace00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 0x7f08a81a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08ace00640 1 -- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a8109580 msgr2=0x7f08a81a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08ace00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a8109580 0x7f08a81a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.025+0000 7f08ace00640 1 -- 172.21.15.146:0/247786411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a819b490 con 0x7f08a819a970 2024-04-24T08:27:56.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.026+0000 7f08a7e00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 0x7f08a81a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.026+0000 7f08ace00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a819a970 0x7f08a819ae20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f08a002a910 tx=0x7f08a0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.026+0000 7f08a5600640 1 -- 172.21.15.146:0/247786411 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a003f070 con 0x7f08a819a970 2024-04-24T08:27:56.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.026+0000 7f08a5600640 1 -- 172.21.15.146:0/247786411 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a0031c90 con 0x7f08a819a970 2024-04-24T08:27:56.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.026+0000 7f08a5600640 1 -- 172.21.15.146:0/247786411 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0035030 con 0x7f08a819a970 2024-04-24T08:27:56.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.026+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a819b720 con 0x7f08a819a970 2024-04-24T08:27:56.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.026+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a81a78c0 con 0x7f08a819a970 2024-04-24T08:27:56.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08a5600640 1 -- 172.21.15.146:0/247786411 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f08a0042070 con 0x7f08a819a970 2024-04-24T08:27:56.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 Processor -- start 2024-04-24T08:27:56.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 -- start start 2024-04-24T08:27:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985faee310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985faee850 0x55985faf0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55985fa491f0 con 0x55985fae3860 2024-04-24T08:27:56.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55985f9ef350 con 0x55985fae4180 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55985fa451b0 con 0x55985faee850 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f089f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985faee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985faee310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58390/0 (socket says 172.21.15.146:58390) 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f089f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43068/0 (socket says 172.21.15.146:43068) 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.030+0000 7f08a7e00640 1 -- 172.21.15.146:0/592956245 learned_addr learned my addr 172.21.15.146:0/592956245 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f089f400640 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985faee850 0x55985faf0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f08a7e00640 1 -- 172.21.15.146:0/592956245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985faee850 msgr2=0x55985faf0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f08a7e00640 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985faee850 0x55985faf0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f08a7e00640 1 -- 172.21.15.146:0/592956245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 msgr2=0x55985fae3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f08a7e00640 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f08a7e00640 1 -- 172.21.15.146:0/592956245 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985fa43f50 con 0x55985fae4180 2024-04-24T08:27:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f089f400640 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f08a7e00640 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985faee310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0898011910 tx=0x7f0898011dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90ab5d59024fe6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.031+0000 7f08a5600640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f088c0a3da0 0x7f088c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f089d600640 1 -- 172.21.15.146:0/592956245 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898011330 con 0x55985fae4180 2024-04-24T08:27:56.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08a5600640 1 -- 172.21.15.146:0/247786411 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f08a00f0d10 con 0x7f08a819a970 2024-04-24T08:27:56.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f089d600640 1 -- 172.21.15.146:0/592956245 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0898010430 con 0x55985fae4180 2024-04-24T08:27:56.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f089d600640 1 -- 172.21.15.146:0/592956245 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898019640 con 0x55985fae4180 2024-04-24T08:27:56.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f089f400640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f088c0a3da0 0x7f088c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08b1bbe740 1 -- 172.21.15.146:0/592956245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 msgr2=0x55985faee310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985faee310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0898011910 tx=0x7f0898011dd0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f089f400640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f088c0a3da0 0x7f088c0a6260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55985fae0d10 tx=0x7f0890015040 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08b1bbe740 1 -- 172.21.15.146:0/592956245 shutdown_connections 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985faee850 0x55985faf0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985faee310 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/592956245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.032+0000 7f08b1bbe740 1 -- 172.21.15.146:0/592956245 >> 172.21.15.146:0/592956245 conn(0x55985fadfb80 msgr2=0x55985fadff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.033+0000 7f08b1bbe740 1 -- 172.21.15.146:0/592956245 shutdown_connections 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.033+0000 7f08b1bbe740 1 -- 172.21.15.146:0/592956245 wait complete. 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.033+0000 7f08b1bbe740 1 -- start start 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.033+0000 7f08b1bbe740 1 Processor -- start 2024-04-24T08:27:56.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faeb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fae4180 0x55985faebba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985faee850 0x55985faec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55985faf1640 con 0x55985fae3860 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55985fa02690 con 0x55985faee850 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55985faf17c0 con 0x55985fae4180 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08b1bbe740 10 client.0 _ll_register_callbacks cb 0x55985fa3e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faeb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faeb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43084/0 (socket says 172.21.15.146:43084) 2024-04-24T08:27:56.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 -- 172.21.15.146:0/3357450713 learned_addr learned my addr 172.21.15.146:0/3357450713 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f089f400640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fae4180 0x55985faebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08ace00640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985faee850 0x55985faec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 -- 172.21.15.146:0/3357450713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fae4180 msgr2=0x55985faebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fae4180 0x55985faebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 -- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985faee850 msgr2=0x55985faec0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985faee850 0x55985faec0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.034+0000 7f08a7e00640 1 -- 172.21.15.146:0/3357450713 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985fa01920 con 0x55985fae3860 2024-04-24T08:27:56.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.035+0000 7f08ace00640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985faee850 0x55985faec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.035+0000 7f08a7e00640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faeb660 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0898010a50 tx=0x7f0898011340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894362dda91444de server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.035+0000 7f0896000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.035+0000 7f0896000640 1 -- 172.21.15.146:0/3357450713 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898028020 con 0x55985fae3860 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.035+0000 7f0896000640 1 -- 172.21.15.146:0/3357450713 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f0898004c40 con 0x55985fae3860 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.035+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0898010430 con 0x55985fae3860 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f0896000640 1 -- 172.21.15.146:0/3357450713 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0898033050 con 0x55985fae3860 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 10 client.16545 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:56.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55985f8257d0 con 0x55985fae3860 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f0896000640 1 -- 172.21.15.146:0/3357450713 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0898010430 con 0x55985fae3860 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f0896000640 1 -- 172.21.15.146:0/3357450713 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0898018070 con 0x55985fae3860 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 10 client.16545 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 20 client.16545 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985fafea30 0x55985fb00e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 4 client.16545 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.036+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55985fa56c10 con 0x55985fafea30 2024-04-24T08:27:56.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.037+0000 7f089f400640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985fafea30 0x55985fb00e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.037+0000 7f089f400640 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985fafea30 0x55985fb00e20 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=300854bb25c4e39f server_cookie=c04d6101af0524ad in_seq=0 out_seq=0 2024-04-24T08:27:56.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.037+0000 7f0896000640 10 client.16545 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:56.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f0896000640 1 -- 172.21.15.146:0/3357450713 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55985fa56c10 con 0x55985fafea30 2024-04-24T08:27:56.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f0896000640 10 client.16545 handle_command_reply: tid=0 2024-04-24T08:27:56.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 client.16545 shutdown 2024-04-24T08:27:56.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 msgr2=0x55985faeb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faeb660 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0898010a50 tx=0x7f0898011340 comp rx=0 tx=0).stop 2024-04-24T08:27:56.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 shutdown_connections 2024-04-24T08:27:56.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985fafea30 0x55985fb00e20 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fae4180 0x55985faebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985faee850 0x55985faec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/3357450713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faeb660 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f0896000640 0 client.16545 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 >> 172.21.15.146:0/3357450713 conn(0x55985fadfb80 msgr2=0x55985fae5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 shutdown_connections 2024-04-24T08:27:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.038+0000 7f08b1bbe740 1 -- 172.21.15.146:0/3357450713 wait complete. 2024-04-24T08:27:56.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.039+0000 7f08b1bbe740 20 client.16545 trim_cache size 0 max 16384 2024-04-24T08:27:56.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.057+0000 7f08b1bbe740 1 Processor -- start 2024-04-24T08:27:56.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08b1bbe740 1 -- start start 2024-04-24T08:27:56.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985fafa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fadeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81b27c0 con 0x55985fae3860 2024-04-24T08:27:56.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81b0090 con 0x55985fae4180 2024-04-24T08:27:56.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81adee0 con 0x55985fadfb80 2024-04-24T08:27:56.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f089f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985fafa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985fafa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38666/0 (socket says 172.21.15.146:38666) 2024-04-24T08:27:56.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f089f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43096/0 (socket says 172.21.15.146:43096) 2024-04-24T08:27:56.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08a7e00640 1 -- 172.21.15.146:0/417161188 learned_addr learned my addr 172.21.15.146:0/417161188 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.058+0000 7f08ace00640 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fadeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f08a7e00640 1 -- 172.21.15.146:0/417161188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 msgr2=0x55985fadeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f08a7e00640 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fadeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f08a7e00640 1 -- 172.21.15.146:0/417161188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 msgr2=0x55985fae5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f08a7e00640 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f08a7e00640 1 -- 172.21.15.146:0/417161188 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985f75b3f0 con 0x55985fadfb80 2024-04-24T08:27:56.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f089f400640 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f08a7e00640 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985fafa530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0898011e10 tx=0x7f08980021f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa2cf12a3068efe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f089cc00640 1 -- 172.21.15.146:0/417161188 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898028040 con 0x55985fadfb80 2024-04-24T08:27:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f089cc00640 1 -- 172.21.15.146:0/417161188 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0898010430 con 0x55985fadfb80 2024-04-24T08:27:56.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.059+0000 7f089cc00640 1 -- 172.21.15.146:0/417161188 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089801f040 con 0x55985fadfb80 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 -- 172.21.15.146:0/417161188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 msgr2=0x55985fafa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985fafa530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0898011e10 tx=0x7f08980021f0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 -- 172.21.15.146:0/417161188 shutdown_connections 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fadeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985fae5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/417161188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985fafa530 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 -- 172.21.15.146:0/417161188 >> 172.21.15.146:0/417161188 conn(0x55985fb01190 msgr2=0x55985fade390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 -- 172.21.15.146:0/417161188 shutdown_connections 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 -- 172.21.15.146:0/417161188 wait complete. 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.060+0000 7f08b1bbe740 1 -- start start 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 1 Processor -- start 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985faf6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faf6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fae7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81ae550 con 0x55985fae3860 2024-04-24T08:27:56.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81af9c0 con 0x55985fae4180 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08a81aec20 con 0x55985fadfb80 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f08b1bbe740 10 client.0 _ll_register_callbacks cb 0x7f0898010980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f089f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faf6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f089f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faf6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43110/0 (socket says 172.21.15.146:43110) 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.061+0000 7f089f400640 1 -- 172.21.15.146:0/140778151 learned_addr learned my addr 172.21.15.146:0/140778151 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f08ace00640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fae7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f08a7e00640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985faf6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f089f400640 1 -- 172.21.15.146:0/140778151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 msgr2=0x55985faf6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f089f400640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985faf6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f089f400640 1 -- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 msgr2=0x55985fae7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f089f400640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fae7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f089f400640 1 -- 172.21.15.146:0/140778151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985f71b0f0 con 0x55985fae3860 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f08ace00640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fae7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f089f400640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faf6f80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f089000f7f0 tx=0x7f0890004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b961fe66bbf456 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f0896000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f0896000640 1 -- 172.21.15.146:0/140778151 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089002c3b0 con 0x55985fae3860 2024-04-24T08:27:56.071 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.062+0000 7f0896000640 1 -- 172.21.15.146:0/140778151 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f089002c550 con 0x55985fae3860 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0898010430 con 0x55985fae3860 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f0896000640 1 -- 172.21.15.146:0/140778151 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f089003c050 con 0x55985fae3860 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 10 client.16551 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55985fa0e810 con 0x55985fae3860 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f0896000640 1 -- 172.21.15.146:0/140778151 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0898010430 con 0x55985fae3860 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f0896000640 1 -- 172.21.15.146:0/140778151 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0890021c80 con 0x55985fae3860 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 10 client.16551 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:56.072 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 20 client.16551 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985faee850 0x55985fb6f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 4 client.16551 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.063+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55985fa50a80 con 0x55985faee850 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.064+0000 7f08a7e00640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985faee850 0x55985fb6f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.064+0000 7f08a7e00640 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985faee850 0x55985fb6f650 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52be9d5d75a1b100 server_cookie=dbbb179041273326 in_seq=0 out_seq=0 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.064+0000 7f0896000640 10 client.16551 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.064+0000 7f0896000640 1 -- 172.21.15.146:0/140778151 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55985fa50a80 con 0x55985faee850 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.064+0000 7f0896000640 10 client.16551 handle_command_reply: tid=0 2024-04-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.064+0000 7f08b1bbe740 1 client.16551 shutdown 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 msgr2=0x55985faf6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faf6f80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f089000f7f0 tx=0x7f0890004730 comp rx=0 tx=0).stop 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 shutdown_connections 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55985faee850 0x55985fb6f650 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55985fae4180 0x55985fae7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f0896000640 0 client.16551 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55985fae3860 0x55985faf6f80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.074 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 --2- 172.21.15.146:0/140778151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55985fadfb80 0x55985faf6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 >> 172.21.15.146:0/140778151 conn(0x55985fb01190 msgr2=0x55985fadec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 shutdown_connections 2024-04-24T08:27:56.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 1 -- 172.21.15.146:0/140778151 wait complete. 2024-04-24T08:27:56.075 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.065+0000 7f08b1bbe740 20 client.16551 trim_cache size 0 max 16384 2024-04-24T08:27:56.075 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:27:56.075 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:27:56.075 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:27:56.076 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:27:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f088c0a3da0 msgr2=0x7f088c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f088c0a3da0 0x7f088c0a6260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x55985fae0d10 tx=0x7f0890015040 comp rx=0 tx=0).stop 2024-04-24T08:27:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a819a970 msgr2=0x7f08a819ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a819a970 0x7f08a819ae20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f08a002a910 tx=0x7f08a0003b90 comp rx=0 tx=0).stop 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 shutdown_connections 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f088c0a3da0 0x7f088c0a6260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08a819a970 0x7f08a819ae20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08a8109580 0x7f08a81a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 --2- 172.21.15.146:0/247786411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08a81044b0 0x7f08a81a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 >> 172.21.15.146:0/247786411 conn(0x7f08a80ffd90 msgr2=0x7f08a8102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 shutdown_connections 2024-04-24T08:27:56.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.066+0000 7f08aee00640 1 -- 172.21.15.146:0/247786411 wait complete. 2024-04-24T08:27:56.085 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:27:56.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:27:56.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 -- 172.21.15.146:0/2629646414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50104e00 msgr2=0x7f4c50108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 --2- 172.21.15.146:0/2629646414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50104e00 0x7f4c50108e40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c009860 tx=0x7f4c3c02a340 comp rx=0 tx=0).stop 2024-04-24T08:27:56.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 -- 172.21.15.146:0/2629646414 shutdown_connections 2024-04-24T08:27:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 --2- 172.21.15.146:0/2629646414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50109520 0x7f4c501110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 --2- 172.21.15.146:0/2629646414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50104e00 0x7f4c50108e40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 --2- 172.21.15.146:0/2629646414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c50104450 0x7f4c50104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 -- 172.21.15.146:0/2629646414 >> 172.21.15.146:0/2629646414 conn(0x7f4c500ffcf0 msgr2=0x7f4c50102110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 -- 172.21.15.146:0/2629646414 shutdown_connections 2024-04-24T08:27:56.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.277+0000 7f4c57e00640 1 -- 172.21.15.146:0/2629646414 wait complete. 2024-04-24T08:27:56.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.278+0000 7f4c57e00640 1 Processor -- start 2024-04-24T08:27:56.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.278+0000 7f4c57e00640 1 -- start start 2024-04-24T08:27:56.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c57e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50104450 0x7f4c501a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c57e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50109520 0x7f4c501a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c57e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5019a950 0x7f4c5019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c57e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4c501145b0 con 0x7f4c5019a950 2024-04-24T08:27:56.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c57e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c50114430 con 0x7f4c50109520 2024-04-24T08:27:56.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c57e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c50114730 con 0x7f4c50104450 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c55e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5019a950 0x7f4c5019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c54a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50109520 0x7f4c501a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c55400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50104450 0x7f4c501a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c55e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5019a950 0x7f4c5019ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43126/0 (socket says 172.21.15.146:43126) 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c55400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50104450 0x7f4c501a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38686/0 (socket says 172.21.15.146:38686) 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c54a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50109520 0x7f4c501a0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58416/0 (socket says 172.21.15.146:58416) 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c55e00640 1 -- 172.21.15.146:0/710836896 learned_addr learned my addr 172.21.15.146:0/710836896 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c54a00640 1 -- 172.21.15.146:0/710836896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50104450 msgr2=0x7f4c501a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c54a00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50104450 0x7f4c501a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c54a00640 1 -- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5019a950 msgr2=0x7f4c5019ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c54a00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5019a950 0x7f4c5019ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c54a00640 1 -- 172.21.15.146:0/710836896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5019b470 con 0x7f4c50109520 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c55400640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50104450 0x7f4c501a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.279+0000 7f4c55e00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5019a950 0x7f4c5019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.280+0000 7f4c54a00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50109520 0x7f4c501a0ce0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c02a860 tx=0x7f4c3c033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.280+0000 7f4c4e000640 1 -- 172.21.15.146:0/710836896 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c037040 con 0x7f4c50109520 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.280+0000 7f4c4e000640 1 -- 172.21.15.146:0/710836896 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c3c033d50 con 0x7f4c50109520 2024-04-24T08:27:56.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.280+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5019b700 con 0x7f4c50109520 2024-04-24T08:27:56.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.280+0000 7f4c4e000640 1 -- 172.21.15.146:0/710836896 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c003920 con 0x7f4c50109520 2024-04-24T08:27:56.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.280+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c501a78f0 con 0x7f4c50109520 2024-04-24T08:27:56.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.281+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c50069270 con 0x7f4c50109520 2024-04-24T08:27:56.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.284+0000 7f4c4e000640 1 -- 172.21.15.146:0/710836896 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f4c3c042020 con 0x7f4c50109520 2024-04-24T08:27:56.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.285+0000 7f4c4e000640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c380a3d20 0x7f4c380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.285+0000 7f4c4e000640 1 -- 172.21.15.146:0/710836896 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4c3c0f01a0 con 0x7f4c50109520 2024-04-24T08:27:56.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.285+0000 7f4c55400640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c380a3d20 0x7f4c380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.286+0000 7f4c55400640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c380a3d20 0x7f4c380a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c440096c0 tx=0x7f4c44009260 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.287+0000 7f4c4e000640 1 -- 172.21.15.146:0/710836896 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c3c049020 con 0x7f4c50109520 2024-04-24T08:27:56.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.532+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c50104e00 con 0x7f4c50109520 2024-04-24T08:27:56.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.533+0000 7f4c4e000640 1 -- 172.21.15.146:0/710836896 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f4c3c0bc900 con 0x7f4c50109520 2024-04-24T08:27:56.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:27:56.536 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:27:56.536 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:27:56.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c380a3d20 msgr2=0x7f4c380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c380a3d20 0x7f4c380a61e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c440096c0 tx=0x7f4c44009260 comp rx=0 tx=0).stop 2024-04-24T08:27:56.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50109520 msgr2=0x7f4c501a0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50109520 0x7f4c501a0ce0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c02a860 tx=0x7f4c3c033b60 comp rx=0 tx=0).stop 2024-04-24T08:27:56.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 shutdown_connections 2024-04-24T08:27:56.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4c380a3d20 0x7f4c380a61e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c5019a950 0x7f4c5019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c50109520 0x7f4c501a0ce0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 --2- 172.21.15.146:0/710836896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c50104450 0x7f4c501a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 >> 172.21.15.146:0/710836896 conn(0x7f4c500ffcf0 msgr2=0x7f4c50101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 shutdown_connections 2024-04-24T08:27:56.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.536+0000 7f4c57e00640 1 -- 172.21.15.146:0/710836896 wait complete. 2024-04-24T08:27:56.557 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:27:56.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 -- 172.21.15.146:0/3585973175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc103060 msgr2=0x7feffc103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 --2- 172.21.15.146:0/3585973175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc103060 0x7feffc103440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feff4009900 tx=0x7feff402a480 comp rx=0 tx=0).stop 2024-04-24T08:27:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 -- 172.21.15.146:0/3585973175 shutdown_connections 2024-04-24T08:27:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 --2- 172.21.15.146:0/3585973175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feffc103980 0x7feffc10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 --2- 172.21.15.146:0/3585973175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feffc10c480 0x7feffc10e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 --2- 172.21.15.146:0/3585973175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc103060 0x7feffc103440 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 -- 172.21.15.146:0/3585973175 >> 172.21.15.146:0/3585973175 conn(0x7feffc0fc880 msgr2=0x7feffc0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 -- 172.21.15.146:0/3585973175 shutdown_connections 2024-04-24T08:27:56.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.761+0000 7feffb200640 1 -- 172.21.15.146:0/3585973175 wait complete. 2024-04-24T08:27:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.762+0000 7feffb200640 1 Processor -- start 2024-04-24T08:27:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.762+0000 7feffb200640 1 -- start start 2024-04-24T08:27:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feffb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feffc103060 0x7feffc1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feffb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feffc103980 0x7feffc1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feffb200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc10c480 0x7feffc19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feffb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feffc112290 con 0x7feffc103980 2024-04-24T08:27:56.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feffb200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feffc112110 con 0x7feffc10c480 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feffb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feffc112410 con 0x7feffc103060 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feff9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feffc103980 0x7feffc1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feff9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feffc103980 0x7feffc1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43150/0 (socket says 172.21.15.146:43150) 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feff9400640 1 -- 172.21.15.146:0/405737221 learned_addr learned my addr 172.21.15.146:0/405737221 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feffa800640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc10c480 0x7feffc19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.763+0000 7feff9e00640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feffc103060 0x7feffc1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffa800640 1 -- 172.21.15.146:0/405737221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feffc103060 msgr2=0x7feffc1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffa800640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feffc103060 0x7feffc1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffa800640 1 -- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feffc103980 msgr2=0x7feffc1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffa800640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feffc103980 0x7feffc1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffa800640 1 -- 172.21.15.146:0/405737221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc19b000 con 0x7feffc10c480 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffa800640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc10c480 0x7feffc19a740 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fefec00d680 tx=0x7fefec00db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feff2a00640 1 -- 172.21.15.146:0/405737221 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec017070 con 0x7feffc10c480 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feff2a00640 1 -- 172.21.15.146:0/405737221 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefec009820 con 0x7feffc10c480 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc19b2f0 con 0x7feffc10c480 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feff2a00640 1 -- 172.21.15.146:0/405737221 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec004e00 con 0x7feffc10c480 2024-04-24T08:27:56.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.764+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc1a7600 con 0x7feffc10c480 2024-04-24T08:27:56.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 Processor -- start 2024-04-24T08:27:56.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7feff2a00640 1 -- 172.21.15.146:0/405737221 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fefec0099d0 con 0x7feffc10c480 2024-04-24T08:27:56.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 -- start start 2024-04-24T08:27:56.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8db40 0x559ec8e8df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e985f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e9af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x559ec8df31d0 con 0x559ec8e8e460 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559ec8d99450 con 0x559ec8e8db40 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.768+0000 7ff003866740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ec8def190 con 0x559ec8e98b30 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8db40 0x559ec8e8df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8db40 0x559ec8e8df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58440/0 (socket says 172.21.15.146:58440) 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 -- 172.21.15.146:0/2604141712 learned_addr learned my addr 172.21.15.146:0/2604141712 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e9af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9e00640 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 -- 172.21.15.146:0/2604141712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 msgr2=0x559ec8e9af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e9af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 -- 172.21.15.146:0/2604141712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 msgr2=0x559ec8e985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 -- 172.21.15.146:0/2604141712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ec8e56310 con 0x559ec8e8db40 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff2a00640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fefe00a3da0 0x7fefe00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.769+0000 7feff9400640 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8db40 0x559ec8e8df20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fefe4013b80 tx=0x7fefe4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2255d54fd8f7fdff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.770+0000 7feff2a00640 1 -- 172.21.15.146:0/405737221 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fefec0cff60 con 0x7feffc10c480 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.770+0000 7feff9e00640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fefe00a3da0 0x7fefe00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.770+0000 7fefeb000640 1 -- 172.21.15.146:0/2604141712 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe4013c70 con 0x559ec8e8db40 2024-04-24T08:27:56.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.770+0000 7fefeb000640 1 -- 172.21.15.146:0/2604141712 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefe4013e10 con 0x559ec8e8db40 2024-04-24T08:27:56.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.770+0000 7fefeb000640 1 -- 172.21.15.146:0/2604141712 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe400f360 con 0x559ec8e8db40 2024-04-24T08:27:56.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.770+0000 7ff003866740 1 -- 172.21.15.146:0/2604141712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8db40 msgr2=0x559ec8e8df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.770+0000 7ff003866740 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8db40 0x559ec8e8df20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fefe4013b80 tx=0x7fefe4009b80 comp rx=0 tx=0).stop 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7feff9e00640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fefe00a3da0 0x7fefe00a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff4005c00 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 -- 172.21.15.146:0/2604141712 shutdown_connections 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e9af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 --2- 172.21.15.146:0/2604141712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8db40 0x559ec8e8df20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 -- 172.21.15.146:0/2604141712 >> 172.21.15.146:0/2604141712 conn(0x559ec8e89e60 msgr2=0x559ec8e8a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 -- 172.21.15.146:0/2604141712 shutdown_connections 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 -- 172.21.15.146:0/2604141712 wait complete. 2024-04-24T08:27:56.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 -- start start 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.771+0000 7ff003866740 1 Processor -- start 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7ff003866740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8e985d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7ff003866740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8e460 0x559ec8e958c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7ff003866740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e95fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7ff003866740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x559ec8e9b9c0 con 0x559ec8e8db40 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7ff003866740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559ec8dac810 con 0x559ec8e8e460 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7ff003866740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ec8e9bb40 con 0x559ec8e98b30 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7ff003866740 10 client.0 _ll_register_callbacks cb 0x559ec8d7a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:56.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7feff9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8e985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7feff9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8e985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43166/0 (socket says 172.21.15.146:43166) 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7feff9e00640 1 -- 172.21.15.146:0/2734381227 learned_addr learned my addr 172.21.15.146:0/2734381227 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7feff9400640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8e460 0x559ec8e958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.772+0000 7feffa800640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e95fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7feff9400640 1 -- 172.21.15.146:0/2734381227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 msgr2=0x559ec8e95fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7feff9400640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e95fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7feff9400640 1 -- 172.21.15.146:0/2734381227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 msgr2=0x559ec8e985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7feff9400640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8e985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7feff9400640 1 -- 172.21.15.146:0/2734381227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ec8dbd3f0 con 0x559ec8e8e460 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7feffa800640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e95fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7feff9400640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8e460 0x559ec8e958c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fefe4020fd0 tx=0x7fefe4005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=401cd141476a0826 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7fefdf400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7fefdf400640 1 -- 172.21.15.146:0/2734381227 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe4009670 con 0x559ec8e8e460 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefe4013e10 con 0x559ec8e8e460 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.773+0000 7fefdf400640 1 -- 172.21.15.146:0/2734381227 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fefe4011e20 con 0x559ec8e8e460 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7fefdf400640 1 -- 172.21.15.146:0/2734381227 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fefe4004a40 con 0x559ec8e8e460 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 10 client.26717 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:56.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ec8b053f0 con 0x559ec8e8e460 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7fefdf400640 1 -- 172.21.15.146:0/2734381227 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefe4013e10 con 0x559ec8e8e460 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7fefdf400640 1 -- 172.21.15.146:0/2734381227 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fefe400f040 con 0x559ec8e8e460 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 10 client.26717 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 10 client.26717 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 20 client.26717 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8ea8ce0 0x559ec8eab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 4 client.26717 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:56.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.774+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ec8e00bf0 con 0x559ec8ea8ce0 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.775+0000 7feff9e00640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8ea8ce0 0x559ec8eab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.775+0000 7feff9e00640 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8ea8ce0 0x559ec8eab0d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7fbc83da63f8b7ee server_cookie=aaa591e50d87176d in_seq=0 out_seq=0 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.775+0000 7fefdf400640 10 client.26717 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7fefdf400640 1 -- 172.21.15.146:0/2734381227 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ec8e00bf0 con 0x559ec8ea8ce0 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7fefdf400640 10 client.26717 handle_command_reply: tid=0 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 client.26717 shutdown 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8e460 msgr2=0x559ec8e958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8e460 0x559ec8e958c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fefe4020fd0 tx=0x7fefe4005bd0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 shutdown_connections 2024-04-24T08:27:56.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8ea8ce0 0x559ec8eab0d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e98b30 0x559ec8e95fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e8e460 0x559ec8e958c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7fefdf400640 0 client.26717 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 --2- 172.21.15.146:0/2734381227 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8e985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.776+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 >> 172.21.15.146:0/2734381227 conn(0x559ec8e89e60 msgr2=0x559ec8e8fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.777+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 shutdown_connections 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.777+0000 7ff003866740 1 -- 172.21.15.146:0/2734381227 wait complete. 2024-04-24T08:27:56.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.777+0000 7ff003866740 20 client.26717 trim_cache size 0 max 16384 2024-04-24T08:27:56.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 Processor -- start 2024-04-24T08:27:56.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 -- start start 2024-04-24T08:27:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8db40 0x559ec8ea4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e98100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8eb0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1b2570 con 0x559ec8e8e460 2024-04-24T08:27:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1afe40 con 0x559ec8e98b30 2024-04-24T08:27:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.792+0000 7ff003866740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1adc90 con 0x559ec8e8db40 2024-04-24T08:27:56.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e98100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feffa800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8eb0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8db40 0x559ec8ea4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e98100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43178/0 (socket says 172.21.15.146:43178) 2024-04-24T08:27:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9400640 1 -- 172.21.15.146:0/2456472520 learned_addr learned my addr 172.21.15.146:0/2456472520 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9e00640 1 -- 172.21.15.146:0/2456472520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 msgr2=0x559ec8eb0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9e00640 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8eb0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9e00640 1 -- 172.21.15.146:0/2456472520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 msgr2=0x559ec8e98100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9e00640 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e98100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9e00640 1 -- 172.21.15.146:0/2456472520 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ec8e8e290 con 0x559ec8e8db40 2024-04-24T08:27:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9400640 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e98100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feffa800640 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8eb0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:56.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.793+0000 7feff9e00640 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8db40 0x559ec8ea4880 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x559ec8eab7a0 tx=0x7feff403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72b74aa7f4314a7a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7fefea600640 1 -- 172.21.15.146:0/2456472520 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4035040 con 0x559ec8e8db40 2024-04-24T08:27:56.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7fefea600640 1 -- 172.21.15.146:0/2456472520 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feff4054880 con 0x559ec8e8db40 2024-04-24T08:27:56.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7fefea600640 1 -- 172.21.15.146:0/2456472520 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40325c0 con 0x559ec8e8db40 2024-04-24T08:27:56.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7ff003866740 1 -- 172.21.15.146:0/2456472520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8db40 msgr2=0x559ec8ea4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7ff003866740 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8db40 0x559ec8ea4880 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x559ec8eab7a0 tx=0x7feff403f000 comp rx=0 tx=0).stop 2024-04-24T08:27:56.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7ff003866740 1 -- 172.21.15.146:0/2456472520 shutdown_connections 2024-04-24T08:27:56.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7ff003866740 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8eb0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7ff003866740 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8e460 0x559ec8e98100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7ff003866740 1 --2- 172.21.15.146:0/2456472520 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8db40 0x559ec8ea4880 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.794+0000 7ff003866740 1 -- 172.21.15.146:0/2456472520 >> 172.21.15.146:0/2456472520 conn(0x559ec8e89e60 msgr2=0x559ec8e90b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.795+0000 7ff003866740 1 -- 172.21.15.146:0/2456472520 shutdown_connections 2024-04-24T08:27:56.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.795+0000 7ff003866740 1 -- 172.21.15.146:0/2456472520 wait complete. 2024-04-24T08:27:56.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.795+0000 7ff003866740 1 -- start start 2024-04-24T08:27:56.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.795+0000 7ff003866740 1 Processor -- start 2024-04-24T08:27:56.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7ff003866740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8ea1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7ff003866740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8e460 0x559ec8e995c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7ff003866740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8e99b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7ff003866740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1ae300 con 0x559ec8e8db40 2024-04-24T08:27:56.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7ff003866740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1af770 con 0x559ec8e98b30 2024-04-24T08:27:56.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7ff003866740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1ae9d0 con 0x559ec8e8e460 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7ff003866740 10 client.0 _ll_register_callbacks cb 0x7feff4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8ea1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8ea1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43182/0 (socket says 172.21.15.146:43182) 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 -- 172.21.15.146:0/3015525041 learned_addr learned my addr 172.21.15.146:0/3015525041 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9400640 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8e460 0x559ec8e995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feffa800640 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8e99b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 -- 172.21.15.146:0/3015525041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8e460 msgr2=0x559ec8e995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8e460 0x559ec8e995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 -- 172.21.15.146:0/3015525041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 msgr2=0x559ec8e99b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:56.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8e99b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.796+0000 7feff9e00640 1 -- 172.21.15.146:0/3015525041 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ec8bcf7d0 con 0x559ec8e8db40 2024-04-24T08:27:56.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7feff9e00640 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8ea1190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7feff402a4c0 tx=0x7feff4005190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e4c33110043e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:56.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7fefdf400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:56.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7fefdf400640 1 -- 172.21.15.146:0/3015525041 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4061020 con 0x559ec8e8db40 2024-04-24T08:27:56.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7fefdf400640 1 -- 172.21.15.146:0/3015525041 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7feff40046c0 con 0x559ec8e8db40 2024-04-24T08:27:56.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feff4054880 con 0x559ec8e8db40 2024-04-24T08:27:56.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7fefdf400640 1 -- 172.21.15.146:0/3015525041 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feff406c050 con 0x559ec8e8db40 2024-04-24T08:27:56.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7ff003866740 10 client.16575 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:56.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ec8f0f0d0 con 0x559ec8e8db40 2024-04-24T08:27:56.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.797+0000 7fefdf400640 1 -- 172.21.15.146:0/3015525041 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feff4054880 con 0x559ec8e8db40 2024-04-24T08:27:56.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7fefdf400640 1 -- 172.21.15.146:0/3015525041 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7feff4003b70 con 0x559ec8e8db40 2024-04-24T08:27:56.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7ff003866740 10 client.16575 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:56.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7ff003866740 10 client.16575 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:27:56.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7ff003866740 20 client.16575 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:56.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7ff003866740 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8f0f630 0x559ec8f11a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:56.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7ff003866740 4 client.16575 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:27:56.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ec8dfaa60 con 0x559ec8f0f630 2024-04-24T08:27:56.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.798+0000 7feff9400640 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8f0f630 0x559ec8f11a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:56.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7feff9400640 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8f0f630 0x559ec8f11a20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4b4613f49eb0587 server_cookie=12e147163c7f6506 in_seq=0 out_seq=0 2024-04-24T08:27:56.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7fefdf400640 10 client.16575 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:56.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7fefdf400640 1 -- 172.21.15.146:0/3015525041 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ec8dfaa60 con 0x559ec8f0f630 2024-04-24T08:27:56.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7fefdf400640 10 client.16575 handle_command_reply: tid=0 2024-04-24T08:27:56.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 client.16575 shutdown 2024-04-24T08:27:56.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 msgr2=0x559ec8ea1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8ea1190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7feff402a4c0 tx=0x7feff4005190 comp rx=0 tx=0).stop 2024-04-24T08:27:56.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 shutdown_connections 2024-04-24T08:27:56.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x559ec8f0f630 0x559ec8f11a20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ec8e8e460 0x559ec8e995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559ec8e98b30 0x559ec8e99b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 --2- 172.21.15.146:0/3015525041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559ec8e8db40 0x559ec8ea1190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 >> 172.21.15.146:0/3015525041 conn(0x559ec8e89e60 msgr2=0x559ec8e886c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7fefdf400640 0 client.16575 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 shutdown_connections 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 1 -- 172.21.15.146:0/3015525041 wait complete. 2024-04-24T08:27:56.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.799+0000 7ff003866740 20 client.16575 trim_cache size 0 max 16384 2024-04-24T08:27:56.814 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:27:56.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fefe00a3da0 msgr2=0x7fefe00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fefe00a3da0 0x7fefe00a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7feff4000c00 tx=0x7feff4005c00 comp rx=0 tx=0).stop 2024-04-24T08:27:56.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc10c480 msgr2=0x7feffc19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:56.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc10c480 0x7feffc19a740 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fefec00d680 tx=0x7fefec00db40 comp rx=0 tx=0).stop 2024-04-24T08:27:56.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 shutdown_connections 2024-04-24T08:27:56.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fefe00a3da0 0x7fefe00a6260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feffc10c480 0x7feffc19a740 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feffc103980 0x7feffc1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 --2- 172.21.15.146:0/405737221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feffc103060 0x7feffc1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:56.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 >> 172.21.15.146:0/405737221 conn(0x7feffc0fc880 msgr2=0x7feffc100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:56.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 shutdown_connections 2024-04-24T08:27:56.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:56.800+0000 7feffb200640 1 -- 172.21.15.146:0/405737221 wait complete. 2024-04-24T08:27:56.817 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:56 smithi146 ceph-mon[27805]: pgmap v205: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:56.817 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:56 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/710836896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:27:56.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:27:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:56 smithi153 ceph-mon[32746]: pgmap v205: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:56 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/710836896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:27:57.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 -- 172.21.15.146:0/1280141941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c10e740 msgr2=0x7fc69c110b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 --2- 172.21.15.146:0/1280141941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c10e740 0x7fc69c110b50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc694009be0 tx=0x7fc69402a750 comp rx=0 tx=0).stop 2024-04-24T08:27:57.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 -- 172.21.15.146:0/1280141941 shutdown_connections 2024-04-24T08:27:57.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 --2- 172.21.15.146:0/1280141941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c10e740 0x7fc69c110b50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 --2- 172.21.15.146:0/1280141941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c074cf0 0x7fc69c10e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 --2- 172.21.15.146:0/1280141941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc69c0743d0 0x7fc69c0747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 -- 172.21.15.146:0/1280141941 >> 172.21.15.146:0/1280141941 conn(0x7fc69c0fd9d0 msgr2=0x7fc69c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 -- 172.21.15.146:0/1280141941 shutdown_connections 2024-04-24T08:27:57.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.012+0000 7fc69b200640 1 -- 172.21.15.146:0/1280141941 wait complete. 2024-04-24T08:27:57.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.013+0000 7fc69b200640 1 Processor -- start 2024-04-24T08:27:57.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.013+0000 7fc69b200640 1 -- start start 2024-04-24T08:27:57.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc69b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc69c0743d0 0x7fc69c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc69b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c074cf0 0x7fc69c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc69b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c10e740 0x7fc69c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc69b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c114460 con 0x7fc69c0743d0 2024-04-24T08:27:57.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc69b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1142e0 con 0x7fc69c10e740 2024-04-24T08:27:57.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc69b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1145e0 con 0x7fc69c074cf0 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc691400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c074cf0 0x7fc69c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc699e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc69c0743d0 0x7fc69c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc691400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c074cf0 0x7fc69c1a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38740/0 (socket says 172.21.15.146:38740) 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc691400640 1 -- 172.21.15.146:0/2590307041 learned_addr learned my addr 172.21.15.146:0/2590307041 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.014+0000 7fc69a800640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c10e740 0x7fc69c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc691400640 1 -- 172.21.15.146:0/2590307041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c10e740 msgr2=0x7fc69c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc691400640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c10e740 0x7fc69c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc691400640 1 -- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc69c0743d0 msgr2=0x7fc69c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc691400640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc69c0743d0 0x7fc69c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc691400640 1 -- 172.21.15.146:0/2590307041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c19af70 con 0x7fc69c074cf0 2024-04-24T08:27:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc69a800640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c10e740 0x7fc69c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:27:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc691400640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c074cf0 0x7fc69c1a0b70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc68400e9c0 tx=0x7fc68400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc693400640 1 -- 172.21.15.146:0/2590307041 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68400cde0 con 0x7fc69c074cf0 2024-04-24T08:27:57.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc693400640 1 -- 172.21.15.146:0/2590307041 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6840035f0 con 0x7fc69c074cf0 2024-04-24T08:27:57.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c19b260 con 0x7fc69c074cf0 2024-04-24T08:27:57.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc693400640 1 -- 172.21.15.146:0/2590307041 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684010690 con 0x7fc69c074cf0 2024-04-24T08:27:57.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.015+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c1a7480 con 0x7fc69c074cf0 2024-04-24T08:27:57.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc693400640 1 -- 172.21.15.146:0/2590307041 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fc6840037a0 con 0x7fc69c074cf0 2024-04-24T08:27:57.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 Processor -- start 2024-04-24T08:27:57.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 -- start start 2024-04-24T08:27:57.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b4ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b59b30 0x557149b5bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557149ab41d0 con 0x557149b59b30 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557149a5a450 con 0x557149b4f460 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.019+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557149ab0190 con 0x557149b4eb40 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b4ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b59b30 0x557149b5bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b4ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38744/0 (socket says 172.21.15.146:38744) 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 -- 172.21.15.146:0/1783904368 learned_addr learned my addr 172.21.15.146:0/1783904368 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc699e00640 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 -- 172.21.15.146:0/1783904368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 msgr2=0x557149b595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 -- 172.21.15.146:0/1783904368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b59b30 msgr2=0x557149b5bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b59b30 0x557149b5bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc69a800640 1 -- 172.21.15.146:0/1783904368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557149b17310 con 0x557149b4eb40 2024-04-24T08:27:57.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.020+0000 7fc699e00640 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.021+0000 7fc69a800640 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b4ef20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc69402ac70 tx=0x7fc694030a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee96fff3bca35dc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.021+0000 7fc693400640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc6800a3d20 0x7fc6800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.021+0000 7fc68b400640 1 -- 172.21.15.146:0/1783904368 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694040030 con 0x557149b4eb40 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.021+0000 7fc68b400640 1 -- 172.21.15.146:0/1783904368 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc694030c20 con 0x557149b4eb40 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.021+0000 7fc68b400640 1 -- 172.21.15.146:0/1783904368 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694036040 con 0x557149b4eb40 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.021+0000 7fc693400640 1 -- 172.21.15.146:0/2590307041 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc6840d0b50 con 0x7fc69c074cf0 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.021+0000 7fc699e00640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc6800a3d20 0x7fc6800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 -- 172.21.15.146:0/1783904368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 msgr2=0x557149b4ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b4ef20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc69402ac70 tx=0x7fc694030a30 comp rx=0 tx=0).stop 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 -- 172.21.15.146:0/1783904368 shutdown_connections 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b59b30 0x557149b5bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/1783904368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b4ef20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 -- 172.21.15.146:0/1783904368 >> 172.21.15.146:0/1783904368 conn(0x557149b4ae60 msgr2=0x557149b4b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 -- 172.21.15.146:0/1783904368 shutdown_connections 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc6a3be0740 1 -- 172.21.15.146:0/1783904368 wait complete. 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.022+0000 7fc699e00640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc6800a3d20 0x7fc6800a61e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x557149b50100 tx=0x7fc68c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 -- start start 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 Processor -- start 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b594f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b567e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b56df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557149b5c960 con 0x557149b4eb40 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557149a6d810 con 0x557149b4f460 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557149b5cae0 con 0x557149b59b30 2024-04-24T08:27:57.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.023+0000 7fc6a3be0740 10 client.0 _ll_register_callbacks cb 0x557149a3b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc699e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b56df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b56df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38760/0 (socket says 172.21.15.146:38760) 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 -- 172.21.15.146:0/984685401 learned_addr learned my addr 172.21.15.146:0/984685401 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc691400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 -- 172.21.15.146:0/984685401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 msgr2=0x557149b567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 -- 172.21.15.146:0/984685401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 msgr2=0x557149b594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 -- 172.21.15.146:0/984685401 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557149a7e3f0 con 0x557149b59b30 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc69a800640 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b56df0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc694034040 tx=0x7fc694030e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca871aca2dc67c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.024+0000 7fc67fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc67fe00640 1 -- 172.21.15.146:0/984685401 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694036050 con 0x557149b59b30 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc67fe00640 1 -- 172.21.15.146:0/984685401 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc6940303a0 con 0x557149b59b30 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc694030c20 con 0x557149b59b30 2024-04-24T08:27:57.034 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc67fe00640 1 -- 172.21.15.146:0/984685401 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc694040040 con 0x557149b59b30 2024-04-24T08:27:57.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc6a3be0740 10 client.26716 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:57.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5571497c63f0 con 0x557149b59b30 2024-04-24T08:27:57.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc67fe00640 1 -- 172.21.15.146:0/984685401 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc694030c20 con 0x557149b59b30 2024-04-24T08:27:57.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.025+0000 7fc67fe00640 1 -- 172.21.15.146:0/984685401 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc6940046e0 con 0x557149b59b30 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc6a3be0740 10 client.26716 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc6a3be0740 10 client.26716 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc6a3be0740 20 client.26716 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149b69ab0 0x557149b6bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc6a3be0740 4 client.26716 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557149ac1bf0 con 0x557149b69ab0 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc699e00640 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149b69ab0 0x557149b6bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc699e00640 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149b69ab0 0x557149b6bea0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=37d021ffa390908d server_cookie=92a63f5cf67d49da in_seq=0 out_seq=0 2024-04-24T08:27:57.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.026+0000 7fc67fe00640 10 client.26716 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.027+0000 7fc67fe00640 1 -- 172.21.15.146:0/984685401 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557149ac1bf0 con 0x557149b69ab0 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.027+0000 7fc67fe00640 10 client.26716 handle_command_reply: tid=0 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.027+0000 7fc6a3be0740 1 client.26716 shutdown 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.027+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 msgr2=0x557149b56df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.027+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b56df0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc694034040 tx=0x7fc694030e60 comp rx=0 tx=0).stop 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 shutdown_connections 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149b69ab0 0x557149b6bea0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b56df0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/984685401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 >> 172.21.15.146:0/984685401 conn(0x557149b4ae60 msgr2=0x557149b50c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc67fe00640 0 client.26716 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:57.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 shutdown_connections 2024-04-24T08:27:57.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 1 -- 172.21.15.146:0/984685401 wait complete. 2024-04-24T08:27:57.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.028+0000 7fc6a3be0740 20 client.26716 trim_cache size 0 max 16384 2024-04-24T08:27:57.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 Processor -- start 2024-04-24T08:27:57.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 -- start start 2024-04-24T08:27:57.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b65820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b59020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b71470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1b2650 con 0x557149b4eb40 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1aff20 con 0x557149b4f460 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1add70 con 0x557149b59b30 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.043+0000 7fc699e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b65820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc69a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b71470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc691400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b59020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc69a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b71470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38774/0 (socket says 172.21.15.146:38774) 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc699e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b65820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43240/0 (socket says 172.21.15.146:43240) 2024-04-24T08:27:57.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc69a800640 1 -- 172.21.15.146:0/798202389 learned_addr learned my addr 172.21.15.146:0/798202389 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc691400640 1 -- 172.21.15.146:0/798202389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 msgr2=0x557149b71470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc691400640 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b71470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc691400640 1 -- 172.21.15.146:0/798202389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 msgr2=0x557149b65820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc691400640 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b65820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc691400640 1 -- 172.21.15.146:0/798202389 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571498907d0 con 0x557149b4f460 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc69a800640 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b71470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc691400640 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b59020 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc6840ce2b0 tx=0x7fc6840bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b421504c1dbadfed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.044+0000 7fc68aa00640 1 -- 172.21.15.146:0/798202389 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684098da0 con 0x557149b4f460 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc68aa00640 1 -- 172.21.15.146:0/798202389 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc68409d200 con 0x557149b4f460 2024-04-24T08:27:57.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc68aa00640 1 -- 172.21.15.146:0/798202389 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68409c460 con 0x557149b4f460 2024-04-24T08:27:57.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 -- 172.21.15.146:0/798202389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 msgr2=0x557149b59020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b59020 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc6840ce2b0 tx=0x7fc6840bd040 comp rx=0 tx=0).stop 2024-04-24T08:27:57.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 -- 172.21.15.146:0/798202389 shutdown_connections 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b59b30 0x557149b71470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b4f460 0x557149b59020 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/798202389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4eb40 0x557149b65820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 -- 172.21.15.146:0/798202389 >> 172.21.15.146:0/798202389 conn(0x557149b4ae60 msgr2=0x557149b49f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 -- 172.21.15.146:0/798202389 shutdown_connections 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.045+0000 7fc6a3be0740 1 -- 172.21.15.146:0/798202389 wait complete. 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 -- start start 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 Processor -- start 2024-04-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b621c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4f460 0x557149b53bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b59b30 0x557149b541c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1ae3e0 con 0x557149b4f460 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1af850 con 0x557149b59b30 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1aeab0 con 0x557149b4eb40 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.046+0000 7fc6a3be0740 10 client.0 _ll_register_callbacks cb 0x7fc6840d4560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4f460 0x557149b53bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4f460 0x557149b53bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43256/0 (socket says 172.21.15.146:43256) 2024-04-24T08:27:57.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 -- 172.21.15.146:0/3830162317 learned_addr learned my addr 172.21.15.146:0/3830162317 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc699e00640 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc69a800640 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b59b30 0x557149b541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 -- 172.21.15.146:0/3830162317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 msgr2=0x557149b621c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b621c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 -- 172.21.15.146:0/3830162317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b59b30 msgr2=0x557149b541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b59b30 0x557149b541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 -- 172.21.15.146:0/3830162317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557149b5a7f0 con 0x557149b4f460 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc691400640 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4f460 0x557149b53bb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc684009230 tx=0x7fc68409c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bbcd82da73d51f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc67fe00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.047+0000 7fc67fe00640 1 -- 172.21.15.146:0/3830162317 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6840b5020 con 0x557149b4f460 2024-04-24T08:27:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc67fe00640 1 -- 172.21.15.146:0/3830162317 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fc6840ce530 con 0x557149b4f460 2024-04-24T08:27:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc68409d200 con 0x557149b4f460 2024-04-24T08:27:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc67fe00640 1 -- 172.21.15.146:0/3830162317 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc68409c640 con 0x557149b4f460 2024-04-24T08:27:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 10 client.16593 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557149bcff30 con 0x557149b4f460 2024-04-24T08:27:57.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc67fe00640 1 -- 172.21.15.146:0/3830162317 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc68409d200 con 0x557149b4f460 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc67fe00640 1 -- 172.21.15.146:0/3830162317 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc684098850 con 0x557149b4f460 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 10 client.16593 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 10 client.16593 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 20 client.16593 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149bd03e0 0x557149bd2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 4 client.16593 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.048+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557149abba60 con 0x557149bd03e0 2024-04-24T08:27:57.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.049+0000 7fc699e00640 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149bd03e0 0x557149bd2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.049+0000 7fc699e00640 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149bd03e0 0x557149bd2800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=daf49bfd768b40dc server_cookie=6e4b2e08da4ba2fb in_seq=0 out_seq=0 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.049+0000 7fc67fe00640 10 client.16593 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.049+0000 7fc67fe00640 1 -- 172.21.15.146:0/3830162317 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557149abba60 con 0x557149bd03e0 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.049+0000 7fc67fe00640 10 client.16593 handle_command_reply: tid=0 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.049+0000 7fc6a3be0740 1 client.16593 shutdown 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4f460 msgr2=0x557149b53bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4f460 0x557149b53bb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc684009230 tx=0x7fc68409c4f0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 shutdown_connections 2024-04-24T08:27:57.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x557149bd03e0 0x557149bd2800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557149b59b30 0x557149b541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557149b4f460 0x557149b53bb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc67fe00640 0 client.16593 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 --2- 172.21.15.146:0/3830162317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557149b4eb40 0x557149b621c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 >> 172.21.15.146:0/3830162317 conn(0x557149b4ae60 msgr2=0x557149b502d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 shutdown_connections 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 1 -- 172.21.15.146:0/3830162317 wait complete. 2024-04-24T08:27:57.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.050+0000 7fc6a3be0740 20 client.16593 trim_cache size 0 max 16384 2024-04-24T08:27:57.060 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:27:57.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc6800a3d20 msgr2=0x7fc6800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc6800a3d20 0x7fc6800a61e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x557149b50100 tx=0x7fc68c005f90 comp rx=0 tx=0).stop 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c074cf0 msgr2=0x7fc69c1a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c074cf0 0x7fc69c1a0b70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc68400e9c0 tx=0x7fc68400ee80 comp rx=0 tx=0).stop 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 shutdown_connections 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc6800a3d20 0x7fc6800a61e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc69c074cf0 0x7fc69c1a0b70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc69c10e740 0x7fc69c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 --2- 172.21.15.146:0/2590307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc69c0743d0 0x7fc69c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 >> 172.21.15.146:0/2590307041 conn(0x7fc69c0fd9d0 msgr2=0x7fc69c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 shutdown_connections 2024-04-24T08:27:57.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.051+0000 7fc69b200640 1 -- 172.21.15.146:0/2590307041 wait complete. 2024-04-24T08:27:57.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:56 smithi177 ceph-mon[33046]: pgmap v205: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:27:57.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:56 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/710836896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:27:57.071 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:27:57.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.271+0000 7fb31ce00640 1 -- 172.21.15.146:0/724273328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318102110 msgr2=0x7fb3181024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.271+0000 7fb31ce00640 1 --2- 172.21.15.146:0/724273328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318102110 0x7fb3181024f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb308009900 tx=0x7fb30802a400 comp rx=0 tx=0).stop 2024-04-24T08:27:57.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.272+0000 7fb31ce00640 1 -- 172.21.15.146:0/724273328 shutdown_connections 2024-04-24T08:27:57.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.272+0000 7fb31ce00640 1 --2- 172.21.15.146:0/724273328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3181071e0 0x7fb31810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.272+0000 7fb31ce00640 1 --2- 172.21.15.146:0/724273328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102ac0 0x7fb318106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.272+0000 7fb31ce00640 1 --2- 172.21.15.146:0/724273328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318102110 0x7fb3181024f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.272+0000 7fb31ce00640 1 -- 172.21.15.146:0/724273328 >> 172.21.15.146:0/724273328 conn(0x7fb3180fd9d0 msgr2=0x7fb3180ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.272+0000 7fb31ce00640 1 -- 172.21.15.146:0/724273328 shutdown_connections 2024-04-24T08:27:57.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.272+0000 7fb31ce00640 1 -- 172.21.15.146:0/724273328 wait complete. 2024-04-24T08:27:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.273+0000 7fb31ce00640 1 Processor -- start 2024-04-24T08:27:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.273+0000 7fb31ce00640 1 -- start start 2024-04-24T08:27:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb31ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102110 0x7fb31819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb31ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3181071e0 0x7fb31819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb31ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318198620 0x7fb318198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb31ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181122e0 con 0x7fb3181071e0 2024-04-24T08:27:57.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb31ce00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb318112160 con 0x7fb318102110 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb31ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb318112460 con 0x7fb318198620 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb317400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102110 0x7fb31819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb316a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3181071e0 0x7fb31819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb317400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102110 0x7fb31819e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58530/0 (socket says 172.21.15.146:58530) 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb317400640 1 -- 172.21.15.146:0/1155002159 learned_addr learned my addr 172.21.15.146:0/1155002159 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb317e00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318198620 0x7fb318198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb316a00640 1 -- 172.21.15.146:0/1155002159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318198620 msgr2=0x7fb318198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb316a00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318198620 0x7fb318198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb316a00640 1 -- 172.21.15.146:0/1155002159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102110 msgr2=0x7fb31819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb316a00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102110 0x7fb31819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb316a00640 1 -- 172.21.15.146:0/1155002159 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb318199140 con 0x7fb3181071e0 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb317400640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102110 0x7fb31819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.274+0000 7fb317e00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318198620 0x7fb318198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.275+0000 7fb316a00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3181071e0 0x7fb31819e9b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e9c0 tx=0x7fb30000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.275+0000 7fb30fe00640 1 -- 172.21.15.146:0/1155002159 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30000ccc0 con 0x7fb3181071e0 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.275+0000 7fb30fe00640 1 -- 172.21.15.146:0/1155002159 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb300005020 con 0x7fb3181071e0 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.275+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb318199430 con 0x7fb3181071e0 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.275+0000 7fb30fe00640 1 -- 172.21.15.146:0/1155002159 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300004a00 con 0x7fb3181071e0 2024-04-24T08:27:57.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.275+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3181a5630 con 0x7fb3181071e0 2024-04-24T08:27:57.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.279+0000 7fb30fe00640 1 -- 172.21.15.146:0/1155002159 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fb3000051d0 con 0x7fb3181071e0 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.279+0000 7fb321201740 1 Processor -- start 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.279+0000 7fb321201740 1 -- start start 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.279+0000 7fb321201740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a76b40 0x55c349a76f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.279+0000 7fb321201740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a77460 0x55c349a815f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.279+0000 7fb321201740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a83f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb321201740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c3499dc1d0 con 0x55c349a77460 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb321201740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c349982450 con 0x55c349a76b40 2024-04-24T08:27:57.282 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb321201740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c3499d8190 con 0x55c349a81b30 2024-04-24T08:27:57.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a76b40 0x55c349a76f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a77460 0x55c349a815f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a76b40 0x55c349a76f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58540/0 (socket says 172.21.15.146:58540) 2024-04-24T08:27:57.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 -- 172.21.15.146:0/3686362184 learned_addr learned my addr 172.21.15.146:0/3686362184 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.283 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb316a00640 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a83f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 -- 172.21.15.146:0/3686362184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 msgr2=0x55c349a83f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a83f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 -- 172.21.15.146:0/3686362184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a77460 msgr2=0x55c349a815f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a77460 0x55c349a815f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.280+0000 7fb317e00640 1 -- 172.21.15.146:0/3686362184 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c349a3f310 con 0x55c349a76b40 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.281+0000 7fb317e00640 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a76b40 0x55c349a76f20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb310011790 tx=0x7fb310011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d28ac66bdfaea56 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.281+0000 7fb30fe00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb2fc0a3d20 0x7fb2fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.281+0000 7fb30cc00640 1 -- 172.21.15.146:0/3686362184 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310003c30 con 0x55c349a76b40 2024-04-24T08:27:57.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.281+0000 7fb30cc00640 1 -- 172.21.15.146:0/3686362184 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb31000fe80 con 0x55c349a76b40 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.281+0000 7fb30cc00640 1 -- 172.21.15.146:0/3686362184 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3100193e0 con 0x55c349a76b40 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.281+0000 7fb30fe00640 1 -- 172.21.15.146:0/1155002159 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb3000d07e0 con 0x7fb3181071e0 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.281+0000 7fb317400640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb2fc0a3d20 0x7fb2fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 -- 172.21.15.146:0/3686362184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a76b40 msgr2=0x55c349a76f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a76b40 0x55c349a76f20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb310011790 tx=0x7fb310011c50 comp rx=0 tx=0).stop 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb317400640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb2fc0a3d20 0x7fb2fc0a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb3080096c0 tx=0x7fb3080046c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 -- 172.21.15.146:0/3686362184 shutdown_connections 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a83f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a77460 0x55c349a815f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 --2- 172.21.15.146:0/3686362184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a76b40 0x55c349a76f20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 -- 172.21.15.146:0/3686362184 >> 172.21.15.146:0/3686362184 conn(0x55c349a72e60 msgr2=0x55c349a73240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 -- 172.21.15.146:0/3686362184 shutdown_connections 2024-04-24T08:27:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.282+0000 7fb321201740 1 -- 172.21.15.146:0/3686362184 wait complete. 2024-04-24T08:27:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 -- start start 2024-04-24T08:27:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 Processor -- start 2024-04-24T08:27:57.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a7e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a7ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a7f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c349a849c0 con 0x55c349a76b40 2024-04-24T08:27:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c349995810 con 0x55c349a81b30 2024-04-24T08:27:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.283+0000 7fb321201740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c349a84b40 con 0x55c349a77460 2024-04-24T08:27:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb321201740 10 client.0 _ll_register_callbacks cb 0x55c349963140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:57.291 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a7e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a7e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43294/0 (socket says 172.21.15.146:43294) 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 -- 172.21.15.146:0/933782873 learned_addr learned my addr 172.21.15.146:0/933782873 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb316a00640 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a7ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317e00640 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a7f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 -- 172.21.15.146:0/933782873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 msgr2=0x55c349a7ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a7ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 -- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 msgr2=0x55c349a7f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a7f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.284+0000 7fb317400640 1 -- 172.21.15.146:0/933782873 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3499a63f0 con 0x55c349a76b40 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb317400640 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a7e8f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb308006010 tx=0x7fb308009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7cf2fc6a77d137 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb305600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb305600640 1 -- 172.21.15.146:0/933782873 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308035040 con 0x55c349a76b40 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb305600640 1 -- 172.21.15.146:0/933782873 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb308031bc0 con 0x55c349a76b40 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c3499e9bf0 con 0x55c349a76b40 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb305600640 1 -- 172.21.15.146:0/933782873 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb308031dc0 con 0x55c349a76b40 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb321201740 10 client.16605 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c3496ee3f0 con 0x55c349a76b40 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.285+0000 7fb305600640 1 -- 172.21.15.146:0/933782873 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c3499e9bf0 con 0x55c349a76b40 2024-04-24T08:27:57.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb305600640 1 -- 172.21.15.146:0/933782873 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb308044030 con 0x55c349a76b40 2024-04-24T08:27:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb321201740 10 client.16605 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb321201740 10 client.16605 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb321201740 20 client.16605 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb321201740 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349a91cd0 0x55c349a94120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb321201740 4 client.16605 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:57.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3499e3a60 con 0x55c349a91cd0 2024-04-24T08:27:57.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.286+0000 7fb316a00640 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349a91cd0 0x55c349a94120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.287+0000 7fb316a00640 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349a91cd0 0x55c349a94120 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f7433b44e0d1410 server_cookie=c7558f97e739a6e5 in_seq=0 out_seq=0 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.287+0000 7fb305600640 10 client.16605 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.287+0000 7fb305600640 1 -- 172.21.15.146:0/933782873 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c3499e3a60 con 0x55c349a91cd0 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.287+0000 7fb305600640 10 client.16605 handle_command_reply: tid=0 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.287+0000 7fb321201740 1 client.16605 shutdown 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 msgr2=0x55c349a7e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a7e8f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb308006010 tx=0x7fb308009260 comp rx=0 tx=0).stop 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 shutdown_connections 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349a91cd0 0x55c349a94120 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a7ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a7f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 --2- 172.21.15.146:0/933782873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a7e8f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb305600640 0 client.16605 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:57.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 >> 172.21.15.146:0/933782873 conn(0x55c349a72e60 msgr2=0x55c349a78b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 shutdown_connections 2024-04-24T08:27:57.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 1 -- 172.21.15.146:0/933782873 wait complete. 2024-04-24T08:27:57.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.288+0000 7fb321201740 20 client.16605 trim_cache size 0 max 16384 2024-04-24T08:27:57.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 Processor -- start 2024-04-24T08:27:57.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 -- start start 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a792a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a77460 0x55c349a8d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a7d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181b0420 con 0x55c349a76b40 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181adcf0 con 0x55c349a77460 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb321201740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181abb60 con 0x55c349a81b30 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb317400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a792a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb317400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a792a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43308/0 (socket says 172.21.15.146:43308) 2024-04-24T08:27:57.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.303+0000 7fb317400640 1 -- 172.21.15.146:0/2222418559 learned_addr learned my addr 172.21.15.146:0/2222418559 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb317e00640 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a7d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb316a00640 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a77460 0x55c349a8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb317400640 1 -- 172.21.15.146:0/2222418559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 msgr2=0x55c349a7d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb317400640 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a7d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb317400640 1 -- 172.21.15.146:0/2222418559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a77460 msgr2=0x55c349a8d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb317400640 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a77460 0x55c349a8d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb317400640 1 -- 172.21.15.146:0/2222418559 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c349a77290 con 0x55c349a76b40 2024-04-24T08:27:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb317400640 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a792a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb308042040 tx=0x7fb308003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82eccdfe85df18e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb307e00640 1 -- 172.21.15.146:0/2222418559 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3080319c0 con 0x55c349a76b40 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb307e00640 1 -- 172.21.15.146:0/2222418559 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb30805f070 con 0x55c349a76b40 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.304+0000 7fb307e00640 1 -- 172.21.15.146:0/2222418559 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30805aa00 con 0x55c349a76b40 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 -- 172.21.15.146:0/2222418559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 msgr2=0x55c349a792a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a792a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb308042040 tx=0x7fb308003f20 comp rx=0 tx=0).stop 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 -- 172.21.15.146:0/2222418559 shutdown_connections 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a81b30 0x55c349a7d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a77460 0x55c349a8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 --2- 172.21.15.146:0/2222418559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a792a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 -- 172.21.15.146:0/2222418559 >> 172.21.15.146:0/2222418559 conn(0x55c349a72e60 msgr2=0x55c349a71f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 -- 172.21.15.146:0/2222418559 shutdown_connections 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.305+0000 7fb321201740 1 -- 172.21.15.146:0/2222418559 wait complete. 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 -- start start 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 Processor -- start 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a82a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a82f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a83490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181ac1b0 con 0x55c349a76b40 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181ad620 con 0x55c349a81b30 2024-04-24T08:27:57.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3181ac880 con 0x55c349a77460 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.306+0000 7fb321201740 10 client.0 _ll_register_callbacks cb 0x7fb308009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a82f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb317400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a82a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a82f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38866/0 (socket says 172.21.15.146:38866) 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 -- 172.21.15.146:0/29544909 learned_addr learned my addr 172.21.15.146:0/29544909 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb317e00640 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a83490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 -- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 msgr2=0x55c349a83490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a83490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 -- 172.21.15.146:0/29544909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 msgr2=0x55c349a82a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a82a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 -- 172.21.15.146:0/29544909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3497b87d0 con 0x55c349a77460 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb316a00640 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a82f50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb300099100 tx=0x7fb3000d2c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea4afbbf98838651 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb305600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.307+0000 7fb305600640 1 -- 172.21.15.146:0/29544909 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30001c040 con 0x55c349a77460 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb30805f070 con 0x55c349a77460 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb305600640 1 -- 172.21.15.146:0/29544909 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7fb300005560 con 0x55c349a77460 2024-04-24T08:27:57.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb305600640 1 -- 172.21.15.146:0/29544909 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb30009ce30 con 0x55c349a77460 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 10 client.26740 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c349afc900 con 0x55c349a77460 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb305600640 1 -- 172.21.15.146:0/29544909 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb30805f070 con 0x55c349a77460 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb305600640 1 -- 172.21.15.146:0/29544909 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb3000ce160 con 0x55c349a77460 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 10 client.26740 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 10 client.26740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 20 client.26740 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:57.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349afd0b0 0x55c349aff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 4 client.26740 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:27:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.308+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c349a72cb0 con 0x55c349afd0b0 2024-04-24T08:27:57.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb317400640 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349afd0b0 0x55c349aff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:57.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb317400640 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349afd0b0 0x55c349aff4d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ddd84081cce09f server_cookie=13dd6d4b8cef905a in_seq=0 out_seq=0 2024-04-24T08:27:57.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb305600640 10 client.26740 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:57.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb305600640 1 -- 172.21.15.146:0/29544909 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c349a72cb0 con 0x55c349afd0b0 2024-04-24T08:27:57.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb305600640 10 client.26740 handle_command_reply: tid=0 2024-04-24T08:27:57.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb321201740 1 client.26740 shutdown 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 msgr2=0x55c349a82f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.309+0000 7fb321201740 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a82f50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb300099100 tx=0x7fb3000d2c30 comp rx=0 tx=0).stop 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 shutdown_connections 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c349afd0b0 0x55c349aff4d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c349a77460 0x55c349a82f50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c349a81b30 0x55c349a83490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 --2- 172.21.15.146:0/29544909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c349a76b40 0x55c349a82a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb305600640 0 client.26740 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:57.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 >> 172.21.15.146:0/29544909 conn(0x55c349a72e60 msgr2=0x55c349a716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 shutdown_connections 2024-04-24T08:27:57.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 1 -- 172.21.15.146:0/29544909 wait complete. 2024-04-24T08:27:57.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb321201740 20 client.26740 trim_cache size 0 max 16384 2024-04-24T08:27:57.319 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:27:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb2fc0a3d20 msgr2=0x7fb2fc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.310+0000 7fb31ce00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb2fc0a3d20 0x7fb2fc0a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb3080096c0 tx=0x7fb3080046c0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3181071e0 msgr2=0x7fb31819e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3181071e0 0x7fb31819e9b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e9c0 tx=0x7fb30000ee80 comp rx=0 tx=0).stop 2024-04-24T08:27:57.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 shutdown_connections 2024-04-24T08:27:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb2fc0a3d20 0x7fb2fc0a61e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb318198620 0x7fb318198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3181071e0 0x7fb31819e9b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 --2- 172.21.15.146:0/1155002159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb318102110 0x7fb31819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 >> 172.21.15.146:0/1155002159 conn(0x7fb3180fd9d0 msgr2=0x7fb3180ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 shutdown_connections 2024-04-24T08:27:57.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:57.311+0000 7fb31ce00640 1 -- 172.21.15.146:0/1155002159 wait complete. 2024-04-24T08:27:58.332 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:27:58.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:27:58.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:27:58.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 -- 172.21.15.146:0/3283085000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50101071e0 msgr2=0x7f501010edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 --2- 172.21.15.146:0/3283085000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50101071e0 0x7f501010edb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5008009900 tx=0x7f500802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 -- 172.21.15.146:0/3283085000 shutdown_connections 2024-04-24T08:27:58.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 --2- 172.21.15.146:0/3283085000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50101071e0 0x7f501010edb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 --2- 172.21.15.146:0/3283085000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5010102ac0 0x7f5010106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 --2- 172.21.15.146:0/3283085000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5010102110 0x7f50101024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 -- 172.21.15.146:0/3283085000 >> 172.21.15.146:0/3283085000 conn(0x7f50100fd9d0 msgr2=0x7f50100ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 -- 172.21.15.146:0/3283085000 shutdown_connections 2024-04-24T08:27:58.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.553+0000 7f5016e00640 1 -- 172.21.15.146:0/3283085000 wait complete. 2024-04-24T08:27:58.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.554+0000 7f5016e00640 1 Processor -- start 2024-04-24T08:27:58.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.554+0000 7f5016e00640 1 -- start start 2024-04-24T08:27:58.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.554+0000 7f5016e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5010102110 0x7f50101a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.558 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50101071e0 0x7f50101a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f501019a840 0x7f501019acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5010112280 con 0x7f501019a840 2024-04-24T08:27:58.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5010112100 con 0x7f50101071e0 2024-04-24T08:27:58.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5010112400 con 0x7f5010102110 2024-04-24T08:27:58.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f501019a840 0x7f501019acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5010102110 0x7f50101a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f501019a840 0x7f501019acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43332/0 (socket says 172.21.15.146:43332) 2024-04-24T08:27:58.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5010102110 0x7f50101a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38886/0 (socket says 172.21.15.146:38886) 2024-04-24T08:27:58.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5016400640 1 -- 172.21.15.146:0/3522315899 learned_addr learned my addr 172.21.15.146:0/3522315899 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015000640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50101071e0 0x7f50101a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015a00640 1 -- 172.21.15.146:0/3522315899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50101071e0 msgr2=0x7f50101a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015a00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50101071e0 0x7f50101a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015a00640 1 -- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f501019a840 msgr2=0x7f501019acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015a00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f501019a840 0x7f501019acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.555+0000 7f5015a00640 1 -- 172.21.15.146:0/3522315899 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501019b3d0 con 0x7f5010102110 2024-04-24T08:27:58.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.556+0000 7f5015a00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5010102110 0x7f50101a06d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f500400eb00 tx=0x7f500400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.556+0000 7f500e000640 1 -- 172.21.15.146:0/3522315899 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004010430 con 0x7f5010102110 2024-04-24T08:27:58.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.556+0000 7f500e000640 1 -- 172.21.15.146:0/3522315899 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5004010ab0 con 0x7f5010102110 2024-04-24T08:27:58.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.556+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501019b6c0 con 0x7f5010102110 2024-04-24T08:27:58.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.556+0000 7f500e000640 1 -- 172.21.15.146:0/3522315899 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004017bb0 con 0x7f5010102110 2024-04-24T08:27:58.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.556+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50101a78f0 con 0x7f5010102110 2024-04-24T08:27:58.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.559+0000 7f501b187740 1 Processor -- start 2024-04-24T08:27:58.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.559+0000 7f501b187740 1 -- start start 2024-04-24T08:27:58.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f501b187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231b270 0x5622f231b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f501b187740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2325d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f501b187740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f2326260 0x5622f2328650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f501b187740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5622f2280e70 con 0x5622f2326260 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f501b187740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5622f2227380 con 0x5622f231bb90 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f501b187740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5622f227ce30 con 0x5622f231b270 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f500e000640 1 -- 172.21.15.146:0/3522315899 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f5004010c60 con 0x7f5010102110 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f5015000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f2326260 0x5622f2328650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f5015000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f2326260 0x5622f2328650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43340/0 (socket says 172.21.15.146:43340) 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f5015000640 1 -- 172.21.15.146:0/2713175685 learned_addr learned my addr 172.21.15.146:0/2713175685 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f5016400640 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231b270 0x5622f231b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.560+0000 7f5015a00640 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2325d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5016400640 1 -- 172.21.15.146:0/2713175685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 msgr2=0x5622f2325d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5016400640 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2325d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5016400640 1 -- 172.21.15.146:0/2713175685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f2326260 msgr2=0x5622f2328650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5016400640 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f2326260 0x5622f2328650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5016400640 1 -- 172.21.15.146:0/2713175685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f205b7e0 con 0x5622f231b270 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5015000640 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f2326260 0x5622f2328650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5016400640 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231b270 0x5622f231b650 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5008005ab0 tx=0x7f5008005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=714d85fd6bfb303e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5002a00640 1 -- 172.21.15.146:0/2713175685 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008040070 con 0x5622f231b270 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5002a00640 1 -- 172.21.15.146:0/2713175685 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50080318f0 con 0x5622f231b270 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.561+0000 7f5002a00640 1 -- 172.21.15.146:0/2713175685 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008035040 con 0x5622f231b270 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f500e000640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4ff80a3da0 0x7f4ff80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f500e000640 1 -- 172.21.15.146:0/3522315899 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f50040d10d0 con 0x7f5010102110 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f5015a00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4ff80a3da0 0x7f4ff80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 -- 172.21.15.146:0/2713175685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231b270 msgr2=0x5622f231b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231b270 0x5622f231b650 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5008005ab0 tx=0x7f5008005d50 comp rx=0 tx=0).stop 2024-04-24T08:27:58.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 -- 172.21.15.146:0/2713175685 shutdown_connections 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f2326260 0x5622f2328650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2325d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 --2- 172.21.15.146:0/2713175685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231b270 0x5622f231b650 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 -- 172.21.15.146:0/2713175685 >> 172.21.15.146:0/2713175685 conn(0x5622f2317590 msgr2=0x5622f2317970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 -- 172.21.15.146:0/2713175685 shutdown_connections 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.562+0000 7f501b187740 1 -- 172.21.15.146:0/2713175685 wait complete. 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f5015a00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4ff80a3da0 0x7f4ff80a6260 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f50040ce8e0 tx=0x7f500400cf10 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 -- start start 2024-04-24T08:27:58.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 Processor -- start 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f2323240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2323780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f2326260 0x5622f2323cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5622f2328ff0 con 0x5622f231b270 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5622f223a660 con 0x5622f231bb90 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.563+0000 7f501b187740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5622f2329170 con 0x5622f2326260 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f501b187740 10 client.0 _ll_register_callbacks cb 0x5622f2228aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f2323240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f2323240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43346/0 (socket says 172.21.15.146:43346) 2024-04-24T08:27:58.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 -- 172.21.15.146:0/1971632676 learned_addr learned my addr 172.21.15.146:0/1971632676 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5015000640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2323780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f2326260 0x5622f2323cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 -- 172.21.15.146:0/1971632676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f2326260 msgr2=0x5622f2323cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f2326260 0x5622f2323cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 -- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 msgr2=0x5622f2323780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2323780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5016400640 1 -- 172.21.15.146:0/1971632676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f20bbf70 con 0x5622f231b270 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.564+0000 7f5015000640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2323780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f5016400640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f2323240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5008031490 tx=0x7f5008031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457bb69964346118 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f4ff7400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f4ff7400640 1 -- 172.21.15.146:0/1971632676 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008052060 con 0x5622f231b270 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f4ff7400640 1 -- 172.21.15.146:0/1971632676 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f5008004970 con 0x5622f231b270 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50080318f0 con 0x5622f231b270 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f4ff7400640 1 -- 172.21.15.146:0/1971632676 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5008040070 con 0x5622f231b270 2024-04-24T08:27:58.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f501b187740 10 client.16623 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622f1f53170 con 0x5622f231b270 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.565+0000 7f4ff7400640 1 -- 172.21.15.146:0/1971632676 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50080318f0 con 0x5622f231b270 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f4ff7400640 1 -- 172.21.15.146:0/1971632676 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f500803cb50 con 0x5622f231b270 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f501b187740 10 client.16623 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f501b187740 10 client.16623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f501b187740 20 client.16623 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f501b187740 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f2336460 0x5622f2338850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f501b187740 4 client.16623 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:58.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622f228e890 con 0x5622f2336460 2024-04-24T08:27:58.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.566+0000 7f5015000640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f2336460 0x5622f2338850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.567+0000 7f5015000640 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f2336460 0x5622f2338850 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d04e85dfe537b8e9 server_cookie=c5b42de0d9927fad in_seq=0 out_seq=0 2024-04-24T08:27:58.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.567+0000 7f4ff7400640 10 client.16623 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f4ff7400640 1 -- 172.21.15.146:0/1971632676 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5622f228e890 con 0x5622f2336460 2024-04-24T08:27:58.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f4ff7400640 10 client.16623 handle_command_reply: tid=0 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 client.16623 shutdown 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 msgr2=0x5622f2323240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f2323240 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5008031490 tx=0x7f5008031400 comp rx=0 tx=0).stop 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 shutdown_connections 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f2336460 0x5622f2338850 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f2326260 0x5622f2323cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f231bb90 0x5622f2323780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f4ff7400640 0 client.16623 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 --2- 172.21.15.146:0/1971632676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f2323240 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 >> 172.21.15.146:0/1971632676 conn(0x5622f2317590 msgr2=0x5622f231d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 shutdown_connections 2024-04-24T08:27:58.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.568+0000 7f501b187740 1 -- 172.21.15.146:0/1971632676 wait complete. 2024-04-24T08:27:58.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.569+0000 7f501b187740 20 client.16623 trim_cache size 0 max 16384 2024-04-24T08:27:58.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.588+0000 7f501b187740 1 Processor -- start 2024-04-24T08:27:58.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.588+0000 7f501b187740 1 -- start start 2024-04-24T08:27:58.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.588+0000 7f501b187740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f23229f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f501b187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2331ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f501b187740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f233b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f501b187740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f50101b25b0 con 0x5622f231b270 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f501b187740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50101afe80 con 0x5622f2326260 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f501b187740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f50101adcd0 con 0x5622f231bb90 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f23229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f23229f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43358/0 (socket says 172.21.15.146:43358) 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 -- 172.21.15.146:0/2287404504 learned_addr learned my addr 172.21.15.146:0/2287404504 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5015000640 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2331ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f233b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 -- 172.21.15.146:0/2287404504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 msgr2=0x5622f2331ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2331ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 -- 172.21.15.146:0/2287404504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 msgr2=0x5622f233b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f233b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.589+0000 7f5016400640 1 -- 172.21.15.146:0/2287404504 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f231c090 con 0x5622f231b270 2024-04-24T08:27:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.590+0000 7f5016400640 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f23229f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5008005ec0 tx=0x7f5008005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0b303168e8b70f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.590+0000 7f5002000640 1 -- 172.21.15.146:0/2287404504 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008048070 con 0x5622f231b270 2024-04-24T08:27:58.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.590+0000 7f5002000640 1 -- 172.21.15.146:0/2287404504 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5008049040 con 0x5622f231b270 2024-04-24T08:27:58.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.590+0000 7f5002000640 1 -- 172.21.15.146:0/2287404504 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008052040 con 0x5622f231b270 2024-04-24T08:27:58.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 -- 172.21.15.146:0/2287404504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 msgr2=0x5622f23229f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f23229f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5008005ec0 tx=0x7f5008005d50 comp rx=0 tx=0).stop 2024-04-24T08:27:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 -- 172.21.15.146:0/2287404504 shutdown_connections 2024-04-24T08:27:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2331ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f233b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 --2- 172.21.15.146:0/2287404504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f23229f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 -- 172.21.15.146:0/2287404504 >> 172.21.15.146:0/2287404504 conn(0x5622f2317590 msgr2=0x5622f23313c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 -- 172.21.15.146:0/2287404504 shutdown_connections 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 -- 172.21.15.146:0/2287404504 wait complete. 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 -- start start 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.591+0000 7f501b187740 1 Processor -- start 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f501b187740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f233bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f501b187740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2338fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f501b187740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f23394e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f501b187740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f50101ae340 con 0x5622f231b270 2024-04-24T08:27:58.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f501b187740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f50101af7b0 con 0x5622f2326260 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f501b187740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f50101aea10 con 0x5622f231bb90 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f501b187740 10 client.0 _ll_register_callbacks cb 0x7f4ffc009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f233bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f5015000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2338fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f5016400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f233bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43368/0 (socket says 172.21.15.146:43368) 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f5016400640 1 -- 172.21.15.146:0/2556466740 learned_addr learned my addr 172.21.15.146:0/2556466740 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.592+0000 7f5016400640 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f23394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f5015000640 1 -- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 msgr2=0x5622f23394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f5015000640 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f23394e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f5015000640 1 -- 172.21.15.146:0/2556466740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 msgr2=0x5622f233bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f5015000640 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f233bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f5015000640 1 -- 172.21.15.146:0/2556466740 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f2339cf0 con 0x5622f231bb90 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f5016400640 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f23394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f5015000640 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2338fa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5622f23291d0 tx=0x7f4ffc012ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4b04d087264647d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f4ff7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f4ff7400640 1 -- 172.21.15.146:0/2556466740 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc004220 con 0x5622f231bb90 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5008049040 con 0x5622f231bb90 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f4ff7400640 1 -- 172.21.15.146:0/2556466740 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f4ffc0043c0 con 0x5622f231bb90 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f4ff7400640 1 -- 172.21.15.146:0/2556466740 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4ffc030050 con 0x5622f231bb90 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f501b187740 10 client.26758 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.593+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622f239c6b0 con 0x5622f231bb90 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f4ff7400640 1 -- 172.21.15.146:0/2556466740 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5008049040 con 0x5622f231bb90 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f4ff7400640 1 -- 172.21.15.146:0/2556466740 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4ffc003a80 con 0x5622f231bb90 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f501b187740 10 client.26758 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:58.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f501b187740 10 client.26758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f501b187740 20 client.26758 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f501b187740 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f239ce00 0x5622f239f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f501b187740 4 client.26758 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5622f2288700 con 0x5622f239ce00 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.594+0000 7f5016400640 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f239ce00 0x5622f239f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.595+0000 7f5016400640 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f239ce00 0x5622f239f220 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7b16a26029d7e4f server_cookie=19026e88f94469e8 in_seq=0 out_seq=0 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.595+0000 7f4ff7400640 10 client.26758 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f4ff7400640 1 -- 172.21.15.146:0/2556466740 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5622f2288700 con 0x5622f239ce00 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f4ff7400640 10 client.26758 handle_command_reply: tid=0 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 client.26758 shutdown 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 msgr2=0x5622f2338fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2338fa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5622f23291d0 tx=0x7f4ffc012ae0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 shutdown_connections 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5622f239ce00 0x5622f239f220 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5622f231bb90 0x5622f2338fa0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5622f2326260 0x5622f23394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f4ff7400640 0 client.26758 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 --2- 172.21.15.146:0/2556466740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5622f231b270 0x5622f233bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 >> 172.21.15.146:0/2556466740 conn(0x5622f2317590 msgr2=0x5622f2331c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 shutdown_connections 2024-04-24T08:27:58.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.596+0000 7f501b187740 1 -- 172.21.15.146:0/2556466740 wait complete. 2024-04-24T08:27:58.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f501b187740 20 client.26758 trim_cache size 0 max 16384 2024-04-24T08:27:58.606 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:27:58.607 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:27:58.607 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "1f8b0df2-c0e9-421d-9feb-248321294426", 2024-04-24T08:27:58.607 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:27:58.607 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:27:58.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4ff80a3da0 msgr2=0x7f4ff80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4ff80a3da0 0x7f4ff80a6260 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f50040ce8e0 tx=0x7f500400cf10 comp rx=0 tx=0).stop 2024-04-24T08:27:58.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5010102110 msgr2=0x7f50101a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5010102110 0x7f50101a06d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f500400eb00 tx=0x7f500400c2d0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 shutdown_connections 2024-04-24T08:27:58.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4ff80a3da0 0x7f4ff80a6260 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f501019a840 0x7f501019acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50101071e0 0x7f50101a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 --2- 172.21.15.146:0/3522315899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5010102110 0x7f50101a06d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.597+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 >> 172.21.15.146:0/3522315899 conn(0x7f50100fd9d0 msgr2=0x7f50100fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.598+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 shutdown_connections 2024-04-24T08:27:58.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.598+0000 7f5016e00640 1 -- 172.21.15.146:0/3522315899 wait complete. 2024-04-24T08:27:58.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:27:58.620 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:27:58.677 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:58 smithi146 ceph-mon[27805]: pgmap v206: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:58.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.847+0000 7f76cbe00640 1 -- 172.21.15.146:0/633916873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc0743a0 msgr2=0x7f76cc074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.847+0000 7f76cbe00640 1 --2- 172.21.15.146:0/633916873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc0743a0 0x7f76cc074780 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009900 tx=0x7f76bc02a480 comp rx=0 tx=0).stop 2024-04-24T08:27:58.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.848+0000 7f76cbe00640 1 -- 172.21.15.146:0/633916873 shutdown_connections 2024-04-24T08:27:58.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.848+0000 7f76cbe00640 1 --2- 172.21.15.146:0/633916873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76cc112e90 0x7f76cc115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.848+0000 7f76cbe00640 1 --2- 172.21.15.146:0/633916873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc074cc0 0x7f76cc112950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.848+0000 7f76cbe00640 1 --2- 172.21.15.146:0/633916873 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc0743a0 0x7f76cc074780 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.848+0000 7f76cbe00640 1 -- 172.21.15.146:0/633916873 >> 172.21.15.146:0/633916873 conn(0x7f76cc0fd9d0 msgr2=0x7f76cc0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.848+0000 7f76cbe00640 1 -- 172.21.15.146:0/633916873 shutdown_connections 2024-04-24T08:27:58.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.848+0000 7f76cbe00640 1 -- 172.21.15.146:0/633916873 wait complete. 2024-04-24T08:27:58.850 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.849+0000 7f76cbe00640 1 Processor -- start 2024-04-24T08:27:58.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.850+0000 7f76cbe00640 1 -- start start 2024-04-24T08:27:58.851 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.850+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0743a0 0x7f76cc19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.850+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc074cc0 0x7f76cc19f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.850+0000 7f76cbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76cc112e90 0x7f76cc1a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.850+0000 7f76cbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1188b0 con 0x7f76cc0743a0 2024-04-24T08:27:58.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.850+0000 7f76cbe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc118730 con 0x7f76cc074cc0 2024-04-24T08:27:58.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.850+0000 7f76cbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc118a30 con 0x7f76cc112e90 2024-04-24T08:27:58.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0743a0 0x7f76cc19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.852 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0743a0 0x7f76cc19edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43390/0 (socket says 172.21.15.146:43390) 2024-04-24T08:27:58.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 -- 172.21.15.146:0/1099468121 learned_addr learned my addr 172.21.15.146:0/1099468121 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc074cc0 0x7f76cc19f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.853 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76cb400640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76cc112e90 0x7f76cc1a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 -- 172.21.15.146:0/1099468121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76cc112e90 msgr2=0x7f76cc1a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76cc112e90 0x7f76cc1a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 -- 172.21.15.146:0/1099468121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc074cc0 msgr2=0x7f76cc19f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc074cc0 0x7f76cc19f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76caa00640 1 -- 172.21.15.146:0/1099468121 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc1a3de0 con 0x7f76cc0743a0 2024-04-24T08:27:58.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.851+0000 7f76c3e00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc074cc0 0x7f76cc19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:27:58.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.852+0000 7f76caa00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0743a0 0x7f76cc19edf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f76bc02a9f0 tx=0x7f76bc003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.853+0000 7f76c3400640 1 -- 172.21.15.146:0/1099468121 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc03f070 con 0x7f76cc0743a0 2024-04-24T08:27:58.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.853+0000 7f76c3400640 1 -- 172.21.15.146:0/1099468121 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76bc031c90 con 0x7f76cc0743a0 2024-04-24T08:27:58.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.853+0000 7f76c3400640 1 -- 172.21.15.146:0/1099468121 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc035030 con 0x7f76cc0743a0 2024-04-24T08:27:58.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.853+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76cc1a4070 con 0x7f76cc0743a0 2024-04-24T08:27:58.855 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.853+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76cc1ab960 con 0x7f76cc0743a0 2024-04-24T08:27:58.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.858+0000 7f76d4413740 1 Processor -- start 2024-04-24T08:27:58.859 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.858+0000 7f76d4413740 1 -- start start 2024-04-24T08:27:58.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76d4413740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429fe0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76d4413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429feb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76c3400640 1 -- 172.21.15.146:0/1099468121 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f76bc042070 con 0x7f76cc0743a0 2024-04-24T08:27:58.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76d4413740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x560429fede10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76d4413740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560429f461f0 con 0x560429feba20 2024-04-24T08:27:58.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76d4413740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560429eec350 con 0x560429fe0a30 2024-04-24T08:27:58.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76d4413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560429f421b0 con 0x560429fe1350 2024-04-24T08:27:58.861 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.859+0000 7f76caa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429feb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429fe0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429fe0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58666/0 (socket says 172.21.15.146:58666) 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 -- 172.21.15.146:0/1668873339 learned_addr learned my addr 172.21.15.146:0/1668873339 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 -- 172.21.15.146:0/1668873339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 msgr2=0x560429feb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429feb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 -- 172.21.15.146:0/1668873339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 msgr2=0x560429fede10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76c3e00640 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x560429fede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x560429fede10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.860+0000 7f76cb400640 1 -- 172.21.15.146:0/1668873339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560429f40f50 con 0x560429fe0a30 2024-04-24T08:27:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.861+0000 7f76c3e00640 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x560429fede10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:27:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.861+0000 7f76cb400640 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429fe0e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009f30 tx=0x7f76c40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a5d0e327d2ab143 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.862+0000 7f76bb400640 1 -- 172.21.15.146:0/1668873339 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4004d20 con 0x560429fe0a30 2024-04-24T08:27:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.862+0000 7f76bb400640 1 -- 172.21.15.146:0/1668873339 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4004050 con 0x560429fe0a30 2024-04-24T08:27:58.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.862+0000 7f76bb400640 1 -- 172.21.15.146:0/1668873339 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4005260 con 0x560429fe0a30 2024-04-24T08:27:58.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.863+0000 7f76d4413740 1 -- 172.21.15.146:0/1668873339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 msgr2=0x560429fe0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.863+0000 7f76d4413740 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429fe0e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009f30 tx=0x7f76c40118d0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76c3400640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f76b00a3de0 0x7f76b00a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76d4413740 1 -- 172.21.15.146:0/1668873339 shutdown_connections 2024-04-24T08:27:58.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76d4413740 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429feb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76d4413740 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x560429fede10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76d4413740 1 --2- 172.21.15.146:0/1668873339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429fe0e10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76d4413740 1 -- 172.21.15.146:0/1668873339 >> 172.21.15.146:0/1668873339 conn(0x560429fdcd50 msgr2=0x560429fdd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76c3400640 1 -- 172.21.15.146:0/1099468121 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f76bc0f0e40 con 0x7f76cc0743a0 2024-04-24T08:27:58.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76d4413740 1 -- 172.21.15.146:0/1668873339 shutdown_connections 2024-04-24T08:27:58.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76cb400640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f76b00a3de0 0x7f76b00a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.864+0000 7f76d4413740 1 -- 172.21.15.146:0/1668873339 wait complete. 2024-04-24T08:27:58.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.865+0000 7f76d4413740 1 -- start start 2024-04-24T08:27:58.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.865+0000 7f76d4413740 1 Processor -- start 2024-04-24T08:27:58.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76d4413740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429fe89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76d4413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76d4413740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429feba20 0x560429fe9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76d4413740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560429fee810 con 0x560429fe0a30 2024-04-24T08:27:58.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76d4413740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560429eff690 con 0x560429feba20 2024-04-24T08:27:58.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76d4413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560429fee990 con 0x560429fe1350 2024-04-24T08:27:58.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76cb400640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f76b00a3de0 0x7f76b00a62a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f76c40049d0 tx=0x7f76c40096f0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76d4413740 10 client.0 _ll_register_callbacks cb 0x560429f3b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:58.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38952/0 (socket says 172.21.15.146:38952) 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76c3e00640 1 -- 172.21.15.146:0/3139068527 learned_addr learned my addr 172.21.15.146:0/3139068527 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.867+0000 7f76caa00640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429fe89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.868+0000 7f76cb400640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429feba20 0x560429fe9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.868+0000 7f76c3e00640 1 -- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429feba20 msgr2=0x560429fe9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.868+0000 7f76c3e00640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429feba20 0x560429fe9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.868+0000 7f76c3e00640 1 -- 172.21.15.146:0/3139068527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 msgr2=0x560429fe89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.869+0000 7f76c3e00640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429fe89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.869+0000 7f76c3e00640 1 -- 172.21.15.146:0/3139068527 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560429efe920 con 0x560429fe1350 2024-04-24T08:27:58.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.869+0000 7f76caa00640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429fe89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:27:58.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.869+0000 7f76c3e00640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe8ee0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f76b4011690 tx=0x7f76b4011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34f014e428176f81 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.869+0000 7f76afe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:58.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.870+0000 7f76afe00640 1 -- 172.21.15.146:0/3139068527 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4018070 con 0x560429fe1350 2024-04-24T08:27:58.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.870+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76c4004050 con 0x560429fe1350 2024-04-24T08:27:58.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.870+0000 7f76afe00640 1 -- 172.21.15.146:0/3139068527 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f76b400fb40 con 0x560429fe1350 2024-04-24T08:27:58.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.870+0000 7f76afe00640 1 -- 172.21.15.146:0/3139068527 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f76b400fd40 con 0x560429fe1350 2024-04-24T08:27:58.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.870+0000 7f76d4413740 10 client.26770 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:58.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.870+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560429d227d0 con 0x560429fe1350 2024-04-24T08:27:58.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76afe00640 1 -- 172.21.15.146:0/3139068527 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4004050 con 0x560429fe1350 2024-04-24T08:27:58.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76afe00640 1 -- 172.21.15.146:0/3139068527 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f76b40124d0 con 0x560429fe1350 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76d4413740 10 client.26770 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76d4413740 10 client.26770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76d4413740 20 client.26770 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76d4413740 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560429ffbe10 0x560429ffe250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76d4413740 4 client.26770 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.871+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560429f53c10 con 0x560429ffbe10 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.872+0000 7f76caa00640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560429ffbe10 0x560429ffe250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.873+0000 7f76caa00640 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560429ffbe10 0x560429ffe250 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b23020bea2adb688 server_cookie=bc844aaeb077f226 in_seq=0 out_seq=0 2024-04-24T08:27:58.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.873+0000 7f76afe00640 10 client.26770 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.874+0000 7f76afe00640 1 -- 172.21.15.146:0/3139068527 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560429f53c10 con 0x560429ffbe10 2024-04-24T08:27:58.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.874+0000 7f76afe00640 10 client.26770 handle_command_reply: tid=0 2024-04-24T08:27:58.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 client.26770 shutdown 2024-04-24T08:27:58.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 msgr2=0x560429fe8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe8ee0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f76b4011690 tx=0x7f76b4011b50 comp rx=0 tx=0).stop 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 shutdown_connections 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560429ffbe10 0x560429ffe250 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe8ee0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429feba20 0x560429fe9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 --2- 172.21.15.146:0/3139068527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429fe89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 >> 172.21.15.146:0/3139068527 conn(0x560429fdcd50 msgr2=0x560429fe2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.875+0000 7f76afe00640 0 client.26770 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.876+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 shutdown_connections 2024-04-24T08:27:58.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.876+0000 7f76d4413740 1 -- 172.21.15.146:0/3139068527 wait complete. 2024-04-24T08:27:58.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.876+0000 7f76d4413740 20 client.26770 trim_cache size 0 max 16384 2024-04-24T08:27:58.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.911+0000 7f76d4413740 1 Processor -- start 2024-04-24T08:27:58.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76d4413740 1 -- start start 2024-04-24T08:27:58.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76d4413740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429ff7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76d4413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76d4413740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x56042a003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76d4413740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1b6a90 con 0x560429feba20 2024-04-24T08:27:58.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76d4413740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1b4360 con 0x560429fe0a30 2024-04-24T08:27:58.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76d4413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1b21b0 con 0x560429fe1350 2024-04-24T08:27:58.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.912+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:38960/0 (socket says 172.21.15.146:38960) 2024-04-24T08:27:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76c3e00640 1 -- 172.21.15.146:0/434847040 learned_addr learned my addr 172.21.15.146:0/434847040 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76caa00640 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429ff7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76c3e00640 1 -- 172.21.15.146:0/434847040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 msgr2=0x560429ff7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76c3e00640 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429ff7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76c3e00640 1 -- 172.21.15.146:0/434847040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 msgr2=0x56042a003b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76c3e00640 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x56042a003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76c3e00640 1 -- 172.21.15.146:0/434847040 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560429c583f0 con 0x560429fe1350 2024-04-24T08:27:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.913+0000 7f76c3e00640 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe32a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f76b4011660 tx=0x7f76b4005900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e250af2fb4fbac82 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.914+0000 7f76baa00640 1 -- 172.21.15.146:0/434847040 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4034070 con 0x560429fe1350 2024-04-24T08:27:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.914+0000 7f76baa00640 1 -- 172.21.15.146:0/434847040 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76b4005520 con 0x560429fe1350 2024-04-24T08:27:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.914+0000 7f76baa00640 1 -- 172.21.15.146:0/434847040 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4018040 con 0x560429fe1350 2024-04-24T08:27:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 -- 172.21.15.146:0/434847040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 msgr2=0x560429fe32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe32a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f76b4011660 tx=0x7f76b4005900 comp rx=0 tx=0).stop 2024-04-24T08:27:58.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 -- 172.21.15.146:0/434847040 shutdown_connections 2024-04-24T08:27:58.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429fe1350 0x560429fe32a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429feba20 0x56042a003b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 --2- 172.21.15.146:0/434847040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe0a30 0x560429ff7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 -- 172.21.15.146:0/434847040 >> 172.21.15.146:0/434847040 conn(0x560429fdcd50 msgr2=0x560429fe91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 -- 172.21.15.146:0/434847040 shutdown_connections 2024-04-24T08:27:58.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.915+0000 7f76d4413740 1 -- 172.21.15.146:0/434847040 wait complete. 2024-04-24T08:27:58.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.916+0000 7f76d4413740 1 -- start start 2024-04-24T08:27:58.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.916+0000 7f76d4413740 1 Processor -- start 2024-04-24T08:27:58.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.917+0000 7f76d4413740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429ff3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.917+0000 7f76d4413740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe1350 0x560429fef220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.917+0000 7f76d4413740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429feba20 0x560429fef760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.917+0000 7f76d4413740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1b2820 con 0x560429fe0a30 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.917+0000 7f76d4413740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1b3c90 con 0x560429fe1350 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.917+0000 7f76d4413740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1b2ef0 con 0x560429feba20 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.917+0000 7f76d4413740 10 client.0 _ll_register_callbacks cb 0x7f76b400ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.918+0000 7f76caa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429ff3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.918+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe1350 0x560429fef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.918+0000 7f76cb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429feba20 0x560429fef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.918+0000 7f76caa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429ff3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43416/0 (socket says 172.21.15.146:43416) 2024-04-24T08:27:58.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.918+0000 7f76caa00640 1 -- 172.21.15.146:0/1586813727 learned_addr learned my addr 172.21.15.146:0/1586813727 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:27:58.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76cb400640 1 -- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe1350 msgr2=0x560429fef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76cb400640 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe1350 0x560429fef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76cb400640 1 -- 172.21.15.146:0/1586813727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 msgr2=0x560429ff3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76cb400640 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429ff3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76cb400640 1 -- 172.21.15.146:0/1586813727 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560429c180f0 con 0x560429feba20 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76c3e00640 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe1350 0x560429fef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76cb400640 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429feba20 0x560429fef760 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009f00 tx=0x7f76c4000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dac76e6ecb2c29fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.919+0000 7f76afe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.920+0000 7f76afe00640 1 -- 172.21.15.146:0/1586813727 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4036020 con 0x560429feba20 2024-04-24T08:27:58.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.920+0000 7f76afe00640 1 -- 172.21.15.146:0/1586813727 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8498+0+0 (secure 0 0 0) 0x7f76c4005460 con 0x560429feba20 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.920+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76b4005520 con 0x560429feba20 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76afe00640 1 -- 172.21.15.146:0/1586813727 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f76c4005060 con 0x560429feba20 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 10 client.26776 fetch_fsmap learned FSMap version 26 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560429f0b810 con 0x560429feba20 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76afe00640 1 -- 172.21.15.146:0/1586813727 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76b4005520 con 0x560429feba20 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76afe00640 1 -- 172.21.15.146:0/1586813727 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f76c402f070 con 0x560429feba20 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 10 client.26776 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:27:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 10 client.26776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:27:58.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 20 client.26776 populate_metadata read hostname 'smithi146' 2024-04-24T08:27:58.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56042a0623a0 0x56042a064790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:27:58.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 4 client.26776 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:27:58.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.921+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560429f4da80 con 0x56042a0623a0 2024-04-24T08:27:58.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.922+0000 7f76caa00640 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56042a0623a0 0x56042a064790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:27:58.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.923+0000 7f76caa00640 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56042a0623a0 0x56042a064790 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3124ae97fa2df0d server_cookie=c2ba5170218b329b in_seq=0 out_seq=0 2024-04-24T08:27:58.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.923+0000 7f76afe00640 10 client.26776 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.923+0000 7f76afe00640 1 -- 172.21.15.146:0/1586813727 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x560429f4da80 con 0x56042a0623a0 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76afe00640 10 client.26776 handle_command_reply: tid=0 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 client.26776 shutdown 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429feba20 msgr2=0x560429fef760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429feba20 0x560429fef760 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009f00 tx=0x7f76c4000f40 comp rx=0 tx=0).stop 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 shutdown_connections 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56042a0623a0 0x56042a064790 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560429feba20 0x560429fef760 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560429fe1350 0x560429fef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 --2- 172.21.15.146:0/1586813727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560429fe0a30 0x560429ff3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 >> 172.21.15.146:0/1586813727 conn(0x560429fdcd50 msgr2=0x560429fe8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76afe00640 0 client.26776 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:27:58.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.924+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 shutdown_connections 2024-04-24T08:27:58.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.925+0000 7f76d4413740 1 -- 172.21.15.146:0/1586813727 wait complete. 2024-04-24T08:27:58.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.925+0000 7f76d4413740 20 client.26776 trim_cache size 0 max 16384 2024-04-24T08:27:58.929 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:27:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "scrub active (41 inodes in the stack)", 2024-04-24T08:27:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:27:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: "1f8b0df2-c0e9-421d-9feb-248321294426": { 2024-04-24T08:27:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:27:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "1f8b0df2-c0e9-421d-9feb-248321294426", 2024-04-24T08:27:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:27:58.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:27:58.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:27:58.930 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:27:58.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f76b00a3de0 msgr2=0x7f76b00a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f76b00a3de0 0x7f76b00a62a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f76c40049d0 tx=0x7f76c40096f0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0743a0 msgr2=0x7f76cc19edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0743a0 0x7f76cc19edf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f76bc02a9f0 tx=0x7f76bc003b90 comp rx=0 tx=0).stop 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 shutdown_connections 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f76b00a3de0 0x7f76b00a62a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76cc112e90 0x7f76cc1a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f76cc074cc0 0x7f76cc19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 --2- 172.21.15.146:0/1099468121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76cc0743a0 0x7f76cc19edf0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 >> 172.21.15.146:0/1099468121 conn(0x7f76cc0fd9d0 msgr2=0x7f76cc0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:27:58.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 shutdown_connections 2024-04-24T08:27:58.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:27:58.926+0000 7f76cbe00640 1 -- 172.21.15.146:0/1099468121 wait complete. 2024-04-24T08:27:58.951 INFO:tasks.cephfs.filesystem:scrub status for tag:1f8b0df2-c0e9-421d-9feb-248321294426 - {'path': '/', 'tag': '1f8b0df2-c0e9-421d-9feb-248321294426', 'options': 'recursive,force'} 2024-04-24T08:27:58.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:58 smithi153 ceph-mon[32746]: pgmap v206: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:59.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:58 smithi177 ceph-mon[33046]: pgmap v206: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:27:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:27:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:27:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:27:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:27:59.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:59.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:27:59 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:59.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:27:59.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:59.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:27:59.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:27:59.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:59.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:27:59.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:27:59.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:27:59 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:28:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:28:00.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:28:00.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:00.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:28:00.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:00.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:27:59 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:00.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:00 smithi153 ceph-mon[32746]: pgmap v207: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:00.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:00 smithi146 ceph-mon[27805]: pgmap v207: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:01.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:00 smithi177 ceph-mon[33046]: pgmap v207: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:02.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:02 smithi153 ceph-mon[32746]: pgmap v208: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:02.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:02 smithi146 ceph-mon[27805]: pgmap v208: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:03.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:02 smithi177 ceph-mon[33046]: pgmap v208: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:04 smithi153 ceph-mon[32746]: pgmap v209: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:28:04.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:04 smithi146 ceph-mon[27805]: pgmap v209: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:28:05.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:04 smithi177 ceph-mon[33046]: pgmap v209: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:28:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:06 smithi153 ceph-mon[32746]: pgmap v210: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:28:06.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:06 smithi146 ceph-mon[27805]: pgmap v210: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:28:07.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:06 smithi177 ceph-mon[33046]: pgmap v210: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:28:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:07 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:28:07.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:07 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:28:07.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:07 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:28:07.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:07 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:28:08.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:07 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:28:08.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:07 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:28:08.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:08 smithi153 ceph-mon[32746]: pgmap v211: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:28:08.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:08 smithi153 ceph-mon[32746]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T08:28:08.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:08 smithi146 ceph-mon[27805]: pgmap v211: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:28:08.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:08 smithi146 ceph-mon[27805]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T08:28:09.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:08 smithi177 ceph-mon[33046]: pgmap v211: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:28:09.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:08 smithi177 ceph-mon[33046]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T08:28:10.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:10 smithi153 ceph-mon[32746]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:28:10.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:10 smithi146 ceph-mon[27805]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:28:11.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:10 smithi177 ceph-mon[33046]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:28:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:12 smithi153 ceph-mon[32746]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:28:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:12 smithi146 ceph-mon[27805]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:28:13.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:12 smithi177 ceph-mon[33046]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:28:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:14 smithi153 ceph-mon[32746]: pgmap v215: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:14.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:14 smithi146 ceph-mon[27805]: pgmap v215: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:15.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:14 smithi177 ceph-mon[33046]: pgmap v215: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:16 smithi153 ceph-mon[32746]: pgmap v216: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:16 smithi153 ceph-mon[32746]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T08:28:17.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:16 smithi146 ceph-mon[27805]: pgmap v216: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:17.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:16 smithi146 ceph-mon[27805]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T08:28:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:16 smithi177 ceph-mon[33046]: pgmap v216: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:16 smithi177 ceph-mon[33046]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T08:28:19.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:18 smithi153 ceph-mon[32746]: pgmap v218: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2024-04-24T08:28:19.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:18 smithi146 ceph-mon[27805]: pgmap v218: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2024-04-24T08:28:19.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:18 smithi177 ceph-mon[33046]: pgmap v218: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 212 B/s wr, 6 op/s 2024-04-24T08:28:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:20 smithi153 ceph-mon[32746]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:21.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:20 smithi146 ceph-mon[27805]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:20 smithi177 ceph-mon[33046]: pgmap v219: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:23.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:22 smithi153 ceph-mon[32746]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:23.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:22 smithi146 ceph-mon[27805]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:23.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:22 smithi177 ceph-mon[33046]: pgmap v220: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:28:25.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:24 smithi153 ceph-mon[32746]: pgmap v221: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:28:25.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:24 smithi146 ceph-mon[27805]: pgmap v221: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:28:25.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:24 smithi177 ceph-mon[33046]: pgmap v221: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:28:27.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:26 smithi153 ceph-mon[32746]: pgmap v222: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:28:27.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:26 smithi146 ceph-mon[27805]: pgmap v222: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:28:27.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:26 smithi177 ceph-mon[33046]: pgmap v222: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:28:28.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:28:28.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:28:29.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 -- 172.21.15.146:0/3312138632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 msgr2=0x7f74a41153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 --2- 172.21.15.146:0/3312138632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 0x7f74a41153b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f749c009cd0 tx=0x7f749c02a8f0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 -- 172.21.15.146:0/3312138632 shutdown_connections 2024-04-24T08:28:29.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 --2- 172.21.15.146:0/3312138632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74a4108580 0x7f74a4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 --2- 172.21.15.146:0/3312138632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 0x7f74a41153b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 --2- 172.21.15.146:0/3312138632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a410d500 0x7f74a410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 -- 172.21.15.146:0/3312138632 >> 172.21.15.146:0/3312138632 conn(0x7f74a4077e20 msgr2=0x7f74a4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 -- 172.21.15.146:0/3312138632 shutdown_connections 2024-04-24T08:28:29.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.165+0000 7f74aaa00640 1 -- 172.21.15.146:0/3312138632 wait complete. 2024-04-24T08:28:29.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.166+0000 7f74aaa00640 1 Processor -- start 2024-04-24T08:28:29.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.166+0000 7f74aaa00640 1 -- start start 2024-04-24T08:28:29.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.166+0000 7f74aaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74a4108580 0x7f74a41a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 0x7f74a41a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aaa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a410d500 0x7f74a419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aaa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41188b0 con 0x7f74a4108f40 2024-04-24T08:28:29.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aaa00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4118730 con 0x7f74a410d500 2024-04-24T08:28:29.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aaa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4118a30 con 0x7f74a4108580 2024-04-24T08:28:29.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aa000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a410d500 0x7f74a419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 0x7f74a41a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74a9600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74a4108580 0x7f74a41a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aa000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a410d500 0x7f74a419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37472/0 (socket says 172.21.15.146:37472) 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74aa000640 1 -- 172.21.15.146:0/1354897618 learned_addr learned my addr 172.21.15.146:0/1354897618 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74a9600640 1 -- 172.21.15.146:0/1354897618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a410d500 msgr2=0x7f74a419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74a9600640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a410d500 0x7f74a419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74a9600640 1 -- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 msgr2=0x7f74a41a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74a9600640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 0x7f74a41a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.167+0000 7f74a9600640 1 -- 172.21.15.146:0/1354897618 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a419f250 con 0x7f74a4108580 2024-04-24T08:28:29.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.168+0000 7f74a9600640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74a4108580 0x7f74a41a4920 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f749800e9c0 tx=0x7f749800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.168+0000 7f74a2000640 1 -- 172.21.15.146:0/1354897618 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749800cd80 con 0x7f74a4108580 2024-04-24T08:28:29.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.168+0000 7f74a2000640 1 -- 172.21.15.146:0/1354897618 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7498010910 con 0x7f74a4108580 2024-04-24T08:28:29.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.168+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a419f540 con 0x7f74a4108580 2024-04-24T08:28:29.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.168+0000 7f74a2000640 1 -- 172.21.15.146:0/1354897618 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498018980 con 0x7f74a4108580 2024-04-24T08:28:29.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.168+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a41ab7e0 con 0x7f74a4108580 2024-04-24T08:28:29.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.171+0000 7f74aef97740 1 Processor -- start 2024-04-24T08:28:29.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.171+0000 7f74aef97740 1 -- start start 2024-04-24T08:28:29.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.171+0000 7f74a2000640 1 -- 172.21.15.146:0/1354897618 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f749801f020 con 0x7f74a4108580 2024-04-24T08:28:29.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.171+0000 7f74aef97740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aef97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c8350 0x55c9855d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aef97740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aef97740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c98552d1f0 con 0x55c9855c7a30 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aef97740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c9854d3350 con 0x55c9855d2a20 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aef97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c9855291b0 con 0x55c9855c8350 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855c7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49340/0 (socket says 172.21.15.146:49340) 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 -- 172.21.15.146:0/2032029058 learned_addr learned my addr 172.21.15.146:0/2032029058 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74a9600640 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c8350 0x55c9855d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 -- 172.21.15.146:0/2032029058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c8350 msgr2=0x55c9855d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c8350 0x55c9855d24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 -- 172.21.15.146:0/2032029058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 msgr2=0x55c9855d4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74aa000640 1 -- 172.21.15.146:0/2032029058 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c985527f50 con 0x55c9855c7a30 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.172+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.173+0000 7f74aa000640 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855c7e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f749c009430 tx=0x7f749c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1e4efe03323149 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.173+0000 7f7496600640 1 -- 172.21.15.146:0/2032029058 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c030760 con 0x55c9855c7a30 2024-04-24T08:28:29.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.173+0000 7f7496600640 1 -- 172.21.15.146:0/2032029058 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749c036030 con 0x55c9855c7a30 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.173+0000 7f7496600640 1 -- 172.21.15.146:0/2032029058 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c0043e0 con 0x55c9855c7a30 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.173+0000 7f74a2000640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f748c0a3d20 0x7f748c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.173+0000 7f74a2000640 1 -- 172.21.15.146:0/1354897618 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f74980d0800 con 0x7f74a4108580 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74a9600640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f748c0a3d20 0x7f748c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 -- 172.21.15.146:0/2032029058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 msgr2=0x55c9855c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855c7e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f749c009430 tx=0x7f749c009a20 comp rx=0 tx=0).stop 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 -- 172.21.15.146:0/2032029058 shutdown_connections 2024-04-24T08:28:29.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c8350 0x55c9855d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 --2- 172.21.15.146:0/2032029058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855c7e10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 -- 172.21.15.146:0/2032029058 >> 172.21.15.146:0/2032029058 conn(0x55c9855c3d50 msgr2=0x55c9855c4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 -- 172.21.15.146:0/2032029058 shutdown_connections 2024-04-24T08:28:29.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 -- 172.21.15.146:0/2032029058 wait complete. 2024-04-24T08:28:29.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74a9600640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f748c0a3d20 0x7f748c0a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55c9855c8ff0 tx=0x7f749800efb0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.174+0000 7f74aef97740 1 -- start start 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 1 Processor -- start 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c8350 0x55c9855cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c9855d57b0 con 0x55c9855c8350 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c9854e6690 con 0x55c9855d2a20 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c9855d5930 con 0x55c9855c7a30 2024-04-24T08:28:29.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aef97740 10 client.0 _ll_register_callbacks cb 0x55c985522650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:29.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c8350 0x55c9855cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.175+0000 7f74aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:43948/0 (socket says 172.21.15.146:43948) 2024-04-24T08:28:29.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c8350 0x55c9855cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49356/0 (socket says 172.21.15.146:49356) 2024-04-24T08:28:29.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74aa000640 1 -- 172.21.15.146:0/2747301526 learned_addr learned my addr 172.21.15.146:0/2747301526 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74aa000640 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74a8c00640 1 -- 172.21.15.146:0/2747301526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 msgr2=0x55c9855cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74a8c00640 1 -- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 msgr2=0x55c9855d0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74a8c00640 1 -- 172.21.15.146:0/2747301526 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9854e5920 con 0x55c9855c8350 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c8350 0x55c9855cff50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f74a4107fb0 tx=0x7f7490012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be9a08946aed4920 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f748aa00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.176+0000 7f748aa00640 1 -- 172.21.15.146:0/2747301526 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749000fa30 con 0x55c9855c8350 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f748aa00640 1 -- 172.21.15.146:0/2747301526 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f749000fbd0 con 0x55c9855c8350 2024-04-24T08:28:29.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f749c036030 con 0x55c9855c8350 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f748aa00640 1 -- 172.21.15.146:0/2747301526 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f749000fdd0 con 0x55c9855c8350 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 10 client.16653 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9853097d0 con 0x55c9855c8350 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f748aa00640 1 -- 172.21.15.146:0/2747301526 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749c036030 con 0x55c9855c8350 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f748aa00640 1 -- 172.21.15.146:0/2747301526 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7490010430 con 0x55c9855c8350 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 10 client.16653 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:29.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 20 client.16653 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c9855e2ef0 0x55c9855e5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 4 client.16653 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.177+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c98553ac10 con 0x55c9855e2ef0 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.178+0000 7f74aa000640 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c9855e2ef0 0x55c9855e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.178+0000 7f74aa000640 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c9855e2ef0 0x55c9855e5310 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48f164c59c598302 server_cookie=2c6fd29991181dc0 in_seq=0 out_seq=0 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.178+0000 7f748aa00640 10 client.16653 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.179+0000 7f748aa00640 1 -- 172.21.15.146:0/2747301526 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c98553ac10 con 0x55c9855e2ef0 2024-04-24T08:28:29.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.179+0000 7f748aa00640 10 client.16653 handle_command_reply: tid=0 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.179+0000 7f74aef97740 1 client.16653 shutdown 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c8350 msgr2=0x55c9855cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c8350 0x55c9855cff50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f74a4107fb0 tx=0x7f7490012b60 comp rx=0 tx=0).stop 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 shutdown_connections 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c9855e2ef0 0x55c9855e5310 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855d2a20 0x55c9855d0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c8350 0x55c9855cff50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 --2- 172.21.15.146:0/2747301526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 >> 172.21.15.146:0/2747301526 conn(0x55c9855c3d50 msgr2=0x55c9855c9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f748aa00640 0 client.16653 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:29.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 shutdown_connections 2024-04-24T08:28:29.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 1 -- 172.21.15.146:0/2747301526 wait complete. 2024-04-24T08:28:29.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.180+0000 7f74aef97740 20 client.16653 trim_cache size 0 max 16384 2024-04-24T08:28:29.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.199+0000 7f74aef97740 1 Processor -- start 2024-04-24T08:28:29.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.199+0000 7f74aef97740 1 -- start start 2024-04-24T08:28:29.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.199+0000 7f74aef97740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.199+0000 7f74aef97740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855eac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.199+0000 7f74aef97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855d2a20 0x55c9855eb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74aef97740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b6950 con 0x55c9855c7a30 2024-04-24T08:28:29.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74aef97740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b4220 con 0x55c9855c8350 2024-04-24T08:28:29.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74aef97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b2090 con 0x55c9855d2a20 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74aa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855eac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855eac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37512/0 (socket says 172.21.15.146:37512) 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 -- 172.21.15.146:0/2499849713 learned_addr learned my addr 172.21.15.146:0/2499849713 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 -- 172.21.15.146:0/2499849713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855d2a20 msgr2=0x55c9855eb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74aa000640 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855d2a20 0x55c9855eb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855d2a20 0x55c9855eb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 -- 172.21.15.146:0/2499849713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 msgr2=0x55c9855d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.200+0000 7f74a8c00640 1 -- 172.21.15.146:0/2499849713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c98523f3f0 con 0x55c9855c8350 2024-04-24T08:28:29.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.201+0000 7f74a8c00640 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855eac00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f749000ffd0 tx=0x7f7490009d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=def77aff177dc18e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.201+0000 7f7495c00640 1 -- 172.21.15.146:0/2499849713 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490028030 con 0x55c9855c8350 2024-04-24T08:28:29.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.201+0000 7f7495c00640 1 -- 172.21.15.146:0/2499849713 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74900055c0 con 0x55c9855c8350 2024-04-24T08:28:29.207 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.201+0000 7f7495c00640 1 -- 172.21.15.146:0/2499849713 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490010430 con 0x55c9855c8350 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.201+0000 7f74aef97740 1 -- 172.21.15.146:0/2499849713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 msgr2=0x55c9855eac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.201+0000 7f74aef97740 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855eac00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f749000ffd0 tx=0x7f7490009d10 comp rx=0 tx=0).stop 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 -- 172.21.15.146:0/2499849713 shutdown_connections 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855d2a20 0x55c9855eb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855eac00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 --2- 172.21.15.146:0/2499849713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855c7a30 0x55c9855d02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 -- 172.21.15.146:0/2499849713 >> 172.21.15.146:0/2499849713 conn(0x55c9855c3d50 msgr2=0x55c9855c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 -- 172.21.15.146:0/2499849713 shutdown_connections 2024-04-24T08:28:29.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 -- 172.21.15.146:0/2499849713 wait complete. 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 -- start start 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.202+0000 7f74aef97740 1 Processor -- start 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aef97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855d6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aef97740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855d6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aef97740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855d2a20 0x55c985648470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aef97740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b26e0 con 0x55c9855d2a20 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aef97740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b3b50 con 0x55c9855c8350 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aef97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b2db0 con 0x55c9855c7a30 2024-04-24T08:28:29.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aef97740 10 client.0 _ll_register_callbacks cb 0x7f7490009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855d6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855d6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855d6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37528/0 (socket says 172.21.15.146:37528) 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 -- 172.21.15.146:0/1162409465 learned_addr learned my addr 172.21.15.146:0/1162409465 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855d6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:43962/0 (socket says 172.21.15.146:43962) 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74aa000640 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855d2a20 0x55c985648470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 -- 172.21.15.146:0/1162409465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 msgr2=0x55c9855d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855d6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 -- 172.21.15.146:0/1162409465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855d2a20 msgr2=0x55c985648470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855d2a20 0x55c985648470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.203+0000 7f74a8c00640 1 -- 172.21.15.146:0/1162409465 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9851ff0f0 con 0x55c9855c8350 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f74a8c00640 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855d6ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7490012420 tx=0x7f7490010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e525db469c2a3310 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f748aa00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f748aa00640 1 -- 172.21.15.146:0/1162409465 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490028040 con 0x55c9855c8350 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74900055c0 con 0x55c9855c8350 2024-04-24T08:28:29.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f748aa00640 1 -- 172.21.15.146:0/1162409465 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f7490020020 con 0x55c9855c8350 2024-04-24T08:28:29.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f748aa00640 1 -- 172.21.15.146:0/1162409465 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f74900050d0 con 0x55c9855c8350 2024-04-24T08:28:29.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f74aef97740 10 client.26807 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:29.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9854f2810 con 0x55c9855c8350 2024-04-24T08:28:29.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.204+0000 7f748aa00640 1 -- 172.21.15.146:0/1162409465 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74900055c0 con 0x55c9855c8350 2024-04-24T08:28:29.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f748aa00640 1 -- 172.21.15.146:0/1162409465 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7490003ad0 con 0x55c9855c8350 2024-04-24T08:28:29.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aef97740 10 client.26807 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:29.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aef97740 10 client.26807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aef97740 20 client.26807 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aef97740 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c985649f00 0x55c98564c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aef97740 4 client.26807 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c985534a80 con 0x55c985649f00 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aa000640 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c985649f00 0x55c98564c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f74aa000640 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c985649f00 0x55c98564c320 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c8937a8372aaf5 server_cookie=aa4e74c5e071cf1e in_seq=0 out_seq=0 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.205+0000 7f748aa00640 10 client.26807 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f748aa00640 1 -- 172.21.15.146:0/1162409465 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c985534a80 con 0x55c985649f00 2024-04-24T08:28:29.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f748aa00640 10 client.26807 handle_command_reply: tid=0 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 client.26807 shutdown 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 msgr2=0x55c9855d6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855d6ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7490012420 tx=0x7f7490010660 comp rx=0 tx=0).stop 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 shutdown_connections 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55c985649f00 0x55c98564c320 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c9855d2a20 0x55c985648470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c9855c8350 0x55c9855d6ec0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f748aa00640 0 client.26807 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:29.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 --2- 172.21.15.146:0/1162409465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9855c7a30 0x55c9855d6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 >> 172.21.15.146:0/1162409465 conn(0x55c9855c3d50 msgr2=0x55c9855c9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 shutdown_connections 2024-04-24T08:28:29.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 1 -- 172.21.15.146:0/1162409465 wait complete. 2024-04-24T08:28:29.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.206+0000 7f74aef97740 20 client.26807 trim_cache size 0 max 16384 2024-04-24T08:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:28:29.215 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:28:29.216 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:28:29.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f748c0a3d20 msgr2=0x7f748c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f748c0a3d20 0x7f748c0a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55c9855c8ff0 tx=0x7f749800efb0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74a4108580 msgr2=0x7f74a41a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74a4108580 0x7f74a41a4920 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f749800e9c0 tx=0x7f749800ee80 comp rx=0 tx=0).stop 2024-04-24T08:28:29.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 shutdown_connections 2024-04-24T08:28:29.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f748c0a3d20 0x7f748c0a61e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f74a410d500 0x7f74a419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f74a4108f40 0x7f74a41a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 --2- 172.21.15.146:0/1354897618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74a4108580 0x7f74a41a4920 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 >> 172.21.15.146:0/1354897618 conn(0x7f74a4077e20 msgr2=0x7f74a4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 shutdown_connections 2024-04-24T08:28:29.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.207+0000 7f74aaa00640 1 -- 172.21.15.146:0/1354897618 wait complete. 2024-04-24T08:28:29.218 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:28 smithi153 ceph-mon[32746]: pgmap v223: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:28:29.222 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:28 smithi146 ceph-mon[27805]: pgmap v223: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:28:29.225 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:28:29.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:28:29.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:28 smithi177 ceph-mon[33046]: pgmap v223: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:28:29.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 -- 172.21.15.146:0/740350961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c0103260 msgr2=0x7f98c0103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 --2- 172.21.15.146:0/740350961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c0103260 0x7f98c0103640 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009900 tx=0x7f98b402a480 comp rx=0 tx=0).stop 2024-04-24T08:28:29.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 -- 172.21.15.146:0/740350961 shutdown_connections 2024-04-24T08:28:29.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 --2- 172.21.15.146:0/740350961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98c010c630 0x7f98c010ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 --2- 172.21.15.146:0/740350961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98c0103b80 0x7f98c010c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 --2- 172.21.15.146:0/740350961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c0103260 0x7f98c0103640 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 -- 172.21.15.146:0/740350961 >> 172.21.15.146:0/740350961 conn(0x7f98c0077280 msgr2=0x7f98c0077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 -- 172.21.15.146:0/740350961 shutdown_connections 2024-04-24T08:28:29.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.427+0000 7f98c8600640 1 -- 172.21.15.146:0/740350961 wait complete. 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 Processor -- start 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 -- start start 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98c0103260 0x7f98c01a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98c0103b80 0x7f98c01a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c010c630 0x7f98c019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0112420 con 0x7f98c0103b80 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01122a0 con 0x7f98c010c630 2024-04-24T08:28:29.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.429+0000 7f98c8600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01125a0 con 0x7f98c0103260 2024-04-24T08:28:29.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98c0103260 0x7f98c01a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c6600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c010c630 0x7f98c019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98c0103260 0x7f98c01a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:43976/0 (socket says 172.21.15.146:43976) 2024-04-24T08:28:29.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c5c00640 1 -- 172.21.15.146:0/1434953208 learned_addr learned my addr 172.21.15.146:0/1434953208 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c5200640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98c0103b80 0x7f98c01a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c6600640 1 -- 172.21.15.146:0/1434953208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98c0103260 msgr2=0x7f98c01a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c6600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98c0103260 0x7f98c01a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c6600640 1 -- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98c0103b80 msgr2=0x7f98c01a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c6600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98c0103b80 0x7f98c01a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.430+0000 7f98c6600640 1 -- 172.21.15.146:0/1434953208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c019b090 con 0x7f98c010c630 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.431+0000 7f98c6600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c010c630 0x7f98c019a910 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f98b8005fd0 tx=0x7f98b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.431+0000 7f98be000640 1 -- 172.21.15.146:0/1434953208 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b800cdf0 con 0x7f98c010c630 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.431+0000 7f98be000640 1 -- 172.21.15.146:0/1434953208 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98b8005020 con 0x7f98c010c630 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.431+0000 7f98be000640 1 -- 172.21.15.146:0/1434953208 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b80106d0 con 0x7f98c010c630 2024-04-24T08:28:29.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.431+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c019b380 con 0x7f98c010c630 2024-04-24T08:28:29.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.431+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c01a7620 con 0x7f98c010c630 2024-04-24T08:28:29.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.432+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c006ae80 con 0x7f98c010c630 2024-04-24T08:28:29.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.434+0000 7f98be000640 1 -- 172.21.15.146:0/1434953208 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f98b80054d0 con 0x7f98c010c630 2024-04-24T08:28:29.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.436+0000 7f98be000640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98a80a3da0 0x7f98a80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.436+0000 7f98be000640 1 -- 172.21.15.146:0/1434953208 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f98b80d0ec0 con 0x7f98c010c630 2024-04-24T08:28:29.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.436+0000 7f98c5c00640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98a80a3da0 0x7f98a80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.437+0000 7f98c5c00640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98a80a3da0 0x7f98a80a6260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f98b40096c0 tx=0x7f98b40046c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.438+0000 7f98be000640 1 -- 172.21.15.146:0/1434953208 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98b809d4f0 con 0x7f98c010c630 2024-04-24T08:28:29.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.702+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98c0069270 con 0x7f98c010c630 2024-04-24T08:28:29.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.703+0000 7f98be000640 1 -- 172.21.15.146:0/1434953208 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f98c0069270 con 0x7f98c010c630 2024-04-24T08:28:29.704 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:28:29.705 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:28:29.705 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:28:29.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98a80a3da0 msgr2=0x7f98a80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98a80a3da0 0x7f98a80a6260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f98b40096c0 tx=0x7f98b40046c0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c010c630 msgr2=0x7f98c019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c010c630 0x7f98c019a910 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f98b8005fd0 tx=0x7f98b800ef10 comp rx=0 tx=0).stop 2024-04-24T08:28:29.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 shutdown_connections 2024-04-24T08:28:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f98a80a3da0 0x7f98a80a6260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98c010c630 0x7f98c019a910 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98c0103b80 0x7f98c01a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 --2- 172.21.15.146:0/1434953208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f98c0103260 0x7f98c01a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 >> 172.21.15.146:0/1434953208 conn(0x7f98c0077280 msgr2=0x7f98c00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 shutdown_connections 2024-04-24T08:28:29.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.706+0000 7f98c8600640 1 -- 172.21.15.146:0/1434953208 wait complete. 2024-04-24T08:28:29.727 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:28:29.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 -- 172.21.15.146:0/1259959828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 msgr2=0x7fa9a0103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1259959828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a0103440 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa990009860 tx=0x7fa99002a3d0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 -- 172.21.15.146:0/1259959828 shutdown_connections 2024-04-24T08:28:29.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1259959828 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9a010c480 0x7fa9a010e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1259959828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9a0103980 0x7fa9a010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1259959828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a0103440 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 -- 172.21.15.146:0/1259959828 >> 172.21.15.146:0/1259959828 conn(0x7fa9a00fc880 msgr2=0x7fa9a00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 -- 172.21.15.146:0/1259959828 shutdown_connections 2024-04-24T08:28:29.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.923+0000 7fa9a5400640 1 -- 172.21.15.146:0/1259959828 wait complete. 2024-04-24T08:28:29.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.924+0000 7fa9a5400640 1 Processor -- start 2024-04-24T08:28:29.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.924+0000 7fa9a5400640 1 -- start start 2024-04-24T08:28:29.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa9a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa9a5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9a0103980 0x7fa9a019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa9a5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9a010c480 0x7fa9a0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa9a5400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a0112290 con 0x7fa9a0103980 2024-04-24T08:28:29.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa9a5400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a0112110 con 0x7fa9a010c480 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa9a5400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a0112410 con 0x7fa9a0103060 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a019e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44000/0 (socket says 172.21.15.146:44000) 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa99fe00640 1 -- 172.21.15.146:0/1120820425 learned_addr learned my addr 172.21.15.146:0/1120820425 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa9a4a00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9a010c480 0x7fa9a0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.925+0000 7fa99f400640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9a0103980 0x7fa9a019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99fe00640 1 -- 172.21.15.146:0/1120820425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9a010c480 msgr2=0x7fa9a0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99fe00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9a010c480 0x7fa9a0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99fe00640 1 -- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9a0103980 msgr2=0x7fa9a019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99fe00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9a0103980 0x7fa9a019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99fe00640 1 -- 172.21.15.146:0/1120820425 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a0198f40 con 0x7fa9a0103060 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa9a4a00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9a010c480 0x7fa9a0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99fe00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a019e570 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa990009980 tx=0x7fa9900309a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99cc00640 1 -- 172.21.15.146:0/1120820425 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa990040070 con 0x7fa9a0103060 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99cc00640 1 -- 172.21.15.146:0/1120820425 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa990005620 con 0x7fa9a0103060 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa99cc00640 1 -- 172.21.15.146:0/1120820425 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9900049b0 con 0x7fa9a0103060 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a01991d0 con 0x7fa9a0103060 2024-04-24T08:28:29.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.926+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a01a5310 con 0x7fa9a0103060 2024-04-24T08:28:29.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.930+0000 7fa9a9982740 1 Processor -- start 2024-04-24T08:28:29.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a9982740 1 -- start start 2024-04-24T08:28:29.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b8210f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa99cc00640 1 -- 172.21.15.146:0/1120820425 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fa990043070 con 0x7fa9a0103060 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b821bb30 0x5617b821df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a9982740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5617b81761d0 con 0x5617b8210b40 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a9982740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5617b811c450 con 0x5617b821bb30 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a9982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5617b8172190 con 0x5617b8211460 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa9a4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b8210f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa99f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b821bb30 0x5617b821df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa99f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b821bb30 0x5617b821df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37578/0 (socket says 172.21.15.146:37578) 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44014/0 (socket says 172.21.15.146:44014) 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.931+0000 7fa99f400640 1 -- 172.21.15.146:0/3764036958 learned_addr learned my addr 172.21.15.146:0/3764036958 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.932+0000 7fa99f400640 1 -- 172.21.15.146:0/3764036958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 msgr2=0x5617b821b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.932+0000 7fa99f400640 1 --2- 172.21.15.146:0/3764036958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.932+0000 7fa99f400640 1 -- 172.21.15.146:0/3764036958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 msgr2=0x5617b8210f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.932+0000 7fa99f400640 1 --2- 172.21.15.146:0/3764036958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b8210f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.932+0000 7fa99f400640 1 -- 172.21.15.146:0/3764036958 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b81d9310 con 0x5617b821bb30 2024-04-24T08:28:29.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.932+0000 7fa99f400640 1 --2- 172.21.15.146:0/3764036958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b821bb30 0x5617b821df20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009c80 tx=0x7fa98c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a997cb59b304015 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.932+0000 7fa995200640 1 -- 172.21.15.146:0/3764036958 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c01e070 con 0x5617b821bb30 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa995200640 1 -- 172.21.15.146:0/3764036958 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa98c019b10 con 0x5617b821bb30 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa995200640 1 -- 172.21.15.146:0/3764036958 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00ea50 con 0x5617b821bb30 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa99cc00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9840a3da0 0x7fa9840a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a9982740 1 -- 172.21.15.146:0/3764036958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b821bb30 msgr2=0x5617b821df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3764036958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b821bb30 0x5617b821df20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009c80 tx=0x7fa98c011810 comp rx=0 tx=0).stop 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa99cc00640 1 -- 172.21.15.146:0/1120820425 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7fa9900f1460 con 0x7fa9a0103060 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a4a00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9840a3da0 0x7fa9840a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a9982740 1 -- 172.21.15.146:0/3764036958 shutdown_connections 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3764036958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3764036958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b821bb30 0x5617b821df20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3764036958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b8210f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.933+0000 7fa9a9982740 1 -- 172.21.15.146:0/3764036958 >> 172.21.15.146:0/3764036958 conn(0x5617b820ce60 msgr2=0x5617b820d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.934+0000 7fa9a9982740 1 -- 172.21.15.146:0/3764036958 shutdown_connections 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.934+0000 7fa9a9982740 1 -- 172.21.15.146:0/3764036958 wait complete. 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.934+0000 7fa9a4a00640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9840a3da0 0x7fa9840a6260 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa998005fd0 tx=0x7fa9980095c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.934+0000 7fa9a9982740 1 -- start start 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.934+0000 7fa9a9982740 1 Processor -- start 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b821b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8211460 0x5617b8218860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b8218fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a9982740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5617b821e9c0 con 0x5617b8211460 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a9982740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5617b812f810 con 0x5617b8210b40 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a9982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5617b821eb40 con 0x5617b821bb30 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a9982740 10 client.0 _ll_register_callbacks cb 0x5617b80fd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b821b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b8218fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b821b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37592/0 (socket says 172.21.15.146:37592) 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 -- 172.21.15.146:0/3967582408 learned_addr learned my addr 172.21.15.146:0/3967582408 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99f400640 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8211460 0x5617b8218860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 -- 172.21.15.146:0/3967582408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 msgr2=0x5617b8218fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b8218fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 -- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8211460 msgr2=0x5617b8218860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8211460 0x5617b8218860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa99fe00640 1 -- 172.21.15.146:0/3967582408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b81403f0 con 0x5617b8210b40 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.935+0000 7fa9a4a00640 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b8218fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa99fe00640 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b821b570 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa9900bd1a0 tx=0x7fa9900f4ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0470ea9b10dab17 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa989600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa989600640 1 -- 172.21.15.146:0/3967582408 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99004a040 con 0x5617b8210b40 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa989600640 1 -- 172.21.15.146:0/3967582408 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7fa990031400 con 0x5617b8210b40 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa98c019b10 con 0x5617b8210b40 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa989600640 1 -- 172.21.15.146:0/3967582408 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa9900b9740 con 0x5617b8210b40 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa9a9982740 10 client.26825 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:29.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5617b7e883f0 con 0x5617b8210b40 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.936+0000 7fa989600640 1 -- 172.21.15.146:0/3967582408 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa98c019b10 con 0x5617b8210b40 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa989600640 1 -- 172.21.15.146:0/3967582408 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa9900f0030 con 0x5617b8210b40 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa9a9982740 10 client.26825 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa9a9982740 10 client.26825 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa9a9982740 20 client.26825 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b822ff80 0x5617b82323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa9a9982740 4 client.26825 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:28:29.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617b8183bf0 con 0x5617b822ff80 2024-04-24T08:28:29.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.937+0000 7fa99f400640 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b822ff80 0x5617b82323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.938+0000 7fa989600640 10 client.26825 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:28:29.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.938+0000 7fa99f400640 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b822ff80 0x5617b82323d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ac044537e8cd481 server_cookie=dc6dc35be2b5ff30 in_seq=0 out_seq=0 2024-04-24T08:28:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.938+0000 7fa989600640 1 -- 172.21.15.146:0/3967582408 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5617b8183bf0 con 0x5617b822ff80 2024-04-24T08:28:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.938+0000 7fa989600640 10 client.26825 handle_command_reply: tid=0 2024-04-24T08:28:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 client.26825 shutdown 2024-04-24T08:28:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 msgr2=0x5617b821b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b821b570 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa9900bd1a0 tx=0x7fa9900f4ba0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 shutdown_connections 2024-04-24T08:28:29.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b822ff80 0x5617b82323d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b8218fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8211460 0x5617b8218860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa989600640 0 client.26825 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3967582408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b821b570 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 >> 172.21.15.146:0/3967582408 conn(0x5617b820ce60 msgr2=0x5617b8212c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 shutdown_connections 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 1 -- 172.21.15.146:0/3967582408 wait complete. 2024-04-24T08:28:29.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.939+0000 7fa9a9982740 20 client.26825 trim_cache size 0 max 16384 2024-04-24T08:28:29.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 Processor -- start 2024-04-24T08:28:29.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 -- start start 2024-04-24T08:28:29.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b82132a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b821bb30 0x5617b821b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a01ae170 con 0x5617b821bb30 2024-04-24T08:28:29.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a01aba40 con 0x5617b8210b40 2024-04-24T08:28:29.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa9a9982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a01a98b0 con 0x5617b8211460 2024-04-24T08:28:29.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.954+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b82132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b82132a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37596/0 (socket says 172.21.15.146:37596) 2024-04-24T08:28:29.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99fe00640 1 -- 172.21.15.146:0/3098708365 learned_addr learned my addr 172.21.15.146:0/3098708365 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa9a4a00640 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b821bb30 0x5617b821b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99f400640 1 -- 172.21.15.146:0/3098708365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 msgr2=0x5617b82132a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99f400640 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b82132a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99f400640 1 -- 172.21.15.146:0/3098708365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b821bb30 msgr2=0x5617b821b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99f400640 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b821bb30 0x5617b821b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99f400640 1 -- 172.21.15.146:0/3098708365 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b7f527d0 con 0x5617b8211460 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99fe00640 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b82132a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.955+0000 7fa99f400640 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b0a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00e710 tx=0x7fa98c00e610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cd2f711c96c0d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.956+0000 7fa98be00640 1 -- 172.21.15.146:0/3098708365 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c01e040 con 0x5617b8211460 2024-04-24T08:28:29.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.956+0000 7fa98be00640 1 -- 172.21.15.146:0/3098708365 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa98c0054f0 con 0x5617b8211460 2024-04-24T08:28:29.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.956+0000 7fa98be00640 1 -- 172.21.15.146:0/3098708365 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00e9c0 con 0x5617b8211460 2024-04-24T08:28:29.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.956+0000 7fa9a9982740 1 -- 172.21.15.146:0/3098708365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 msgr2=0x5617b821b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.956+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b0a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00e710 tx=0x7fa98c00e610 comp rx=0 tx=0).stop 2024-04-24T08:28:29.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 -- 172.21.15.146:0/3098708365 shutdown_connections 2024-04-24T08:28:29.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b8211460 0x5617b821b0a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b821bb30 0x5617b821b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 --2- 172.21.15.146:0/3098708365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8210b40 0x5617b82132a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 -- 172.21.15.146:0/3098708365 >> 172.21.15.146:0/3098708365 conn(0x5617b820ce60 msgr2=0x5617b8213b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 -- 172.21.15.146:0/3098708365 shutdown_connections 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 -- 172.21.15.146:0/3098708365 wait complete. 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 -- start start 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.957+0000 7fa9a9982740 1 Processor -- start 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b82286f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8211460 0x5617b8215e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a9982740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b82164c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a9982740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a01a9f00 con 0x5617b8210b40 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a9982740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a01ab370 con 0x5617b8211460 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a9982740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a01aa5d0 con 0x5617b821bb30 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a9982740 10 client.0 _ll_register_callbacks cb 0x7fa9900097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8211460 0x5617b8215e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b82286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8211460 0x5617b8215e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:37608/0 (socket says 172.21.15.146:37608) 2024-04-24T08:28:29.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99f400640 1 -- 172.21.15.146:0/456578815 learned_addr learned my addr 172.21.15.146:0/456578815 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa9a4a00640 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b82164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99fe00640 1 -- 172.21.15.146:0/456578815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 msgr2=0x5617b82164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99fe00640 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b82164c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99fe00640 1 -- 172.21.15.146:0/456578815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8211460 msgr2=0x5617b8215e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99fe00640 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8211460 0x5617b8215e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99fe00640 1 -- 172.21.15.146:0/456578815 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b821cc20 con 0x5617b8210b40 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.958+0000 7fa99f400640 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8211460 0x5617b8215e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa99fe00640 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b82286f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa9900b9e70 tx=0x7fa9900bb9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2db71d4deb1592a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:29.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa989600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa989600640 1 -- 172.21.15.146:0/456578815 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9900bce00 con 0x5617b8210b40 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa989600640 1 -- 172.21.15.146:0/456578815 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7fa990031c30 con 0x5617b8210b40 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa98c0054f0 con 0x5617b8210b40 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa989600640 1 -- 172.21.15.146:0/456578815 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa9900bcfe0 con 0x5617b8210b40 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa9a9982740 10 client.16683 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5617b821cc20 con 0x5617b8210b40 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.959+0000 7fa989600640 1 -- 172.21.15.146:0/456578815 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa98c0054f0 con 0x5617b8210b40 2024-04-24T08:28:29.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa989600640 1 -- 172.21.15.146:0/456578815 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa9900d8ce0 con 0x5617b8210b40 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa9a9982740 10 client.16683 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa9a9982740 10 client.16683 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa9a9982740 20 client.16683 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa9a9982740 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b8296d00 0x5617b82990f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa9a9982740 4 client.16683 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617b817da60 con 0x5617b8296d00 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.960+0000 7fa99f400640 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b8296d00 0x5617b82990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:29.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa989600640 10 client.16683 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:28:29.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa99f400640 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b8296d00 0x5617b82990f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3210e3c79d71a4d2 server_cookie=89356a54fedc93db in_seq=0 out_seq=0 2024-04-24T08:28:29.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa989600640 1 -- 172.21.15.146:0/456578815 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617b817da60 con 0x5617b8296d00 2024-04-24T08:28:29.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa989600640 10 client.16683 handle_command_reply: tid=0 2024-04-24T08:28:29.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 client.16683 shutdown 2024-04-24T08:28:29.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 msgr2=0x5617b82286f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b82286f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa9900b9e70 tx=0x7fa9900bb9b0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 shutdown_connections 2024-04-24T08:28:29.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5617b8296d00 0x5617b82990f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5617b821bb30 0x5617b82164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5617b8211460 0x5617b8215e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 --2- 172.21.15.146:0/456578815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5617b8210b40 0x5617b82286f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 >> 172.21.15.146:0/456578815 conn(0x5617b820ce60 msgr2=0x5617b820b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.961+0000 7fa989600640 0 client.16683 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.962+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 shutdown_connections 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.962+0000 7fa9a9982740 1 -- 172.21.15.146:0/456578815 wait complete. 2024-04-24T08:28:29.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.962+0000 7fa9a9982740 20 client.16683 trim_cache size 0 max 16384 2024-04-24T08:28:29.970 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:28:29.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.962+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9840a3da0 msgr2=0x7fa9840a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.962+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9840a3da0 0x7fa9840a6260 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa998005fd0 tx=0x7fa9980095c0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 msgr2=0x7fa9a019e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a019e570 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa990009980 tx=0x7fa9900309a0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 shutdown_connections 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa9840a3da0 0x7fa9840a6260 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9a010c480 0x7fa9a0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9a0103980 0x7fa9a019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 --2- 172.21.15.146:0/1120820425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9a0103060 0x7fa9a019e570 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 >> 172.21.15.146:0/1120820425 conn(0x7fa9a00fc880 msgr2=0x7fa9a00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:29.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 shutdown_connections 2024-04-24T08:28:29.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:29.963+0000 7fa9a5400640 1 -- 172.21.15.146:0/1120820425 wait complete. 2024-04-24T08:28:29.985 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:29 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:28:29.985 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:29 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1434953208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:28:29.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:28:30.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.170+0000 7ffa69c00640 1 -- 172.21.15.146:0/3821011754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa64104e60 msgr2=0x7ffa64108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.170+0000 7ffa69c00640 1 --2- 172.21.15.146:0/3821011754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa64104e60 0x7ffa64108ea0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009900 tx=0x7ffa5002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.170+0000 7ffa69c00640 1 -- 172.21.15.146:0/3821011754 shutdown_connections 2024-04-24T08:28:30.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.170+0000 7ffa69c00640 1 --2- 172.21.15.146:0/3821011754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa64104e60 0x7ffa64108ea0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.170+0000 7ffa69c00640 1 --2- 172.21.15.146:0/3821011754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa64109580 0x7ffa64111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.170+0000 7ffa69c00640 1 --2- 172.21.15.146:0/3821011754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa641044b0 0x7ffa64104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.170+0000 7ffa69c00640 1 -- 172.21.15.146:0/3821011754 >> 172.21.15.146:0/3821011754 conn(0x7ffa640ffd90 msgr2=0x7ffa641021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.171+0000 7ffa69c00640 1 -- 172.21.15.146:0/3821011754 shutdown_connections 2024-04-24T08:28:30.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.171+0000 7ffa69c00640 1 -- 172.21.15.146:0/3821011754 wait complete. 2024-04-24T08:28:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.171+0000 7ffa69c00640 1 Processor -- start 2024-04-24T08:28:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa69c00640 1 -- start start 2024-04-24T08:28:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa69c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa641044b0 0x7ffa641a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa69c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa64109580 0x7ffa641a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa69c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6419a9e0 0x7ffa6419ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa69c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffa64114640 con 0x7ffa64109580 2024-04-24T08:28:30.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa69c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641144c0 con 0x7ffa6419a9e0 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa69c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641147c0 con 0x7ffa641044b0 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa62000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa64109580 0x7ffa641a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa62a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa641044b0 0x7ffa641a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa62000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa64109580 0x7ffa641a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49476/0 (socket says 172.21.15.146:49476) 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa62000640 1 -- 172.21.15.146:0/1008797098 learned_addr learned my addr 172.21.15.146:0/1008797098 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.172+0000 7ffa63400640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6419a9e0 0x7ffa6419ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa62a00640 1 -- 172.21.15.146:0/1008797098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6419a9e0 msgr2=0x7ffa6419ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa62a00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6419a9e0 0x7ffa6419ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa62a00640 1 -- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa64109580 msgr2=0x7ffa641a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa62a00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa64109580 0x7ffa641a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa62a00640 1 -- 172.21.15.146:0/1008797098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6419b500 con 0x7ffa641044b0 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa63400640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6419a9e0 0x7ffa6419ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa62a00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa641044b0 0x7ffa641a07e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400e9f0 tx=0x7ffa5400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa5b400640 1 -- 172.21.15.146:0/1008797098 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5400ccc0 con 0x7ffa641044b0 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa5b400640 1 -- 172.21.15.146:0/1008797098 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa54010910 con 0x7ffa641044b0 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6419b7f0 con 0x7ffa641044b0 2024-04-24T08:28:30.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa5b400640 1 -- 172.21.15.146:0/1008797098 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54018960 con 0x7ffa641044b0 2024-04-24T08:28:30.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.173+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa641a7a40 con 0x7ffa641044b0 2024-04-24T08:28:30.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.176+0000 7ffa6c9a6740 1 Processor -- start 2024-04-24T08:28:30.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.176+0000 7ffa5b400640 1 -- 172.21.15.146:0/1008797098 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7ffa54010ac0 con 0x7ffa641044b0 2024-04-24T08:28:30.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa6c9a6740 1 -- start start 2024-04-24T08:28:30.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7ea160 0x55924a7ea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7f5150 0x55924a7f7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55924a7471d0 con 0x55924a7f5150 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55924a6ed450 con 0x55924a7eaa80 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55924a743190 con 0x55924a7ea160 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa63400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7ea160 0x55924a7ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa63400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7ea160 0x55924a7ea540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44060/0 (socket says 172.21.15.146:44060) 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa63400640 1 -- 172.21.15.146:0/3791925499 learned_addr learned my addr 172.21.15.146:0/3791925499 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa62a00640 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.177+0000 7ffa62000640 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7f5150 0x55924a7f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa63400640 1 -- 172.21.15.146:0/3791925499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 msgr2=0x55924a7f4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa63400640 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa63400640 1 -- 172.21.15.146:0/3791925499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7f5150 msgr2=0x55924a7f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa63400640 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7f5150 0x55924a7f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa63400640 1 -- 172.21.15.146:0/3791925499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55924a7aa310 con 0x55924a7ea160 2024-04-24T08:28:30.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa63400640 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7ea160 0x55924a7ea540 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009fd0 tx=0x7ffa5c00f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0a0f3306afbe67b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa5b400640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa480a3da0 0x7ffa480a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa4fe00640 1 -- 172.21.15.146:0/3791925499 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c015070 con 0x55924a7ea160 2024-04-24T08:28:30.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa4fe00640 1 -- 172.21.15.146:0/3791925499 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5c011be0 con 0x55924a7ea160 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa4fe00640 1 -- 172.21.15.146:0/3791925499 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c010c60 con 0x55924a7ea160 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa5b400640 1 -- 172.21.15.146:0/1008797098 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7ffa540d0c30 con 0x7ffa641044b0 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.178+0000 7ffa62a00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa480a3da0 0x7ffa480a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa62a00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa480a3da0 0x7ffa480a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55924a7eb720 tx=0x7ffa54009700 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/3791925499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7ea160 msgr2=0x55924a7ea540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7ea160 0x55924a7ea540 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009fd0 tx=0x7ffa5c00f7f0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/3791925499 shutdown_connections 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7f5150 0x55924a7f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/3791925499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7ea160 0x55924a7ea540 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/3791925499 >> 172.21.15.146:0/3791925499 conn(0x55924a7e6480 msgr2=0x55924a7e6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/3791925499 shutdown_connections 2024-04-24T08:28:30.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.179+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/3791925499 wait complete. 2024-04-24T08:28:30.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 -- start start 2024-04-24T08:28:30.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 Processor -- start 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7ea160 0x55924a7f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7eaa80 0x55924a7f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55924a7f7fe0 con 0x55924a7eaa80 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55924a700810 con 0x55924a7ea160 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55924a7f8160 con 0x55924a7f5150 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.180+0000 7ffa6c9a6740 10 client.0 _ll_register_callbacks cb 0x55924a6ce140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:30.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7eaa80 0x55924a7f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7eaa80 0x55924a7f1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49504/0 (socket says 172.21.15.146:49504) 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 -- 172.21.15.146:0/2934782440 learned_addr learned my addr 172.21.15.146:0/2934782440 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa63400640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa63400640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7ea160 0x55924a7f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 -- 172.21.15.146:0/2934782440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 msgr2=0x55924a7f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 -- 172.21.15.146:0/2934782440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7ea160 msgr2=0x55924a7f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7ea160 0x55924a7f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa62000640 1 -- 172.21.15.146:0/2934782440 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55924a7113f0 con 0x55924a7eaa80 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.181+0000 7ffa63400640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa62000640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7eaa80 0x55924a7f1ee0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffa5002a9e0 tx=0x7ffa50004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91cbcc0e6e56c8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa4cc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2934782440 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50035040 con 0x55924a7eaa80 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2934782440 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7ffa50005450 con 0x55924a7eaa80 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa5c011be0 con 0x55924a7eaa80 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2934782440 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ffa50005650 con 0x55924a7eaa80 2024-04-24T08:28:30.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa6c9a6740 10 client.16695 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55924a4593f0 con 0x55924a7eaa80 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.182+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2934782440 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5c011be0 con 0x55924a7eaa80 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2934782440 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ffa50003a30 con 0x55924a7eaa80 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa6c9a6740 10 client.16695 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa6c9a6740 10 client.16695 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa6c9a6740 20 client.16695 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a805360 0x55924a807750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa6c9a6740 4 client.16695 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55924a754bf0 con 0x55924a805360 2024-04-24T08:28:30.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa63400640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a805360 0x55924a807750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa4cc00640 10 client.16695 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.183+0000 7ffa63400640 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a805360 0x55924a807750 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c8e48fab8799a8a1 server_cookie=8e38e8b03b87b04d in_seq=0 out_seq=0 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2934782440 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55924a754bf0 con 0x55924a805360 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa4cc00640 10 client.16695 handle_command_reply: tid=0 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 client.16695 shutdown 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7eaa80 msgr2=0x55924a7f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7eaa80 0x55924a7f1ee0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffa5002a9e0 tx=0x7ffa50004440 comp rx=0 tx=0).stop 2024-04-24T08:28:30.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 shutdown_connections 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a805360 0x55924a807750 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7eaa80 0x55924a7f1ee0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2934782440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7ea160 0x55924a7f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa4cc00640 0 client.16695 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 >> 172.21.15.146:0/2934782440 conn(0x55924a7e6480 msgr2=0x55924a7ec280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.185+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 shutdown_connections 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.186+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2934782440 wait complete. 2024-04-24T08:28:30.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.186+0000 7ffa6c9a6740 20 client.16695 trim_cache size 0 max 16384 2024-04-24T08:28:30.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.200+0000 7ffa6c9a6740 1 Processor -- start 2024-04-24T08:28:30.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.200+0000 7ffa6c9a6740 1 -- start start 2024-04-24T08:28:30.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.200+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.200+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a800ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7dcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641ae5c0 con 0x55924a7ea160 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641abe90 con 0x55924a7eaa80 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641a9d20 con 0x55924a7f5150 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49514/0 (socket says 172.21.15.146:49514) 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 -- 172.21.15.146:0/270807314 learned_addr learned my addr 172.21.15.146:0/270807314 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa62000640 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a800ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7dcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 -- 172.21.15.146:0/270807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 msgr2=0x55924a7dcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7dcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 -- 172.21.15.146:0/270807314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 msgr2=0x55924a800ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a800ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa63400640 1 -- 172.21.15.146:0/270807314 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55924a5237d0 con 0x55924a7ea160 2024-04-24T08:28:30.204 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.201+0000 7ffa62000640 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a800ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:30.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.202+0000 7ffa63400640 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f74f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c010850 tx=0x7ffa5c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38174ec0a40a43a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.208 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.202+0000 7ffa4f400640 1 -- 172.21.15.146:0/270807314 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c015040 con 0x55924a7ea160 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.202+0000 7ffa4f400640 1 -- 172.21.15.146:0/270807314 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5c009ae0 con 0x55924a7ea160 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.202+0000 7ffa4f400640 1 -- 172.21.15.146:0/270807314 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c02c750 con 0x55924a7ea160 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/270807314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 msgr2=0x55924a7f74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f74f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c010850 tx=0x7ffa5c000f40 comp rx=0 tx=0).stop 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/270807314 shutdown_connections 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7dcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a800ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/270807314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f74f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/270807314 >> 172.21.15.146:0/270807314 conn(0x55924a7e6480 msgr2=0x55924a7dc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/270807314 shutdown_connections 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/270807314 wait complete. 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 -- start start 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.203+0000 7ffa6c9a6740 1 Processor -- start 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa6c9a6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641aa370 con 0x55924a7ea160 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641ab7c0 con 0x55924a7eaa80 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa6c9a6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa641aaa20 con 0x55924a7f5150 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa6c9a6740 10 client.0 _ll_register_callbacks cb 0x7ffa50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa63400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa63400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49522/0 (socket says 172.21.15.146:49522) 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa63400640 1 -- 172.21.15.146:0/2655429868 learned_addr learned my addr 172.21.15.146:0/2655429868 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa62000640 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.204+0000 7ffa63400640 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa62000640 1 -- 172.21.15.146:0/2655429868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 msgr2=0x55924a7f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa62000640 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa62000640 1 -- 172.21.15.146:0/2655429868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 msgr2=0x55924a7f6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa62000640 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa62000640 1 -- 172.21.15.146:0/2655429868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55924a7fba80 con 0x55924a7eaa80 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa62000640 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f6c70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffa5002a910 tx=0x7ffa50005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd67fcb69663d370 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa4cc00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2655429868 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50035070 con 0x55924a7eaa80 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa5c009ae0 con 0x55924a7eaa80 2024-04-24T08:28:30.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2655429868 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7ffa50031ba0 con 0x55924a7eaa80 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2655429868 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ffa50052040 con 0x55924a7eaa80 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa6c9a6740 10 client.26849 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.205+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55924a86be00 con 0x55924a7eaa80 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2655429868 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5c009ae0 con 0x55924a7eaa80 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2655429868 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ffa500046c0 con 0x55924a7eaa80 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa6c9a6740 10 client.26849 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa6c9a6740 10 client.26849 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:28:30.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa6c9a6740 20 client.26849 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a86c5b0 0x55924a86e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa6c9a6740 4 client.26849 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55924a74ea60 con 0x55924a86c5b0 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.206+0000 7ffa63400640 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a86c5b0 0x55924a86e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa63400640 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a86c5b0 0x55924a86e9d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60c8c63337770f43 server_cookie=5e5d1e166d519d65 in_seq=0 out_seq=0 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa4cc00640 10 client.26849 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa4cc00640 1 -- 172.21.15.146:0/2655429868 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55924a74ea60 con 0x55924a86c5b0 2024-04-24T08:28:30.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa4cc00640 10 client.26849 handle_command_reply: tid=0 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 client.26849 shutdown 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 msgr2=0x55924a7f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f6c70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffa5002a910 tx=0x7ffa50005fd0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 shutdown_connections 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55924a86c5b0 0x55924a86e9d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55924a7f5150 0x55924a7f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55924a7eaa80 0x55924a7f6c70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 --2- 172.21.15.146:0/2655429868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55924a7ea160 0x55924a7f6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa4cc00640 0 client.26849 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:28:30.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 >> 172.21.15.146:0/2655429868 conn(0x55924a7e6480 msgr2=0x55924a7dcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 shutdown_connections 2024-04-24T08:28:30.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.207+0000 7ffa6c9a6740 1 -- 172.21.15.146:0/2655429868 wait complete. 2024-04-24T08:28:30.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa6c9a6740 20 client.26849 trim_cache size 0 max 16384 2024-04-24T08:28:30.216 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:28:30.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa480a3da0 msgr2=0x7ffa480a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa480a3da0 0x7ffa480a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55924a7eb720 tx=0x7ffa54009700 comp rx=0 tx=0).stop 2024-04-24T08:28:30.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa641044b0 msgr2=0x7ffa641a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa641044b0 0x7ffa641a07e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400e9f0 tx=0x7ffa5400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 shutdown_connections 2024-04-24T08:28:30.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa480a3da0 0x7ffa480a6260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa6419a9e0 0x7ffa6419ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa64109580 0x7ffa641a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 --2- 172.21.15.146:0/1008797098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa641044b0 0x7ffa641a07e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 >> 172.21.15.146:0/1008797098 conn(0x7ffa640ffd90 msgr2=0x7ffa64101140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.208+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 shutdown_connections 2024-04-24T08:28:30.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.209+0000 7ffa69c00640 1 -- 172.21.15.146:0/1008797098 wait complete. 2024-04-24T08:28:30.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:29 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:28:30.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:29 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1434953208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:28:30.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:28:30.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:29 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:28:30.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:29 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1434953208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:28:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 -- 172.21.15.146:0/1641288557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60112ee0 msgr2=0x7fcb601152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 --2- 172.21.15.146:0/1641288557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60112ee0 0x7fcb601152d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009cd0 tx=0x7fcb5802a8e0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 -- 172.21.15.146:0/1641288557 shutdown_connections 2024-04-24T08:28:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 --2- 172.21.15.146:0/1641288557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60112ee0 0x7fcb601152d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 --2- 172.21.15.146:0/1641288557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb60076e50 0x7fcb601129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 --2- 172.21.15.146:0/1641288557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60076530 0x7fcb60076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 -- 172.21.15.146:0/1641288557 >> 172.21.15.146:0/1641288557 conn(0x7fcb600ffd90 msgr2=0x7fcb601021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 -- 172.21.15.146:0/1641288557 shutdown_connections 2024-04-24T08:28:30.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.412+0000 7fcb64c00640 1 -- 172.21.15.146:0/1641288557 wait complete. 2024-04-24T08:28:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.413+0000 7fcb64c00640 1 Processor -- start 2024-04-24T08:28:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.413+0000 7fcb64c00640 1 -- start start 2024-04-24T08:28:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.413+0000 7fcb64c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb60076530 0x7fcb6019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.413+0000 7fcb64c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60076e50 0x7fcb6019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.413+0000 7fcb64c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60112ee0 0x7fcb601a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb64c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60118a40 con 0x7fcb60076530 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb64c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601188c0 con 0x7fcb60112ee0 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb64c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcb60118bc0 con 0x7fcb60076e50 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60076e50 0x7fcb6019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60076e50 0x7fcb6019f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44120/0 (socket says 172.21.15.146:44120) 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 -- 172.21.15.146:0/2107590413 learned_addr learned my addr 172.21.15.146:0/2107590413 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb60076530 0x7fcb6019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5e800640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60112ee0 0x7fcb601a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 -- 172.21.15.146:0/2107590413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60112ee0 msgr2=0x7fcb601a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60112ee0 0x7fcb601a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 -- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb60076530 msgr2=0x7fcb6019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb60076530 0x7fcb6019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.414+0000 7fcb5d400640 1 -- 172.21.15.146:0/2107590413 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb601a3f60 con 0x7fcb60076e50 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.415+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60076e50 0x7fcb6019f500 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00d950 tx=0x7fcb4c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.415+0000 7fcb56a00640 1 -- 172.21.15.146:0/2107590413 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c017070 con 0x7fcb60076e50 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.415+0000 7fcb56a00640 1 -- 172.21.15.146:0/2107590413 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb4c009c30 con 0x7fcb60076e50 2024-04-24T08:28:30.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.415+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb601a4250 con 0x7fcb60076e50 2024-04-24T08:28:30.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.415+0000 7fcb56a00640 1 -- 172.21.15.146:0/2107590413 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c00f6b0 con 0x7fcb60076e50 2024-04-24T08:28:30.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.415+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb601abae0 con 0x7fcb60076e50 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.418+0000 7fcb67bb0740 1 Processor -- start 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.418+0000 7fcb67bb0740 1 -- start start 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.418+0000 7fcb56a00640 1 -- 172.21.15.146:0/2107590413 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fcb4c0109d0 con 0x7fcb60076e50 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.418+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cefd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.418+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cf0270 0x557ec6cfa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.418+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557ec6c551d0 con 0x557ec6cef950 2024-04-24T08:28:30.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557ec6bfb450 con 0x557ec6cf0270 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ec6c51190 con 0x557ec6cfa940 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cefd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cf0270 0x557ec6cfa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cefd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49542/0 (socket says 172.21.15.146:49542) 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 -- 172.21.15.146:0/3941155525 learned_addr learned my addr 172.21.15.146:0/3941155525 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5d400640 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 -- 172.21.15.146:0/3941155525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 msgr2=0x557ec6cfcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 -- 172.21.15.146:0/3941155525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cf0270 msgr2=0x557ec6cfa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cf0270 0x557ec6cfa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5e800640 1 -- 172.21.15.146:0/3941155525 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ec6cb8310 con 0x557ec6cef950 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.419+0000 7fcb5de00640 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cf0270 0x557ec6cfa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.420+0000 7fcb5e800640 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cefd30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcb58004ab0 tx=0x7fcb58009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136420c457b65bd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.420+0000 7fcb4b000640 1 -- 172.21.15.146:0/3941155525 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58048070 con 0x557ec6cef950 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.420+0000 7fcb4b000640 1 -- 172.21.15.146:0/3941155525 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb58036070 con 0x557ec6cef950 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.420+0000 7fcb4b000640 1 -- 172.21.15.146:0/3941155525 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb580044a0 con 0x557ec6cef950 2024-04-24T08:28:30.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.420+0000 7fcb56a00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb440a3d20 0x7fcb440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.420+0000 7fcb56a00640 1 -- 172.21.15.146:0/2107590413 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7fcb4c0d0370 con 0x7fcb60076e50 2024-04-24T08:28:30.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.420+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb440a3d20 0x7fcb440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3941155525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 msgr2=0x557ec6cefd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cefd30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcb58004ab0 tx=0x7fcb58009a20 comp rx=0 tx=0).stop 2024-04-24T08:28:30.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3941155525 shutdown_connections 2024-04-24T08:28:30.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cf0270 0x557ec6cfa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3941155525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cefd30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb440a3d20 0x7fcb440a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x557ec6cf0f10 tx=0x7fcb5000b390 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3941155525 >> 172.21.15.146:0/3941155525 conn(0x557ec6cebc70 msgr2=0x557ec6cec050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3941155525 shutdown_connections 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.421+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3941155525 wait complete. 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 -- start start 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 Processor -- start 2024-04-24T08:28:30.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cfa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557ec6cfd770 con 0x557ec6cef950 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557ec6c0e810 con 0x557ec6cfa940 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ec6cfd8f0 con 0x557ec6cf0270 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.422+0000 7fcb67bb0740 10 client.0 _ll_register_callbacks cb 0x557ec6bdc140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cfa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cfa370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49546/0 (socket says 172.21.15.146:49546) 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5de00640 1 -- 172.21.15.146:0/2670434055 learned_addr learned my addr 172.21.15.146:0/2670434055 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5d400640 1 -- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 msgr2=0x557ec6cf7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5d400640 1 -- 172.21.15.146:0/2670434055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 msgr2=0x557ec6cfa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cfa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5d400640 1 -- 172.21.15.146:0/2670434055 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ec6c1f3f0 con 0x557ec6cf0270 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5e800640 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:30.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cfa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf76d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0cda50 tx=0x7fcb4c010d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5abcba19cdca7d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb3f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.423+0000 7fcb3f400640 1 -- 172.21.15.146:0/2670434055 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c020070 con 0x557ec6cf0270 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb3f400640 1 -- 172.21.15.146:0/2670434055 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7fcb4c09b7a0 con 0x557ec6cf0270 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb58036070 con 0x557ec6cf0270 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb3f400640 1 -- 172.21.15.146:0/2670434055 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb4c09be90 con 0x557ec6cf0270 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 10 client.26845 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557ec69673f0 con 0x557ec6cf0270 2024-04-24T08:28:30.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb3f400640 1 -- 172.21.15.146:0/2670434055 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb58036070 con 0x557ec6cf0270 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb3f400640 1 -- 172.21.15.146:0/2670434055 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb4c0a0120 con 0x557ec6cf0270 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 10 client.26845 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 10 client.26845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 20 client.26845 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d0ab50 0x557ec6d0cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 4 client.26845 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.424+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ec6c62bf0 con 0x557ec6d0ab50 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.425+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d0ab50 0x557ec6d0cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.425+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d0ab50 0x557ec6d0cf40 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32f6017e1de09ffe server_cookie=d00df0f0c6565e91 in_seq=0 out_seq=0 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.425+0000 7fcb3f400640 10 client.26845 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb3f400640 1 -- 172.21.15.146:0/2670434055 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ec6c62bf0 con 0x557ec6d0ab50 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb3f400640 10 client.26845 handle_command_reply: tid=0 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 client.26845 shutdown 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 msgr2=0x557ec6cf76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf76d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0cda50 tx=0x7fcb4c010d70 comp rx=0 tx=0).stop 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 shutdown_connections 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d0ab50 0x557ec6d0cf40 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf76d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb3f400640 0 client.26845 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:30.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2670434055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6cfa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 >> 172.21.15.146:0/2670434055 conn(0x557ec6cebc70 msgr2=0x557ec6cf1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 shutdown_connections 2024-04-24T08:28:30.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.426+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2670434055 wait complete. 2024-04-24T08:28:30.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.427+0000 7fcb67bb0740 20 client.26845 trim_cache size 0 max 16384 2024-04-24T08:28:30.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.441+0000 7fcb67bb0740 1 Processor -- start 2024-04-24T08:28:30.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.441+0000 7fcb67bb0740 1 -- start start 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6d06630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601b6c50 con 0x557ec6cef950 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601b4520 con 0x557ec6cfa940 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601b2390 con 0x557ec6cf0270 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6d06630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6d06630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:49552/0 (socket says 172.21.15.146:49552) 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 -- 172.21.15.146:0/2538794256 learned_addr learned my addr 172.21.15.146:0/2538794256 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5e800640 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 -- 172.21.15.146:0/2538794256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 msgr2=0x557ec6cf29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 -- 172.21.15.146:0/2538794256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 msgr2=0x557ec6cf2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5de00640 1 -- 172.21.15.146:0/2538794256 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ec6cf8180 con 0x557ec6cef950 2024-04-24T08:28:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5e800640 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.442+0000 7fcb5d400640 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.443+0000 7fcb5de00640 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6d06630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x557ec6d0d640 tx=0x7fcb50019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8599dc2a5a2576b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.443+0000 7fcb4a600640 1 -- 172.21.15.146:0/2538794256 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5001ea50 con 0x557ec6cef950 2024-04-24T08:28:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.443+0000 7fcb4a600640 1 -- 172.21.15.146:0/2538794256 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb5002f070 con 0x557ec6cef950 2024-04-24T08:28:30.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.443+0000 7fcb4a600640 1 -- 172.21.15.146:0/2538794256 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5001d920 con 0x557ec6cef950 2024-04-24T08:28:30.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2538794256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 msgr2=0x557ec6d06630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6d06630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x557ec6d0d640 tx=0x7fcb50019000 comp rx=0 tx=0).stop 2024-04-24T08:28:30.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2538794256 shutdown_connections 2024-04-24T08:28:30.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cf0270 0x557ec6cf29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cfa940 0x557ec6cf2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/2538794256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cef950 0x557ec6d06630 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2538794256 >> 172.21.15.146:0/2538794256 conn(0x557ec6cebc70 msgr2=0x557ec6cec050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2538794256 shutdown_connections 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 -- 172.21.15.146:0/2538794256 wait complete. 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.444+0000 7fcb67bb0740 1 -- start start 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 1 Processor -- start 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cef950 0x557ec6cfb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cf0270 0x557ec6cfbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601b29e0 con 0x557ec6cf0270 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601b3e50 con 0x557ec6cef950 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcb601b30b0 con 0x557ec6cfa940 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb67bb0740 10 client.0 _ll_register_callbacks cb 0x7fcb4c00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb5d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cf0270 0x557ec6cfbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb5e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb5e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44144/0 (socket says 172.21.15.146:44144) 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb5e800640 1 -- 172.21.15.146:0/3178535919 learned_addr learned my addr 172.21.15.146:0/3178535919 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.445+0000 7fcb5de00640 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cef950 0x557ec6cfb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.452 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb5e800640 1 -- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cef950 msgr2=0x557ec6cfb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb5e800640 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cef950 0x557ec6cfb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb5e800640 1 -- 172.21.15.146:0/3178535919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cf0270 msgr2=0x557ec6cfbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb5e800640 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cf0270 0x557ec6cfbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb5e800640 1 -- 172.21.15.146:0/3178535919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ec6a317d0 con 0x557ec6cfa940 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb5e800640 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfc0b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009bd0 tx=0x7fcb58009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b6b80716fcd9316 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb3f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb3f400640 1 -- 172.21.15.146:0/3178535919 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58048040 con 0x557ec6cfa940 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb3f400640 1 -- 172.21.15.146:0/3178535919 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7fcb58003b70 con 0x557ec6cfa940 2024-04-24T08:28:30.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.446+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb5002f070 con 0x557ec6cfa940 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb3f400640 1 -- 172.21.15.146:0/3178535919 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb58003d70 con 0x557ec6cfa940 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 10 client.26851 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557ec6d70f40 con 0x557ec6cfa940 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb3f400640 1 -- 172.21.15.146:0/3178535919 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb5002f070 con 0x557ec6cfa940 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb3f400640 1 -- 172.21.15.146:0/3178535919 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb58042030 con 0x557ec6cfa940 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 10 client.26851 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 10 client.26851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 20 client.26851 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:30.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d71280 0x557ec6d73670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 4 client.26851 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ec6c5ca60 con 0x557ec6d71280 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.447+0000 7fcb5de00640 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d71280 0x557ec6d73670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb5de00640 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d71280 0x557ec6d73670 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66049244542cc48b server_cookie=d6c58a1aaac92999 in_seq=0 out_seq=0 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb3f400640 10 client.26851 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb3f400640 1 -- 172.21.15.146:0/3178535919 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ec6c5ca60 con 0x557ec6d71280 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb3f400640 10 client.26851 handle_command_reply: tid=0 2024-04-24T08:28:30.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 client.26851 shutdown 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 msgr2=0x557ec6cfc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfc0b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009bd0 tx=0x7fcb58009540 comp rx=0 tx=0).stop 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 shutdown_connections 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x557ec6d71280 0x557ec6d73670 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ec6cfa940 0x557ec6cfc0b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb3f400640 0 client.26851 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557ec6cf0270 0x557ec6cfbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.448+0000 7fcb67bb0740 1 --2- 172.21.15.146:0/3178535919 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557ec6cef950 0x557ec6cfb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 >> 172.21.15.146:0/3178535919 conn(0x557ec6cebc70 msgr2=0x557ec6cf29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 shutdown_connections 2024-04-24T08:28:30.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb67bb0740 1 -- 172.21.15.146:0/3178535919 wait complete. 2024-04-24T08:28:30.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb67bb0740 20 client.26851 trim_cache size 0 max 16384 2024-04-24T08:28:30.457 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:28:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb440a3d20 msgr2=0x7fcb440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb64c00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb440a3d20 0x7fcb440a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x557ec6cf0f10 tx=0x7fcb5000b390 comp rx=0 tx=0).stop 2024-04-24T08:28:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60076e50 msgr2=0x7fcb6019f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.449+0000 7fcb64c00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60076e50 0x7fcb6019f500 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00d950 tx=0x7fcb4c00de10 comp rx=0 tx=0).stop 2024-04-24T08:28:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 shutdown_connections 2024-04-24T08:28:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fcb440a3d20 0x7fcb440a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb60076e50 0x7fcb6019f500 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcb60112ee0 0x7fcb601a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 --2- 172.21.15.146:0/2107590413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb60076530 0x7fcb6019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:30.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 >> 172.21.15.146:0/2107590413 conn(0x7fcb600ffd90 msgr2=0x7fcb60102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:30.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 shutdown_connections 2024-04-24T08:28:30.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:30.450+0000 7fcb64c00640 1 -- 172.21.15.146:0/2107590413 wait complete. 2024-04-24T08:28:31.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:30 smithi153 ceph-mon[32746]: pgmap v224: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:31.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:30 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:28:31.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:30 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:28:31.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:30 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:28:31.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:30 smithi146 ceph-mon[27805]: pgmap v224: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:31.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:30 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:28:31.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:30 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:28:31.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:30 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:28:31.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:30 smithi177 ceph-mon[33046]: pgmap v224: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:31.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:30 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:28:31.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:30 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:28:31.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:30 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:28:31.470 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:28:31.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:28:31.541 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:28:31.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.760+0000 7f435e000640 1 -- 172.21.15.146:0/4183216264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4358103ba0 msgr2=0x7f435810c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.760+0000 7f435e000640 1 --2- 172.21.15.146:0/4183216264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4358103ba0 0x7f435810c080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4340009900 tx=0x7f434002a520 comp rx=0 tx=0).stop 2024-04-24T08:28:31.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.760+0000 7f435e000640 1 -- 172.21.15.146:0/4183216264 shutdown_connections 2024-04-24T08:28:31.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.760+0000 7f435e000640 1 --2- 172.21.15.146:0/4183216264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4358103ba0 0x7f435810c080 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.760+0000 7f435e000640 1 --2- 172.21.15.146:0/4183216264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f435810c5c0 0x7f435810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.760+0000 7f435e000640 1 --2- 172.21.15.146:0/4183216264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4358103280 0x7f4358103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.760+0000 7f435e000640 1 -- 172.21.15.146:0/4183216264 >> 172.21.15.146:0/4183216264 conn(0x7f4358078000 msgr2=0x7f4358078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:31.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.761+0000 7f435e000640 1 -- 172.21.15.146:0/4183216264 shutdown_connections 2024-04-24T08:28:31.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.761+0000 7f435e000640 1 -- 172.21.15.146:0/4183216264 wait complete. 2024-04-24T08:28:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.762+0000 7f435e000640 1 Processor -- start 2024-04-24T08:28:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.762+0000 7f435e000640 1 -- start start 2024-04-24T08:28:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.762+0000 7f435e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4358103280 0x7f43581a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.762+0000 7f435e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4358103ba0 0x7f43581a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f435e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f435810c5c0 0x7f435819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f435e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f43581123f0 con 0x7f4358103280 2024-04-24T08:28:31.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f435e000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4358112270 con 0x7f4358103ba0 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f435e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4358112570 con 0x7f435810c5c0 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4357400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4358103280 0x7f43581a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4357400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4358103280 0x7f43581a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41322/0 (socket says 172.21.15.146:41322) 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4357400640 1 -- 172.21.15.146:0/1588203516 learned_addr learned my addr 172.21.15.146:0/1588203516 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4356a00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4358103ba0 0x7f43581a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4357e00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f435810c5c0 0x7f435819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4356a00640 1 -- 172.21.15.146:0/1588203516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f435810c5c0 msgr2=0x7f435819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4356a00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f435810c5c0 0x7f435819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4356a00640 1 -- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4358103280 msgr2=0x7f43581a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4356a00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4358103280 0x7f43581a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4356a00640 1 -- 172.21.15.146:0/1588203516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435819b090 con 0x7f4358103ba0 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.763+0000 7f4357400640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4358103280 0x7f43581a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.764+0000 7f4356a00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4358103ba0 0x7f43581a0ca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f434002aa40 tx=0x7f4340003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.764+0000 7f434fe00640 1 -- 172.21.15.146:0/1588203516 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434003f070 con 0x7f4358103ba0 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.764+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435819b320 con 0x7f4358103ba0 2024-04-24T08:28:31.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.764+0000 7f434fe00640 1 -- 172.21.15.146:0/1588203516 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4340031c90 con 0x7f4358103ba0 2024-04-24T08:28:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.764+0000 7f434fe00640 1 -- 172.21.15.146:0/1588203516 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340035030 con 0x7f4358103ba0 2024-04-24T08:28:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.764+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43581a7620 con 0x7f4358103ba0 2024-04-24T08:28:31.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.767+0000 7f4360db6740 1 Processor -- start 2024-04-24T08:28:31.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f434fe00640 1 -- 172.21.15.146:0/1588203516 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f4340042070 con 0x7f4358103ba0 2024-04-24T08:28:31.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4360db6740 1 -- start start 2024-04-24T08:28:31.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4360db6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c99090 0x563749c99470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4360db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4360db6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749ca6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4360db6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x563749bfee70 con 0x563749ca4080 2024-04-24T08:28:31.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4360db6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563749ba5380 con 0x563749c999b0 2024-04-24T08:28:31.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4360db6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563749bfae30 con 0x563749c99090 2024-04-24T08:28:31.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4357400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.768+0000 7f4357e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c99090 0x563749c99470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:56614/0 (socket says 172.21.15.146:56614) 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 -- 172.21.15.146:0/4059561833 learned_addr learned my addr 172.21.15.146:0/4059561833 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4356a00640 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749ca6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 -- 172.21.15.146:0/4059561833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c99090 msgr2=0x563749c99470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c99090 0x563749c99470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 -- 172.21.15.146:0/4059561833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 msgr2=0x563749ca6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749ca6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 -- 172.21.15.146:0/4059561833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637499d97e0 con 0x563749c999b0 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4357400640 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca3b40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4348011910 tx=0x7f4348011dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc3c43e680574ff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f434fe00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f433c0a3d20 0x7f433c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f434fe00640 1 -- 172.21.15.146:0/1588203516 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f43400f10b0 con 0x7f4358103ba0 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.769+0000 7f4347e00640 1 -- 172.21.15.146:0/4059561833 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348011330 con 0x563749c999b0 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4347e00640 1 -- 172.21.15.146:0/4059561833 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4348010430 con 0x563749c999b0 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4347e00640 1 -- 172.21.15.146:0/4059561833 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348019640 con 0x563749c999b0 2024-04-24T08:28:31.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4357e00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f433c0a3d20 0x7f433c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4360db6740 1 -- 172.21.15.146:0/4059561833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 msgr2=0x563749ca3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4360db6740 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca3b40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4348011910 tx=0x7f4348011dd0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4357e00640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f433c0a3d20 0x7f433c0a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4350009d20 tx=0x7f4350005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4360db6740 1 -- 172.21.15.146:0/4059561833 shutdown_connections 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4360db6740 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749ca6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4360db6740 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca3b40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4360db6740 1 --2- 172.21.15.146:0/4059561833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c99090 0x563749c99470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.770+0000 7f4360db6740 1 -- 172.21.15.146:0/4059561833 >> 172.21.15.146:0/4059561833 conn(0x563749c953b0 msgr2=0x563749c95790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.771+0000 7f4360db6740 1 -- 172.21.15.146:0/4059561833 shutdown_connections 2024-04-24T08:28:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.771+0000 7f4360db6740 1 -- 172.21.15.146:0/4059561833 wait complete. 2024-04-24T08:28:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.771+0000 7f4360db6740 1 -- start start 2024-04-24T08:28:31.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.771+0000 7f4360db6740 1 Processor -- start 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4360db6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c99090 0x563749ca0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4360db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4360db6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749ca1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4360db6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x563749ca6e70 con 0x563749c99090 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4360db6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563749bb8660 con 0x563749c999b0 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4360db6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563749ca6ff0 con 0x563749ca4080 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4360db6740 10 client.0 _ll_register_callbacks cb 0x563749ba6aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4357400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c99090 0x563749ca0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4356a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4357400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c99090 0x563749ca0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41338/0 (socket says 172.21.15.146:41338) 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4357400640 1 -- 172.21.15.146:0/3232794874 learned_addr learned my addr 172.21.15.146:0/3232794874 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4357e00640 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749ca1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4356a00640 1 -- 172.21.15.146:0/3232794874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 msgr2=0x563749ca1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4356a00640 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749ca1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4356a00640 1 -- 172.21.15.146:0/3232794874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c99090 msgr2=0x563749ca0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4356a00640 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c99090 0x563749ca0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.772+0000 7f4356a00640 1 -- 172.21.15.146:0/3232794874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563749a39f70 con 0x563749c999b0 2024-04-24T08:28:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4356a00640 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca1450 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f43400ee7f0 tx=0x7f4340031490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b3fbe9f7ef91635 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4344c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:28:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4344c00640 1 -- 172.21.15.146:0/3232794874 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340049070 con 0x563749c999b0 2024-04-24T08:28:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4348010430 con 0x563749c999b0 2024-04-24T08:28:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4344c00640 1 -- 172.21.15.146:0/3232794874 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f43400b9760 con 0x563749c999b0 2024-04-24T08:28:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4344c00640 1 -- 172.21.15.146:0/3232794874 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f43400bcc30 con 0x563749c999b0 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4360db6740 10 client.26879 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5637498d1170 con 0x563749c999b0 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.773+0000 7f4344c00640 1 -- 172.21.15.146:0/3232794874 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4348010430 con 0x563749c999b0 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4344c00640 1 -- 172.21.15.146:0/3232794874 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f43400bd350 con 0x563749c999b0 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4360db6740 10 client.26879 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4360db6740 10 client.26879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4360db6740 20 client.26879 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4360db6740 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749cb4190 0x563749cb6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4360db6740 4 client.26879 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:28:31.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563749c0c890 con 0x563749cb4190 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.774+0000 7f4357400640 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749cb4190 0x563749cb6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.775+0000 7f4357400640 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749cb4190 0x563749cb6620 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6960620ec83609b server_cookie=2567ed28fe673285 in_seq=0 out_seq=0 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.775+0000 7f4344c00640 10 client.26879 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4344c00640 1 -- 172.21.15.146:0/3232794874 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563749c0c890 con 0x563749cb4190 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4344c00640 10 client.26879 handle_command_reply: tid=0 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 client.26879 shutdown 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 msgr2=0x563749ca1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca1450 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f43400ee7f0 tx=0x7f4340031490 comp rx=0 tx=0).stop 2024-04-24T08:28:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 shutdown_connections 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749cb4190 0x563749cb6620 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749ca1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c999b0 0x563749ca1450 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 --2- 172.21.15.146:0/3232794874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c99090 0x563749ca0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4344c00640 0 client.26879 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 >> 172.21.15.146:0/3232794874 conn(0x563749c953b0 msgr2=0x563749c9b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 shutdown_connections 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 1 -- 172.21.15.146:0/3232794874 wait complete. 2024-04-24T08:28:31.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.776+0000 7f4360db6740 20 client.26879 trim_cache size 0 max 16384 2024-04-24T08:28:31.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 Processor -- start 2024-04-24T08:28:31.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 -- start start 2024-04-24T08:28:31.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749cafd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c999b0 0x563749c94390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749cbbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f43581b2840 con 0x563749c999b0 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43581b0110 con 0x563749c99090 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.796+0000 7f4360db6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43581adf80 con 0x563749ca4080 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749cbbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4356a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c999b0 0x563749c94390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749cbbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:55174/0 (socket says 172.21.15.146:55174) 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749cafd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357e00640 1 -- 172.21.15.146:0/2726309291 learned_addr learned my addr 172.21.15.146:0/2726309291 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357400640 1 -- 172.21.15.146:0/2726309291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 msgr2=0x563749cbbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357400640 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749cbbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357400640 1 -- 172.21.15.146:0/2726309291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c999b0 msgr2=0x563749c94390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357400640 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c999b0 0x563749c94390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357400640 1 -- 172.21.15.146:0/2726309291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563749c997e0 con 0x563749c99090 2024-04-24T08:28:31.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4356a00640 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c999b0 0x563749c94390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:31.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357e00640 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749cbbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:31.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.797+0000 7f4357400640 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749cafd30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4348010d00 tx=0x7f4348010f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc80bfca5f9c289 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4347400640 1 -- 172.21.15.146:0/2726309291 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434801f040 con 0x563749c99090 2024-04-24T08:28:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4347400640 1 -- 172.21.15.146:0/2726309291 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43480109c0 con 0x563749c99090 2024-04-24T08:28:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4347400640 1 -- 172.21.15.146:0/2726309291 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348004c30 con 0x563749c99090 2024-04-24T08:28:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 -- 172.21.15.146:0/2726309291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 msgr2=0x563749cafd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749cafd30 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4348010d00 tx=0x7f4348010f30 comp rx=0 tx=0).stop 2024-04-24T08:28:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 -- 172.21.15.146:0/2726309291 shutdown_connections 2024-04-24T08:28:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749ca4080 0x563749cbbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749c999b0 0x563749c94390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 --2- 172.21.15.146:0/2726309291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749cafd30 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 -- 172.21.15.146:0/2726309291 >> 172.21.15.146:0/2726309291 conn(0x563749c953b0 msgr2=0x563749c93c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 -- 172.21.15.146:0/2726309291 shutdown_connections 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.798+0000 7f4360db6740 1 -- 172.21.15.146:0/2726309291 wait complete. 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 -- start start 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 Processor -- start 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749ca8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c999b0 0x563749ca8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749d1bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f43581ae5d0 con 0x563749ca4080 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43581afa40 con 0x563749c99090 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.799+0000 7f4360db6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f43581aeca0 con 0x563749c999b0 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4360db6740 10 client.0 _ll_register_callbacks cb 0x7f4340009230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749ca8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749d1bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749ca8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:56638/0 (socket says 172.21.15.146:56638) 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 -- 172.21.15.146:0/3593415282 learned_addr learned my addr 172.21.15.146:0/3593415282 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4356a00640 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c999b0 0x563749ca8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 -- 172.21.15.146:0/3593415282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c999b0 msgr2=0x563749ca8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c999b0 0x563749ca8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 -- 172.21.15.146:0/3593415282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 msgr2=0x563749d1bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749d1bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 -- 172.21.15.146:0/3593415282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563749d1c5a0 con 0x563749c99090 2024-04-24T08:28:31.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4356a00640 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c999b0 0x563749ca8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4357400640 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749ca8590 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4348010d90 tx=0x7f4348010dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53b3c14d2de1a3b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.800+0000 7f4344c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.801+0000 7f4344c00640 1 -- 172.21.15.146:0/3593415282 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348010430 con 0x563749c99090 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.801+0000 7f4344c00640 1 -- 172.21.15.146:0/3593415282 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f434800fa60 con 0x563749c99090 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.801+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43480109c0 con 0x563749c99090 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.801+0000 7f4344c00640 1 -- 172.21.15.146:0/3593415282 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f434800fca0 con 0x563749c99090 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.801+0000 7f4360db6740 10 client.26885 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:31.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.801+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563749d23980 con 0x563749c99090 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.801+0000 7f4344c00640 1 -- 172.21.15.146:0/3593415282 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43480109c0 con 0x563749c99090 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4344c00640 1 -- 172.21.15.146:0/3593415282 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4348018070 con 0x563749c99090 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4360db6740 10 client.26885 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4360db6740 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4360db6740 20 client.26885 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4360db6740 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749d23ef0 0x563749d262e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4360db6740 4 client.26885 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:28:31.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563749c06700 con 0x563749d23ef0 2024-04-24T08:28:31.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4356a00640 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749d23ef0 0x563749d262e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:31.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4356a00640 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749d23ef0 0x563749d262e0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e9258fc39d85618 server_cookie=edb9df8a55224385 in_seq=0 out_seq=0 2024-04-24T08:28:31.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.802+0000 7f4344c00640 10 client.26885 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:31.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4344c00640 1 -- 172.21.15.146:0/3593415282 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563749c06700 con 0x563749d23ef0 2024-04-24T08:28:31.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4344c00640 10 client.26885 handle_command_reply: tid=0 2024-04-24T08:28:31.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 client.26885 shutdown 2024-04-24T08:28:31.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 msgr2=0x563749ca8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749ca8590 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4348010d90 tx=0x7f4348010dc0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 shutdown_connections 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x563749d23ef0 0x563749d262e0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563749c999b0 0x563749ca8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x563749ca4080 0x563749d1bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 --2- 172.21.15.146:0/3593415282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563749c99090 0x563749ca8590 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4344c00640 0 client.26885 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 >> 172.21.15.146:0/3593415282 conn(0x563749c953b0 msgr2=0x563749ca1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 shutdown_connections 2024-04-24T08:28:31.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 1 -- 172.21.15.146:0/3593415282 wait complete. 2024-04-24T08:28:31.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.804+0000 7f4360db6740 20 client.26885 trim_cache size 0 max 16384 2024-04-24T08:28:31.813 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:28:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:28:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "a30fdf47-d01f-4c45-b52f-5a983c51a3fe", 2024-04-24T08:28:31.813 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:28:31.814 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:28:31.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f433c0a3d20 msgr2=0x7f433c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f433c0a3d20 0x7f433c0a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4350009d20 tx=0x7f4350005f90 comp rx=0 tx=0).stop 2024-04-24T08:28:31.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4358103ba0 msgr2=0x7f43581a0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4358103ba0 0x7f43581a0ca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f434002aa40 tx=0x7f4340003b20 comp rx=0 tx=0).stop 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 shutdown_connections 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f433c0a3d20 0x7f433c0a61e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f435810c5c0 0x7f435819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4358103ba0 0x7f43581a0ca0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 --2- 172.21.15.146:0/1588203516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4358103280 0x7f43581a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 >> 172.21.15.146:0/1588203516 conn(0x7f4358078000 msgr2=0x7f4358108490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 shutdown_connections 2024-04-24T08:28:31.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:31.805+0000 7f435e000640 1 -- 172.21.15.146:0/1588203516 wait complete. 2024-04-24T08:28:31.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:28:31.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:28:32.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 -- 172.21.15.146:0/2995506163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 msgr2=0x7f7a141152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 --2- 172.21.15.146:0/2995506163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 0x7f7a141152d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009cd0 tx=0x7f7a0c02a8f0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 -- 172.21.15.146:0/2995506163 shutdown_connections 2024-04-24T08:28:32.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 --2- 172.21.15.146:0/2995506163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 0x7f7a141152d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 --2- 172.21.15.146:0/2995506163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a14076e50 0x7f7a141129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 --2- 172.21.15.146:0/2995506163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 0x7f7a14076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 -- 172.21.15.146:0/2995506163 >> 172.21.15.146:0/2995506163 conn(0x7f7a140ffd90 msgr2=0x7f7a141021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:32.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 -- 172.21.15.146:0/2995506163 shutdown_connections 2024-04-24T08:28:32.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.100+0000 7f7a1a200640 1 -- 172.21.15.146:0/2995506163 wait complete. 2024-04-24T08:28:32.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.101+0000 7f7a1a200640 1 Processor -- start 2024-04-24T08:28:32.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a1a200640 1 -- start start 2024-04-24T08:28:32.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 0x7f7a1419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a14076e50 0x7f7a1419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a1a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 0x7f7a141a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a1a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7a14118a40 con 0x7f7a14076e50 2024-04-24T08:28:32.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a1a200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141188c0 con 0x7f7a14076530 2024-04-24T08:28:32.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a1a200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a14118bc0 con 0x7f7a14112ee0 2024-04-24T08:28:32.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 0x7f7a1419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 0x7f7a1419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:56652/0 (socket says 172.21.15.146:56652) 2024-04-24T08:28:32.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a13400640 1 -- 172.21.15.146:0/239535710 learned_addr learned my addr 172.21.15.146:0/239535710 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:32.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.102+0000 7f7a13e00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 0x7f7a141a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a13e00640 1 -- 172.21.15.146:0/239535710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 msgr2=0x7f7a1419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a13e00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 0x7f7a1419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a13e00640 1 -- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a14076e50 msgr2=0x7f7a1419f4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:28:32.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a12a00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a14076e50 0x7f7a1419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a13e00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a14076e50 0x7f7a1419f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a13e00640 1 -- 172.21.15.146:0/239535710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a141a3f50 con 0x7f7a14112ee0 2024-04-24T08:28:32.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a13400640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 0x7f7a1419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:28:32.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.103+0000 7f7a13e00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 0x7f7a141a3820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c02aee0 tx=0x7f7a0c030b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:32.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.104+0000 7f7a0be00640 1 -- 172.21.15.146:0/239535710 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c003e20 con 0x7f7a14112ee0 2024-04-24T08:28:32.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.104+0000 7f7a0be00640 1 -- 172.21.15.146:0/239535710 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a0c040070 con 0x7f7a14112ee0 2024-04-24T08:28:32.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.104+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a141a41e0 con 0x7f7a14112ee0 2024-04-24T08:28:32.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.104+0000 7f7a0be00640 1 -- 172.21.15.146:0/239535710 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c045b10 con 0x7f7a14112ee0 2024-04-24T08:28:32.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.104+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a141abad0 con 0x7f7a14112ee0 2024-04-24T08:28:32.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.109+0000 7f7a0be00640 1 -- 172.21.15.146:0/239535710 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f7a0c045cd0 con 0x7f7a14112ee0 2024-04-24T08:28:32.111 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.111+0000 7f7a1cffd740 1 Processor -- start 2024-04-24T08:28:32.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.111+0000 7f7a1cffd740 1 -- start start 2024-04-24T08:28:32.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.111+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf3860 0x55fd7ecf3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ed00c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fd7ec591f0 con 0x55fd7ecf4180 2024-04-24T08:28:32.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd7ebff350 con 0x55fd7ecf3860 2024-04-24T08:28:32.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd7ec551b0 con 0x55fd7ecfe850 2024-04-24T08:28:32.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a0be00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f79f80a3de0 0x7f79f80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf3860 0x55fd7ecf3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf3860 0x55fd7ecf3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:56664/0 (socket says 172.21.15.146:56664) 2024-04-24T08:28:32.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 -- 172.21.15.146:0/2116882806 learned_addr learned my addr 172.21.15.146:0/2116882806 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:32.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a0be00640 1 -- 172.21.15.146:0/239535710 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f7a0c0f1850 con 0x7f7a14112ee0 2024-04-24T08:28:32.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ed00c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a13400640 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 -- 172.21.15.146:0/2116882806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 msgr2=0x55fd7ed00c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ed00c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 -- 172.21.15.146:0/2116882806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 msgr2=0x55fd7ecfe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfe310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.112+0000 7f7a12a00640 1 -- 172.21.15.146:0/2116882806 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd7ec53f50 con 0x55fd7ecf3860 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.113+0000 7f7a13e00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f79f80a3de0 0x7f79f80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.113+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf3860 0x55fd7ecf3c40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7a00005fd0 tx=0x7f7a0000e390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c1ccc4d7ce06e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.114+0000 7f7a13e00640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f79f80a3de0 0x7f79f80a62a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0ba2c0 tx=0x7f7a0c005ef0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:32.116 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.114+0000 7f79ffe00640 1 -- 172.21.15.146:0/2116882806 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000b660 con 0x55fd7ecf3860 2024-04-24T08:28:32.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.114+0000 7f79ffe00640 1 -- 172.21.15.146:0/2116882806 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a0000bce0 con 0x55fd7ecf3860 2024-04-24T08:28:32.117 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.114+0000 7f79ffe00640 1 -- 172.21.15.146:0/2116882806 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a00011e10 con 0x55fd7ecf3860 2024-04-24T08:28:32.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2116882806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf3860 msgr2=0x55fd7ecf3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf3860 0x55fd7ecf3c40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7a00005fd0 tx=0x7f7a0000e390 comp rx=0 tx=0).stop 2024-04-24T08:28:32.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2116882806 shutdown_connections 2024-04-24T08:28:32.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ed00c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.118 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2116882806 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf3860 0x55fd7ecf3c40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2116882806 >> 172.21.15.146:0/2116882806 conn(0x55fd7ecefb80 msgr2=0x55fd7eceff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:32.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.115+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2116882806 shutdown_connections 2024-04-24T08:28:32.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.116+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2116882806 wait complete. 2024-04-24T08:28:32.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.116+0000 7f7a1cffd740 1 -- start start 2024-04-24T08:28:32.119 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.117+0000 7f7a1cffd740 1 Processor -- start 2024-04-24T08:28:32.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ecfc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55fd7ed015e0 con 0x55fd7ecf3860 2024-04-24T08:28:32.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fd7ec12690 con 0x55fd7ecf4180 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd7ed01760 con 0x55fd7ecfe850 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a1cffd740 10 client.0 _ll_register_callbacks cb 0x55fd7ec4e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a12a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a12a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41380/0 (socket says 172.21.15.146:41380) 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a12a00640 1 -- 172.21.15.146:0/832190210 learned_addr learned my addr 172.21.15.146:0/832190210 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a12a00640 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ecfc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.118+0000 7f7a13400640 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f7a12a00640 1 -- 172.21.15.146:0/832190210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 msgr2=0x55fd7ecfc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f7a12a00640 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ecfc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f7a12a00640 1 -- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 msgr2=0x55fd7ecfbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f7a12a00640 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f7a12a00640 1 -- 172.21.15.146:0/832190210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd7ec11920 con 0x55fd7ecf3860 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f7a12a00640 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfb7d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000c010 tx=0x7f7a0000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2beeb861db90f65 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f79fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f79fcc00640 1 -- 172.21.15.146:0/832190210 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000b660 con 0x55fd7ecf3860 2024-04-24T08:28:32.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f79fcc00640 1 -- 172.21.15.146:0/832190210 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f7a0000b800 con 0x55fd7ecf3860 2024-04-24T08:28:32.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.119+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a0000bce0 con 0x55fd7ecf3860 2024-04-24T08:28:32.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.120+0000 7f79fcc00640 1 -- 172.21.15.146:0/832190210 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7a0000ba00 con 0x55fd7ecf3860 2024-04-24T08:28:32.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.120+0000 7f7a1cffd740 10 client.16743 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:32.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.120+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fd7ea357d0 con 0x55fd7ecf3860 2024-04-24T08:28:32.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.120+0000 7f79fcc00640 1 -- 172.21.15.146:0/832190210 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a0000bce0 con 0x55fd7ecf3860 2024-04-24T08:28:32.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f79fcc00640 1 -- 172.21.15.146:0/832190210 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7a0000f040 con 0x55fd7ecf3860 2024-04-24T08:28:32.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f7a1cffd740 10 client.16743 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:32.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f7a1cffd740 10 client.16743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:32.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f7a1cffd740 20 client.16743 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:32.126 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed0ead0 0x55fd7ed10ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f7a1cffd740 4 client.16743 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd7ec66c10 con 0x55fd7ed0ead0 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.121+0000 7f7a13400640 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed0ead0 0x55fd7ed10ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.122+0000 7f79fcc00640 10 client.16743 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.122+0000 7f7a13400640 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed0ead0 0x55fd7ed10ec0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95c020c51a50ad18 server_cookie=e853c9788854a6c7 in_seq=0 out_seq=0 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f79fcc00640 1 -- 172.21.15.146:0/832190210 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd7ec66c10 con 0x55fd7ed0ead0 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f79fcc00640 10 client.16743 handle_command_reply: tid=0 2024-04-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 client.16743 shutdown 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf3860 msgr2=0x55fd7ecfb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfb7d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000c010 tx=0x7f7a0000eba0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 shutdown_connections 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed0ead0 0x55fd7ed10ec0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecfe850 0x55fd7ecfc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ecfbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/832190210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfb7d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 >> 172.21.15.146:0/832190210 conn(0x55fd7ecefb80 msgr2=0x55fd7ecf5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:32.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.123+0000 7f79fcc00640 0 client.16743 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:32.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.124+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 shutdown_connections 2024-04-24T08:28:32.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.124+0000 7f7a1cffd740 1 -- 172.21.15.146:0/832190210 wait complete. 2024-04-24T08:28:32.129 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.124+0000 7f7a1cffd740 20 client.16743 trim_cache size 0 max 16384 2024-04-24T08:28:32.148 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.146+0000 7f7a1cffd740 1 Processor -- start 2024-04-24T08:28:32.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.146+0000 7f7a1cffd740 1 -- start start 2024-04-24T08:28:32.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ed0a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecfe850 0x55fd7ed16630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141b6c50 con 0x55fd7ecfe850 2024-04-24T08:28:32.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141b4520 con 0x55fd7ecf4180 2024-04-24T08:28:32.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141b2390 con 0x55fd7ecf3860 2024-04-24T08:28:32.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ed0a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a13400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ed0a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:56674/0 (socket says 172.21.15.146:56674) 2024-04-24T08:28:32.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a13400640 1 -- 172.21.15.146:0/474530388 learned_addr learned my addr 172.21.15.146:0/474530388 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:32.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a12a00640 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.147+0000 7f7a12a00640 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecfe850 0x55fd7ed16630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f7a13400640 1 -- 172.21.15.146:0/474530388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 msgr2=0x55fd7ecfa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f7a13400640 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f7a13400640 1 -- 172.21.15.146:0/474530388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecfe850 msgr2=0x55fd7ed16630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f7a13400640 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecfe850 0x55fd7ed16630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f7a13400640 1 -- 172.21.15.146:0/474530388 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd7e96b3f0 con 0x55fd7ecf4180 2024-04-24T08:28:32.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f7a13400640 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ed0a4d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7a04012620 tx=0x7f7a04014c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29cb1ee085847c90 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:32.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f79ff400640 1 -- 172.21.15.146:0/474530388 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04014310 con 0x55fd7ecf4180 2024-04-24T08:28:32.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f79ff400640 1 -- 172.21.15.146:0/474530388 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a040144b0 con 0x55fd7ecf4180 2024-04-24T08:28:32.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.148+0000 7f79ff400640 1 -- 172.21.15.146:0/474530388 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0401c610 con 0x55fd7ecf4180 2024-04-24T08:28:32.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 -- 172.21.15.146:0/474530388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 msgr2=0x55fd7ed0a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ed0a4d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7a04012620 tx=0x7f7a04014c90 comp rx=0 tx=0).stop 2024-04-24T08:28:32.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 -- 172.21.15.146:0/474530388 shutdown_connections 2024-04-24T08:28:32.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecfe850 0x55fd7ed16630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecf4180 0x55fd7ed0a4d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/474530388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ecfa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 -- 172.21.15.146:0/474530388 >> 172.21.15.146:0/474530388 conn(0x55fd7ecefb80 msgr2=0x55fd7ecfc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:32.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 -- 172.21.15.146:0/474530388 shutdown_connections 2024-04-24T08:28:32.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.149+0000 7f7a1cffd740 1 -- 172.21.15.146:0/474530388 wait complete. 2024-04-24T08:28:32.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 -- start start 2024-04-24T08:28:32.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 Processor -- start 2024-04-24T08:28:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ed02ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ed03220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecfe850 0x55fd7ed6dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141b29e0 con 0x55fd7ecf4180 2024-04-24T08:28:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141b3e50 con 0x55fd7ecfe850 2024-04-24T08:28:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7a141b30b0 con 0x55fd7ecf3860 2024-04-24T08:28:32.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.150+0000 7f7a1cffd740 10 client.0 _ll_register_callbacks cb 0x7f7a0000edb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:28:32.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ed02ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ed02ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:55260/0 (socket says 172.21.15.146:55260) 2024-04-24T08:28:32.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 -- 172.21.15.146:0/2014441707 learned_addr learned my addr 172.21.15.146:0/2014441707 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:28:32.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecfe850 0x55fd7ed6dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a13400640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ed03220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 -- 172.21.15.146:0/2014441707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 msgr2=0x55fd7ed02ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ed02ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 -- 172.21.15.146:0/2014441707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 msgr2=0x55fd7ed03220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:28:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ed03220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 -- 172.21.15.146:0/2014441707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd7e92b0f0 con 0x55fd7ecfe850 2024-04-24T08:28:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a13400640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ed03220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:28:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a12a00640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecfe850 0x55fd7ed6dee0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7a00005370 tx=0x7f7a0000ea90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d363c6dea99ec123 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:28:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f79fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:28:32.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f79fcc00640 1 -- 172.21.15.146:0/2014441707 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0003e060 con 0x55fd7ecfe850 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.151+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a040144b0 con 0x55fd7ecfe850 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f79fcc00640 1 -- 172.21.15.146:0/2014441707 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f7a000119d0 con 0x55fd7ecfe850 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f79fcc00640 1 -- 172.21.15.146:0/2014441707 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7a0000b840 con 0x55fd7ecfe850 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 10 client.26903 fetch_fsmap learned FSMap version 26 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fd7ec1e810 con 0x55fd7ecfe850 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f79fcc00640 1 -- 172.21.15.146:0/2014441707 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a040144b0 con 0x55fd7ecfe850 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f79fcc00640 1 -- 172.21.15.146:0/2014441707 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7a0001e420 con 0x55fd7ecfe850 2024-04-24T08:28:32.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 10 client.26903 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:28:32.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 10 client.26903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:28:32.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 20 client.26903 populate_metadata read hostname 'smithi146' 2024-04-24T08:28:32.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed75b50 0x55fd7ed77f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:28:32.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 4 client.26903 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:28:32.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fd7ec60a80 con 0x55fd7ed75b50 2024-04-24T08:28:32.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.152+0000 7f7a13400640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed75b50 0x55fd7ed77f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:28:32.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f7a13400640 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed75b50 0x55fd7ed77f40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28b37f239df297cf server_cookie=b304ac6d23986353 in_seq=0 out_seq=0 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f79fcc00640 10 client.26903 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f79fcc00640 1 -- 172.21.15.146:0/2014441707 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fd7ec60a80 con 0x55fd7ed75b50 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f79fcc00640 10 client.26903 handle_command_reply: tid=0 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f7a1cffd740 1 client.26903 shutdown 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecfe850 msgr2=0x55fd7ed6dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecfe850 0x55fd7ed6dee0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7a00005370 tx=0x7f7a0000ea90 comp rx=0 tx=0).stop 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 shutdown_connections 2024-04-24T08:28:32.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55fd7ed75b50 0x55fd7ed77f40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f79fcc00640 0 client.26903 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.153+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fd7ecfe850 0x55fd7ed6dee0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55fd7ecf4180 0x55fd7ed03220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1cffd740 1 --2- 172.21.15.146:0/2014441707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd7ecf3860 0x55fd7ed02ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 >> 172.21.15.146:0/2014441707 conn(0x55fd7ecefb80 msgr2=0x55fd7ecee390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 shutdown_connections 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1cffd740 1 -- 172.21.15.146:0/2014441707 wait complete. 2024-04-24T08:28:32.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1cffd740 20 client.26903 trim_cache size 0 max 16384 2024-04-24T08:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:28:32.166 INFO:teuthology.orchestra.run.smithi146.stdout: "a30fdf47-d01f-4c45-b52f-5a983c51a3fe": { 2024-04-24T08:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "a30fdf47-d01f-4c45-b52f-5a983c51a3fe", 2024-04-24T08:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:28:32.167 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:28:32.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f79f80a3de0 msgr2=0x7f79f80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1a200640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f79f80a3de0 0x7f79f80a62a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0ba2c0 tx=0x7f7a0c005ef0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 msgr2=0x7f7a141a3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:28:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.154+0000 7f7a1a200640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 0x7f7a141a3820 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c02aee0 tx=0x7f7a0c030b60 comp rx=0 tx=0).stop 2024-04-24T08:28:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 shutdown_connections 2024-04-24T08:28:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f79f80a3de0 0x7f79f80a62a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a14112ee0 0x7f7a141a3820 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a14076e50 0x7f7a1419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 --2- 172.21.15.146:0/239535710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a14076530 0x7f7a1419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:28:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 >> 172.21.15.146:0/239535710 conn(0x7f7a140ffd90 msgr2=0x7f7a14114a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:28:32.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 shutdown_connections 2024-04-24T08:28:32.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:28:32.155+0000 7f7a1a200640 1 -- 172.21.15.146:0/239535710 wait complete. 2024-04-24T08:28:32.173 INFO:tasks.cephfs.filesystem:scrub status for tag:a30fdf47-d01f-4c45-b52f-5a983c51a3fe - {'path': '/', 'tag': 'a30fdf47-d01f-4c45-b52f-5a983c51a3fe', 'options': 'recursive,force'} 2024-04-24T08:28:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: pgmap v225: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:28:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:28:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:28:33.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:28:33.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:32 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: pgmap v225: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:33.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:28:33.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:28:33.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:28:33.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:28:33.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:32 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: pgmap v225: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:33.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:28:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:28:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:28:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:28:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:32 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:28:34.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:33 smithi153 ceph-mon[32746]: pgmap v226: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:34.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:33 smithi146 ceph-mon[27805]: pgmap v226: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:34.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:33 smithi177 ceph-mon[33046]: pgmap v226: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:36 smithi153 ceph-mon[32746]: pgmap v227: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:36.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:36 smithi146 ceph-mon[27805]: pgmap v227: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:36 smithi177 ceph-mon[33046]: pgmap v227: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:37.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:37 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:28:37.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:37 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:28:37.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:37 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:28:37.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:37 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:28:37.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:37 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:28:37.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:37 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:28:38.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:38 smithi153 ceph-mon[32746]: pgmap v228: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:38.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:38 smithi146 ceph-mon[27805]: pgmap v228: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:38 smithi177 ceph-mon[33046]: pgmap v228: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:40 smithi153 ceph-mon[32746]: pgmap v229: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:40.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:40 smithi146 ceph-mon[27805]: pgmap v229: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:40 smithi177 ceph-mon[33046]: pgmap v229: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:42.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:42 smithi153 ceph-mon[32746]: pgmap v230: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:42.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:42 smithi146 ceph-mon[27805]: pgmap v230: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:42 smithi177 ceph-mon[33046]: pgmap v230: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:28:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:44 smithi153 ceph-mon[32746]: pgmap v231: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:44.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:44 smithi146 ceph-mon[27805]: pgmap v231: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:44.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:44 smithi177 ceph-mon[33046]: pgmap v231: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:28:46.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:46 smithi153 ceph-mon[32746]: pgmap v232: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:46.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:46 smithi146 ceph-mon[27805]: pgmap v232: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:46 smithi177 ceph-mon[33046]: pgmap v232: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:48.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:48 smithi153 ceph-mon[32746]: pgmap v233: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:48.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:48 smithi146 ceph-mon[27805]: pgmap v233: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:48 smithi177 ceph-mon[33046]: pgmap v233: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:50.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:50 smithi153 ceph-mon[32746]: pgmap v234: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:50.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:50 smithi146 ceph-mon[27805]: pgmap v234: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:50 smithi177 ceph-mon[33046]: pgmap v234: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:52.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:52 smithi153 ceph-mon[32746]: pgmap v235: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:52 smithi146 ceph-mon[27805]: pgmap v235: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:52 smithi177 ceph-mon[33046]: pgmap v235: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:54.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:54 smithi153 ceph-mon[32746]: pgmap v236: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:54.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:54 smithi146 ceph-mon[27805]: pgmap v236: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:54 smithi177 ceph-mon[33046]: pgmap v236: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:56 smithi153 ceph-mon[32746]: pgmap v237: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:56.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:56 smithi146 ceph-mon[27805]: pgmap v237: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:56.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:56 smithi177 ceph-mon[33046]: pgmap v237: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:28:58.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:28:58 smithi153 ceph-mon[32746]: pgmap v238: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:28:58 smithi146 ceph-mon[27805]: pgmap v238: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:28:58.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:28:58 smithi177 ceph-mon[33046]: pgmap v238: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:00.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:00 smithi153 ceph-mon[32746]: pgmap v239: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:00 smithi146 ceph-mon[27805]: pgmap v239: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:00.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:00 smithi177 ceph-mon[33046]: pgmap v239: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:02.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:29:02.175 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:29:02.392 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.390+0000 7f009f200640 1 -- 172.21.15.146:0/2178174778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a0112c60 msgr2=0x7f00a0115050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.390+0000 7f009f200640 1 --2- 172.21.15.146:0/2178174778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a0112c60 0x7f00a0115050 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0098009ca0 tx=0x7f009802a950 comp rx=0 tx=0).stop 2024-04-24T08:29:02.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.390+0000 7f009f200640 1 -- 172.21.15.146:0/2178174778 shutdown_connections 2024-04-24T08:29:02.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.390+0000 7f009f200640 1 --2- 172.21.15.146:0/2178174778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a0112c60 0x7f00a0115050 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.390+0000 7f009f200640 1 --2- 172.21.15.146:0/2178174778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a01010f0 0x7f00a0112720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.390+0000 7f009f200640 1 --2- 172.21.15.146:0/2178174778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a01007d0 0x7f00a0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.390+0000 7f009f200640 1 -- 172.21.15.146:0/2178174778 >> 172.21.15.146:0/2178174778 conn(0x7f00a0076120 msgr2=0x7f00a0076530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.391+0000 7f009f200640 1 -- 172.21.15.146:0/2178174778 shutdown_connections 2024-04-24T08:29:02.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.391+0000 7f009f200640 1 -- 172.21.15.146:0/2178174778 wait complete. 2024-04-24T08:29:02.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 Processor -- start 2024-04-24T08:29:02.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 -- start start 2024-04-24T08:29:02.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a01007d0 0x7f00a019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a01010f0 0x7f00a019f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a0112c60 0x7f00a01a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f00a0118880 con 0x7f00a0112c60 2024-04-24T08:29:02.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00a0118700 con 0x7f00a01010f0 2024-04-24T08:29:02.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.392+0000 7f009f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f00a0118a00 con 0x7f00a01007d0 2024-04-24T08:29:02.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a01007d0 0x7f00a019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a0112c60 0x7f00a01a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f0095400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a01010f0 0x7f00a019f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a01007d0 0x7f00a019ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57572/0 (socket says 172.21.15.146:57572) 2024-04-24T08:29:02.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 -- 172.21.15.146:0/846437209 learned_addr learned my addr 172.21.15.146:0/846437209 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 -- 172.21.15.146:0/846437209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a01010f0 msgr2=0x7f00a019f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a01010f0 0x7f00a019f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 -- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a0112c60 msgr2=0x7f00a01a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a0112c60 0x7f00a01a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 -- 172.21.15.146:0/846437209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a01a3d10 con 0x7f00a01007d0 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.393+0000 7f009de00640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a01007d0 0x7f00a019ed20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f009000e9f0 tx=0x7f009000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.394+0000 7f0097400640 1 -- 172.21.15.146:0/846437209 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009000cd80 con 0x7f00a01007d0 2024-04-24T08:29:02.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.394+0000 7f0097400640 1 -- 172.21.15.146:0/846437209 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0090005500 con 0x7f00a01007d0 2024-04-24T08:29:02.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.394+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a01a4000 con 0x7f00a01007d0 2024-04-24T08:29:02.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.394+0000 7f0097400640 1 -- 172.21.15.146:0/846437209 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090010690 con 0x7f00a01007d0 2024-04-24T08:29:02.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.394+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a01ab890 con 0x7f00a01007d0 2024-04-24T08:29:02.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 Processor -- start 2024-04-24T08:29:02.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 -- start start 2024-04-24T08:29:02.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb0fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb1a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb1ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56249ba751f0 con 0x56249bb10350 2024-04-24T08:29:02.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56249ba1b350 con 0x56249bb1aa20 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f00a7a55740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56249ba711b0 con 0x56249bb0fa30 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.397+0000 7f0097400640 1 -- 172.21.15.146:0/846437209 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f00900046c0 con 0x7f00a01007d0 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb0fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb0fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57578/0 (socket says 172.21.15.146:57578) 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 -- 172.21.15.146:0/3077927737 learned_addr learned my addr 172.21.15.146:0/3077927737 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f0095400640 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb1ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009de00640 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb1a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 -- 172.21.15.146:0/3077927737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 msgr2=0x56249bb1ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb1ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 -- 172.21.15.146:0/3077927737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 msgr2=0x56249bb1a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb1a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.398+0000 7f009e800640 1 -- 172.21.15.146:0/3077927737 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56249ba6ff50 con 0x56249bb0fa30 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.399+0000 7f009e800640 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb0fe10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0098009ca0 tx=0x7f00980093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc371ab0ba996a7b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.399+0000 7f008f000640 1 -- 172.21.15.146:0/3077927737 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098048070 con 0x56249bb0fa30 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.399+0000 7f008f000640 1 -- 172.21.15.146:0/3077927737 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f009802f940 con 0x56249bb0fa30 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.399+0000 7f008f000640 1 -- 172.21.15.146:0/3077927737 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098034040 con 0x56249bb0fa30 2024-04-24T08:29:02.404 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.399+0000 7f0097400640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f00840a3d20 0x7f00840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.399+0000 7f0097400640 1 -- 172.21.15.146:0/846437209 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f00900d0c00 con 0x7f00a01007d0 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.399+0000 7f009de00640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f00840a3d20 0x7f00840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 -- 172.21.15.146:0/3077927737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 msgr2=0x56249bb0fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb0fe10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0098009ca0 tx=0x7f00980093e0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 -- 172.21.15.146:0/3077927737 shutdown_connections 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb1ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb1a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.405 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 --2- 172.21.15.146:0/3077927737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb0fe10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 -- 172.21.15.146:0/3077927737 >> 172.21.15.146:0/3077927737 conn(0x56249bb0bd50 msgr2=0x56249bb0c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 -- 172.21.15.146:0/3077927737 shutdown_connections 2024-04-24T08:29:02.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 -- 172.21.15.146:0/3077927737 wait complete. 2024-04-24T08:29:02.406 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f009de00640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f00840a3d20 0x7f00840a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f00900ce3f0 tx=0x7f0090004a40 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.400+0000 7f00a7a55740 1 -- start start 2024-04-24T08:29:02.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 1 Processor -- start 2024-04-24T08:29:02.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb17a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb17f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb18490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x56249bb1d7b0 con 0x56249bb10350 2024-04-24T08:29:02.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56249ba2e690 con 0x56249bb1aa20 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56249bb1d930 con 0x56249bb0fa30 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f00a7a55740 10 client.0 _ll_register_callbacks cb 0x56249ba6a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f0095400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb17f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f0095400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb17f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40284/0 (socket says 172.21.15.146:40284) 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f0095400640 1 -- 172.21.15.146:0/2929322144 learned_addr learned my addr 172.21.15.146:0/2929322144 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.401+0000 7f009e800640 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb17a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f009e800640 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb18490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0095400640 1 -- 172.21.15.146:0/2929322144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 msgr2=0x56249bb17a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0095400640 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb17a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0095400640 1 -- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 msgr2=0x56249bb18490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0095400640 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb18490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0095400640 1 -- 172.21.15.146:0/2929322144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56249ba2d920 con 0x56249bb10350 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0095400640 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb17f50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0088012700 tx=0x7f0088012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e82f081b2f69e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0083400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0083400640 1 -- 172.21.15.146:0/2929322144 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008800f8c0 con 0x56249bb10350 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.402+0000 7f0083400640 1 -- 172.21.15.146:0/2929322144 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f008800fa60 con 0x56249bb10350 2024-04-24T08:29:02.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f009802f940 con 0x56249bb10350 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f0083400640 1 -- 172.21.15.146:0/2929322144 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f008800fc60 con 0x56249bb10350 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 10 client.16761 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56249b8517d0 con 0x56249bb10350 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f0083400640 1 -- 172.21.15.146:0/2929322144 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f009802f940 con 0x56249bb10350 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f0083400640 1 -- 172.21.15.146:0/2929322144 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0088010530 con 0x56249bb10350 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 10 client.16761 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 10 client.16761 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:02.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 20 client.16761 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb2aea0 0x56249bb2d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 4 client.16761 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.403+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56249ba82c10 con 0x56249bb2aea0 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.404+0000 7f009e800640 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb2aea0 0x56249bb2d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.404+0000 7f009e800640 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb2aea0 0x56249bb2d2c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffdc4e9fb1efbb2 server_cookie=8edd7b86a23de677 in_seq=0 out_seq=0 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.404+0000 7f0083400640 10 client.16761 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.405+0000 7f0083400640 1 -- 172.21.15.146:0/2929322144 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56249ba82c10 con 0x56249bb2aea0 2024-04-24T08:29:02.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.405+0000 7f0083400640 10 client.16761 handle_command_reply: tid=0 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 client.16761 shutdown 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 msgr2=0x56249bb17f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb17f50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0088012700 tx=0x7f0088012bc0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 shutdown_connections 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb2aea0 0x56249bb2d2c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb18490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb10350 0x56249bb17f50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 --2- 172.21.15.146:0/2929322144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb0fa30 0x56249bb17a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f0083400640 0 client.16761 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:02.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 >> 172.21.15.146:0/2929322144 conn(0x56249bb0bd50 msgr2=0x56249bb11b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 shutdown_connections 2024-04-24T08:29:02.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 1 -- 172.21.15.146:0/2929322144 wait complete. 2024-04-24T08:29:02.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.406+0000 7f00a7a55740 20 client.16761 trim_cache size 0 max 16384 2024-04-24T08:29:02.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 Processor -- start 2024-04-24T08:29:02.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 -- start start 2024-04-24T08:29:02.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb0fa30 0x56249bb0afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb182b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb32bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f00a01b69c0 con 0x56249bb0fa30 2024-04-24T08:29:02.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00a01b4290 con 0x56249bb1aa20 2024-04-24T08:29:02.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.426+0000 7f00a7a55740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f00a01b2100 con 0x56249bb10350 2024-04-24T08:29:02.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb182b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb182b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57594/0 (socket says 172.21.15.146:57594) 2024-04-24T08:29:02.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 -- 172.21.15.146:0/620049285 learned_addr learned my addr 172.21.15.146:0/620049285 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:02.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f009e800640 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb0fa30 0x56249bb0afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 -- 172.21.15.146:0/620049285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 msgr2=0x56249bb32bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f009e800640 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb32bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb32bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 -- 172.21.15.146:0/620049285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb0fa30 msgr2=0x56249bb0afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb0fa30 0x56249bb0afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f0095400640 1 -- 172.21.15.146:0/620049285 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56249b7873f0 con 0x56249bb10350 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.427+0000 7f009e800640 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb32bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.428+0000 7f0095400640 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb182b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f00880109d0 tx=0x7f008800ff90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f3e089ee6599550 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.428+0000 7f008e600640 1 -- 172.21.15.146:0/620049285 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088028030 con 0x56249bb10350 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.428+0000 7f008e600640 1 -- 172.21.15.146:0/620049285 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0088005550 con 0x56249bb10350 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.428+0000 7f008e600640 1 -- 172.21.15.146:0/620049285 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088010430 con 0x56249bb10350 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 -- 172.21.15.146:0/620049285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 msgr2=0x56249bb182b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb182b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f00880109d0 tx=0x7f008800ff90 comp rx=0 tx=0).stop 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 -- 172.21.15.146:0/620049285 shutdown_connections 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb182b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb1aa20 0x56249bb32bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 --2- 172.21.15.146:0/620049285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb0fa30 0x56249bb0afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 -- 172.21.15.146:0/620049285 >> 172.21.15.146:0/620049285 conn(0x56249bb0bd50 msgr2=0x56249bb11180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 -- 172.21.15.146:0/620049285 shutdown_connections 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 -- 172.21.15.146:0/620049285 wait complete. 2024-04-24T08:29:02.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.429+0000 7f00a7a55740 1 -- start start 2024-04-24T08:29:02.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 1 Processor -- start 2024-04-24T08:29:02.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb0fa30 0x56249bb14a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb14fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb1aa20 0x56249bb90440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f00a01b2750 con 0x56249bb1aa20 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00a01b3bc0 con 0x56249bb0fa30 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f00a01b2e20 con 0x56249bb10350 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f00a7a55740 10 client.0 _ll_register_callbacks cb 0x7f0088009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f009e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb0fa30 0x56249bb14a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f0095400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb14fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f0095400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb14fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57602/0 (socket says 172.21.15.146:57602) 2024-04-24T08:29:02.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.430+0000 7f009e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb0fa30 0x56249bb14a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33028/0 (socket says 172.21.15.146:33028) 2024-04-24T08:29:02.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0095400640 1 -- 172.21.15.146:0/934467097 learned_addr learned my addr 172.21.15.146:0/934467097 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:02.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f009e800640 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb1aa20 0x56249bb90440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0095400640 1 -- 172.21.15.146:0/934467097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb0fa30 msgr2=0x56249bb14a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0095400640 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb0fa30 0x56249bb14a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0095400640 1 -- 172.21.15.146:0/934467097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb1aa20 msgr2=0x56249bb90440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0095400640 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb1aa20 0x56249bb90440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0095400640 1 -- 172.21.15.146:0/934467097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56249b7470f0 con 0x56249bb10350 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0095400640 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb14fb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0088010a70 tx=0x7f008800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1f4cc1635350416 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0083400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0083400640 1 -- 172.21.15.146:0/934467097 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088031020 con 0x56249bb10350 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f0083400640 1 -- 172.21.15.146:0/934467097 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f0088005110 con 0x56249bb10350 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.431+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0088005550 con 0x56249bb10350 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f0083400640 1 -- 172.21.15.146:0/934467097 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0088028040 con 0x56249bb10350 2024-04-24T08:29:02.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 10 client.26905 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:02.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56249ba3a810 con 0x56249bb10350 2024-04-24T08:29:02.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f0083400640 1 -- 172.21.15.146:0/934467097 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0088005550 con 0x56249bb10350 2024-04-24T08:29:02.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f0083400640 1 -- 172.21.15.146:0/934467097 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f008801c470 con 0x56249bb10350 2024-04-24T08:29:02.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 10 client.26905 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:02.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 10 client.26905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:02.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 20 client.26905 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:02.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb91d30 0x56249bb94150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 4 client.26905 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.432+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56249ba7ca80 con 0x56249bb91d30 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.433+0000 7f009e800640 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb91d30 0x56249bb94150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.433+0000 7f009e800640 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb91d30 0x56249bb94150 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ef0e680b79705ae server_cookie=a4578e2fcf141dfe in_seq=0 out_seq=0 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.433+0000 7f0083400640 10 client.26905 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f0083400640 1 -- 172.21.15.146:0/934467097 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56249ba7ca80 con 0x56249bb91d30 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f0083400640 10 client.26905 handle_command_reply: tid=0 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 client.26905 shutdown 2024-04-24T08:29:02.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 msgr2=0x56249bb14fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb14fb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0088010a70 tx=0x7f008800f790 comp rx=0 tx=0).stop 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 shutdown_connections 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56249bb91d30 0x56249bb94150 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56249bb10350 0x56249bb14fb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56249bb1aa20 0x56249bb90440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 --2- 172.21.15.146:0/934467097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56249bb0fa30 0x56249bb14a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f0083400640 0 client.26905 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:02.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 >> 172.21.15.146:0/934467097 conn(0x56249bb0bd50 msgr2=0x56249bb0ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 shutdown_connections 2024-04-24T08:29:02.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 1 -- 172.21.15.146:0/934467097 wait complete. 2024-04-24T08:29:02.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.434+0000 7f00a7a55740 20 client.26905 trim_cache size 0 max 16384 2024-04-24T08:29:02.445 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:29:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:29:02.445 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:29:02.446 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:29:02.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f00840a3d20 msgr2=0x7f00840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f00840a3d20 0x7f00840a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f00900ce3f0 tx=0x7f0090004a40 comp rx=0 tx=0).stop 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a01007d0 msgr2=0x7f00a019ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a01007d0 0x7f00a019ed20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f009000e9f0 tx=0x7f009000edf0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 shutdown_connections 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f00840a3d20 0x7f00840a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a0112c60 0x7f00a01a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f00a01010f0 0x7f00a019f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 --2- 172.21.15.146:0/846437209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a01007d0 0x7f00a019ed20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 >> 172.21.15.146:0/846437209 conn(0x7f00a0076120 msgr2=0x7f00a0110710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 shutdown_connections 2024-04-24T08:29:02.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.435+0000 7f009f200640 1 -- 172.21.15.146:0/846437209 wait complete. 2024-04-24T08:29:02.454 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:29:02.454 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:29:02.518 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:02 smithi146 ceph-mon[27805]: pgmap v240: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:02.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.650+0000 7f59d7000640 1 -- 172.21.15.146:0/631197590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0104e10 msgr2=0x7f59d0108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.650+0000 7f59d7000640 1 --2- 172.21.15.146:0/631197590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0104e10 0x7f59d0108e50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009860 tx=0x7f59b802a340 comp rx=0 tx=0).stop 2024-04-24T08:29:02.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.650+0000 7f59d7000640 1 -- 172.21.15.146:0/631197590 shutdown_connections 2024-04-24T08:29:02.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.650+0000 7f59d7000640 1 --2- 172.21.15.146:0/631197590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0104e10 0x7f59d0108e50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.650+0000 7f59d7000640 1 --2- 172.21.15.146:0/631197590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0109530 0x7f59d0111100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.650+0000 7f59d7000640 1 --2- 172.21.15.146:0/631197590 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0104460 0x7f59d0104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.650+0000 7f59d7000640 1 -- 172.21.15.146:0/631197590 >> 172.21.15.146:0/631197590 conn(0x7f59d00ffcc0 msgr2=0x7f59d01020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.651+0000 7f59d7000640 1 -- 172.21.15.146:0/631197590 shutdown_connections 2024-04-24T08:29:02.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.651+0000 7f59d7000640 1 -- 172.21.15.146:0/631197590 wait complete. 2024-04-24T08:29:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 Processor -- start 2024-04-24T08:29:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 -- start start 2024-04-24T08:29:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0104460 0x7f59d01a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0109530 0x7f59d01a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d019a960 0x7f59d019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01145a0 con 0x7f59d0104460 2024-04-24T08:29:02.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59d0114420 con 0x7f59d0109530 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59d7000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59d0114720 con 0x7f59d019a960 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59cfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0104460 0x7f59d01a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59cfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0104460 0x7f59d01a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40314/0 (socket says 172.21.15.146:40314) 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.652+0000 7f59cfe00640 1 -- 172.21.15.146:0/3923552603 learned_addr learned my addr 172.21.15.146:0/3923552603 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59c7400640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0109530 0x7f59d01a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59d5000640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d019a960 0x7f59d019ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59cfe00640 1 -- 172.21.15.146:0/3923552603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d019a960 msgr2=0x7f59d019ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59cfe00640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d019a960 0x7f59d019ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59cfe00640 1 -- 172.21.15.146:0/3923552603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0109530 msgr2=0x7f59d01a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59cfe00640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0109530 0x7f59d01a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59cfe00640 1 -- 172.21.15.146:0/3923552603 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d019b480 con 0x7f59d0104460 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59c7400640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0109530 0x7f59d01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59cfe00640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0104460 0x7f59d01a07b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f59c000e9f0 tx=0x7f59c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.653+0000 7f59cd600640 1 -- 172.21.15.146:0/3923552603 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c000cd80 con 0x7f59d0104460 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.654+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d019b770 con 0x7f59d0104460 2024-04-24T08:29:02.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.654+0000 7f59cd600640 1 -- 172.21.15.146:0/3923552603 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c0010910 con 0x7f59d0104460 2024-04-24T08:29:02.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.654+0000 7f59cd600640 1 -- 172.21.15.146:0/3923552603 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c00188f0 con 0x7f59d0104460 2024-04-24T08:29:02.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.654+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d01a79c0 con 0x7f59d0104460 2024-04-24T08:29:02.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.655+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d0068e20 con 0x7f59d0104460 2024-04-24T08:29:02.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.657+0000 7f59cd600640 1 -- 172.21.15.146:0/3923552603 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f59c0010ac0 con 0x7f59d0104460 2024-04-24T08:29:02.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.659+0000 7f59cd600640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f59b40a3d20 0x7f59b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:02.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.659+0000 7f59cd600640 1 -- 172.21.15.146:0/3923552603 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f59c00d0c10 con 0x7f59d0104460 2024-04-24T08:29:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.659+0000 7f59c7400640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f59b40a3d20 0x7f59b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:02.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.659+0000 7f59c7400640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f59b40a3d20 0x7f59b40a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009830 tx=0x7f59b80056d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:02.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.663+0000 7f59cd600640 1 -- 172.21.15.146:0/3923552603 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59c009cbf0 con 0x7f59d0104460 2024-04-24T08:29:02.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:02 smithi153 ceph-mon[32746]: pgmap v240: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:02 smithi177 ceph-mon[33046]: pgmap v240: 97 pgs: 97 active+clean; 74 MiB data, 688 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:02.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.910+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59d0104e10 con 0x7f59d0104460 2024-04-24T08:29:02.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.913+0000 7f59cd600640 1 -- 172.21.15.146:0/3923552603 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f59c00a0ef0 con 0x7f59d0104460 2024-04-24T08:29:02.914 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:29:02.914 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:29:02.915 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:29:02.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.915+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f59b40a3d20 msgr2=0x7f59b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.915+0000 7f59d7000640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f59b40a3d20 0x7f59b40a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009830 tx=0x7f59b80056d0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.915+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0104460 msgr2=0x7f59d01a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:02.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.915+0000 7f59d7000640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0104460 0x7f59d01a07b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f59c000e9f0 tx=0x7f59c000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 shutdown_connections 2024-04-24T08:29:02.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f59b40a3d20 0x7f59b40a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d019a960 0x7f59d019ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59d0109530 0x7f59d01a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 --2- 172.21.15.146:0/3923552603 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d0104460 0x7f59d01a07b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:02.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 >> 172.21.15.146:0/3923552603 conn(0x7f59d00ffcc0 msgr2=0x7f59d0101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:02.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 shutdown_connections 2024-04-24T08:29:02.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:02.916+0000 7f59d7000640 1 -- 172.21.15.146:0/3923552603 wait complete. 2024-04-24T08:29:02.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:29:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.144+0000 7f1c1b200640 1 -- 172.21.15.146:0/2257163818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c104320 msgr2=0x7f1c1c104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.144+0000 7f1c1b200640 1 --2- 172.21.15.146:0/2257163818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c104320 0x7f1c1c104700 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009860 tx=0x7f1c0c02a340 comp rx=0 tx=0).stop 2024-04-24T08:29:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.145+0000 7f1c1b200640 1 -- 172.21.15.146:0/2257163818 shutdown_connections 2024-04-24T08:29:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.145+0000 7f1c1b200640 1 --2- 172.21.15.146:0/2257163818 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c1093f0 0x7f1c1c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.145+0000 7f1c1b200640 1 --2- 172.21.15.146:0/2257163818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c104cd0 0x7f1c1c108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.145+0000 7f1c1b200640 1 --2- 172.21.15.146:0/2257163818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c104320 0x7f1c1c104700 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.145+0000 7f1c1b200640 1 -- 172.21.15.146:0/2257163818 >> 172.21.15.146:0/2257163818 conn(0x7f1c1c0ffb40 msgr2=0x7f1c1c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.145+0000 7f1c1b200640 1 -- 172.21.15.146:0/2257163818 shutdown_connections 2024-04-24T08:29:03.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.145+0000 7f1c1b200640 1 -- 172.21.15.146:0/2257163818 wait complete. 2024-04-24T08:29:03.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.146+0000 7f1c1b200640 1 Processor -- start 2024-04-24T08:29:03.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.146+0000 7f1c1b200640 1 -- start start 2024-04-24T08:29:03.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c1b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c104320 0x7f1c1c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c1b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c1093f0 0x7f1c1c1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c1b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c19a7f0 0x7f1c1c19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c1b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1144b0 con 0x7f1c1c1093f0 2024-04-24T08:29:03.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c1b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c114330 con 0x7f1c1c104320 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c1b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c114630 con 0x7f1c1c19a7f0 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c1093f0 0x7f1c1c1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c104320 0x7f1c1c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c104320 0x7f1c1c1a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33066/0 (socket says 172.21.15.146:33066) 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c1093f0 0x7f1c1c1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40338/0 (socket says 172.21.15.146:40338) 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c19e00640 1 -- 172.21.15.146:0/935552750 learned_addr learned my addr 172.21.15.146:0/935552750 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c1a800640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c19a7f0 0x7f1c1c19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c13e00640 1 -- 172.21.15.146:0/935552750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c19a7f0 msgr2=0x7f1c1c19aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c13e00640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c19a7f0 0x7f1c1c19aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c13e00640 1 -- 172.21.15.146:0/935552750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c104320 msgr2=0x7f1c1c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c13e00640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c104320 0x7f1c1c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c13e00640 1 -- 172.21.15.146:0/935552750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c19b380 con 0x7f1c1c1093f0 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.147+0000 7f1c19e00640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c104320 0x7f1c1c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.148+0000 7f1c13e00640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c1093f0 0x7f1c1c1a0bc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1c04005fd0 tx=0x7f1c0400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.148+0000 7f1c12a00640 1 -- 172.21.15.146:0/935552750 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400ccc0 con 0x7f1c1c1093f0 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.148+0000 7f1c12a00640 1 -- 172.21.15.146:0/935552750 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c0400ee70 con 0x7f1c1c1093f0 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.148+0000 7f1c12a00640 1 -- 172.21.15.146:0/935552750 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c040187e0 con 0x7f1c1c1093f0 2024-04-24T08:29:03.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.148+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1c19b670 con 0x7f1c1c1093f0 2024-04-24T08:29:03.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.148+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1c1a7800 con 0x7f1c1c1093f0 2024-04-24T08:29:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.152+0000 7f1c12a00640 1 -- 172.21.15.146:0/935552750 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f1c04010430 con 0x7f1c1c1093f0 2024-04-24T08:29:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.152+0000 7f1c23bfe740 1 Processor -- start 2024-04-24T08:29:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.152+0000 7f1c23bfe740 1 -- start start 2024-04-24T08:29:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b2898cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289975f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28999f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557b288f21d0 con 0x557b2898d460 2024-04-24T08:29:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557b28898450 con 0x557b28997b30 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557b288ee190 con 0x557b2898cb40 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b2898cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28999f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b2898cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57628/0 (socket says 172.21.15.146:57628) 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40344/0 (socket says 172.21.15.146:40344) 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 -- 172.21.15.146:0/3908686436 learned_addr learned my addr 172.21.15.146:0/3908686436 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 -- 172.21.15.146:0/3908686436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 msgr2=0x557b28999f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28999f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 -- 172.21.15.146:0/3908686436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 msgr2=0x557b289975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.153+0000 7f1c1a800640 1 -- 172.21.15.146:0/3908686436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b28955310 con 0x557b2898cb40 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.154+0000 7f1c19e00640 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.154+0000 7f1c1a800640 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b2898cf20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009f50 tx=0x7f1c140118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3ff1c2693289c73 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.154+0000 7f1c0b400640 1 -- 172.21.15.146:0/3908686436 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14004d20 con 0x557b2898cb40 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.154+0000 7f1c0b400640 1 -- 172.21.15.146:0/3908686436 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c14004050 con 0x557b2898cb40 2024-04-24T08:29:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.154+0000 7f1c0b400640 1 -- 172.21.15.146:0/3908686436 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14005280 con 0x557b2898cb40 2024-04-24T08:29:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c12a00640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1c000a3da0 0x7f1c000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3908686436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 msgr2=0x557b2898cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b2898cf20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009f50 tx=0x7f1c140118d0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c19e00640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1c000a3da0 0x7f1c000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c12a00640 1 -- 172.21.15.146:0/935552750 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f1c040d09a0 con 0x7f1c1c1093f0 2024-04-24T08:29:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3908686436 shutdown_connections 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28999f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3908686436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b2898cf20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3908686436 >> 172.21.15.146:0/3908686436 conn(0x557b28988e60 msgr2=0x557b28989240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3908686436 shutdown_connections 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.155+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3908686436 wait complete. 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 -- start start 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c19e00640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1c000a3da0 0x7f1c000a6260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0057c0 tx=0x7f1c0c0056d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 Processor -- start 2024-04-24T08:29:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289975f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b289948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28995020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x557b2899a9c0 con 0x557b2898cb40 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x557b288ab810 con 0x557b28997b30 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557b2899ab40 con 0x557b2898d460 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c23bfe740 10 client.0 _ll_register_callbacks cb 0x557b28879140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40360/0 (socket says 172.21.15.146:40360) 2024-04-24T08:29:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.156+0000 7f1c19e00640 1 -- 172.21.15.146:0/1216163286 learned_addr learned my addr 172.21.15.146:0/1216163286 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c13e00640 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b289948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c1a800640 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28995020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c19e00640 1 -- 172.21.15.146:0/1216163286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 msgr2=0x557b289948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c19e00640 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b289948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c19e00640 1 -- 172.21.15.146:0/1216163286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 msgr2=0x557b28995020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c19e00640 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28995020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c19e00640 1 -- 172.21.15.146:0/1216163286 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b288bc3f0 con 0x557b2898cb40 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1c19e00640 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289975f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009980 tx=0x7f1c0c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622e281e7fe0e145 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.157+0000 7f1bffe00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1bffe00640 1 -- 172.21.15.146:0/1216163286 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c04e060 con 0x557b2898cb40 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1bffe00640 1 -- 172.21.15.146:0/1216163286 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f1c0c003bc0 con 0x557b2898cb40 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c14004050 con 0x557b2898cb40 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1bffe00640 1 -- 172.21.15.146:0/1216163286 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1c0c037040 con 0x557b2898cb40 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1c23bfe740 10 client.16785 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557b286043f0 con 0x557b2898cb40 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1bffe00640 1 -- 172.21.15.146:0/1216163286 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c14004050 con 0x557b2898cb40 2024-04-24T08:29:03.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1bffe00640 1 -- 172.21.15.146:0/1216163286 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1c0c0324d0 con 0x557b2898cb40 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.158+0000 7f1c23bfe740 10 client.16785 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1c23bfe740 10 client.16785 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1c23bfe740 20 client.16785 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b289a7d60 0x557b289aa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1c23bfe740 4 client.16785 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b288ffbf0 con 0x557b289a7d60 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1c13e00640 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b289a7d60 0x557b289aa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1c13e00640 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b289a7d60 0x557b289aa1b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ecbfb06a06b3019f server_cookie=eab0e6d45d484da in_seq=0 out_seq=0 2024-04-24T08:29:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.159+0000 7f1bffe00640 10 client.16785 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1bffe00640 1 -- 172.21.15.146:0/1216163286 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b288ffbf0 con 0x557b289a7d60 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1bffe00640 10 client.16785 handle_command_reply: tid=0 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 client.16785 shutdown 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 msgr2=0x557b289975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289975f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009980 tx=0x7f1c0c005f90 comp rx=0 tx=0).stop 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 shutdown_connections 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b289a7d60 0x557b289aa1b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b289948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28995020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1bffe00640 0 client.16785 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/1216163286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289975f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.160+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 >> 172.21.15.146:0/1216163286 conn(0x557b28988e60 msgr2=0x557b2898ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.161+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 shutdown_connections 2024-04-24T08:29:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.161+0000 7f1c23bfe740 1 -- 172.21.15.146:0/1216163286 wait complete. 2024-04-24T08:29:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.161+0000 7f1c23bfe740 20 client.16785 trim_cache size 0 max 16384 2024-04-24T08:29:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 Processor -- start 2024-04-24T08:29:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 -- start start 2024-04-24T08:29:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b289a3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289941c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28997120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1b2690 con 0x557b2898d460 2024-04-24T08:29:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1aff60 con 0x557b28997b30 2024-04-24T08:29:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.176+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1addd0 con 0x557b2898cb40 2024-04-24T08:29:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289941c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40376/0 (socket says 172.21.15.146:40376) 2024-04-24T08:29:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c13e00640 1 -- 172.21.15.146:0/4254864830 learned_addr learned my addr 172.21.15.146:0/4254864830 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c19e00640 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b289a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c1a800640 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28997120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c1a800640 1 -- 172.21.15.146:0/4254864830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 msgr2=0x557b289a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c1a800640 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b289a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c1a800640 1 -- 172.21.15.146:0/4254864830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 msgr2=0x557b289941c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c1a800640 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289941c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c1a800640 1 -- 172.21.15.146:0/4254864830 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b286ce7d0 con 0x557b28997b30 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c13e00640 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.177+0000 7f1c1a800640 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28997120 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c14012fd0 tx=0x7f1c14003a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f8d0c50ec13b67d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c0aa00640 1 -- 172.21.15.146:0/4254864830 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14003e20 con 0x557b28997b30 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c0aa00640 1 -- 172.21.15.146:0/4254864830 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c14005530 con 0x557b28997b30 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c0aa00640 1 -- 172.21.15.146:0/4254864830 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400fbc0 con 0x557b28997b30 2024-04-24T08:29:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c23bfe740 1 -- 172.21.15.146:0/4254864830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 msgr2=0x557b28997120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28997120 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1c14012fd0 tx=0x7f1c14003a30 comp rx=0 tx=0).stop 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c23bfe740 1 -- 172.21.15.146:0/4254864830 shutdown_connections 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28997120 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898d460 0x557b289941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/4254864830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898cb40 0x557b289a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.178+0000 7f1c23bfe740 1 -- 172.21.15.146:0/4254864830 >> 172.21.15.146:0/4254864830 conn(0x557b28988e60 msgr2=0x557b28987f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 -- 172.21.15.146:0/4254864830 shutdown_connections 2024-04-24T08:29:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 -- 172.21.15.146:0/4254864830 wait complete. 2024-04-24T08:29:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 -- start start 2024-04-24T08:29:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 Processor -- start 2024-04-24T08:29:03.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289a0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b28991c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28992270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1ae420 con 0x557b2898cb40 2024-04-24T08:29:03.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1af890 con 0x557b28997b30 2024-04-24T08:29:03.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.179+0000 7f1c23bfe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1aeaf0 con 0x557b2898d460 2024-04-24T08:29:03.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c23bfe740 10 client.0 _ll_register_callbacks cb 0x7f1c140093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:03.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c1a800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28992270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c19e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40380/0 (socket says 172.21.15.146:40380) 2024-04-24T08:29:03.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c19e00640 1 -- 172.21.15.146:0/3176617995 learned_addr learned my addr 172.21.15.146:0/3176617995 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c13e00640 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b28991c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c1a800640 1 -- 172.21.15.146:0/3176617995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 msgr2=0x557b28991c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c1a800640 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b28991c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c1a800640 1 -- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 msgr2=0x557b289a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c1a800640 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c1a800640 1 -- 172.21.15.146:0/3176617995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b28998820 con 0x557b28997b30 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c19e00640 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1c1a800640 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28992270 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1c1401b040 tx=0x7f1c14005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1502f5b11694ef5e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.180+0000 7f1bffe00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:29:03.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1bffe00640 1 -- 172.21.15.146:0/3176617995 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400fa40 con 0x557b28997b30 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1bffe00640 1 -- 172.21.15.146:0/3176617995 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f1c1400fbe0 con 0x557b28997b30 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c14005530 con 0x557b28997b30 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1bffe00640 1 -- 172.21.15.146:0/3176617995 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1c1400fe00 con 0x557b28997b30 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1c23bfe740 10 client.26942 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557b28a0e4d0 con 0x557b28997b30 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1bffe00640 1 -- 172.21.15.146:0/3176617995 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c14005530 con 0x557b28997b30 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.181+0000 7f1bffe00640 1 -- 172.21.15.146:0/3176617995 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1c1402c480 con 0x557b28997b30 2024-04-24T08:29:03.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c23bfe740 10 client.26942 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c23bfe740 10 client.26942 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c23bfe740 20 client.26942 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b28a0e940 0x557b28a10d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c23bfe740 4 client.26942 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b28988cb0 con 0x557b28a0e940 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c19e00640 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b28a0e940 0x557b28a10d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1c19e00640 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b28a0e940 0x557b28a10d30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1b771a7962fa3172 server_cookie=38687a6c44d9a606 in_seq=0 out_seq=0 2024-04-24T08:29:03.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.182+0000 7f1bffe00640 10 client.26942 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1bffe00640 1 -- 172.21.15.146:0/3176617995 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b28988cb0 con 0x557b28a0e940 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1bffe00640 10 client.26942 handle_command_reply: tid=0 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 client.26942 shutdown 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 msgr2=0x557b28992270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28992270 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1c1401b040 tx=0x7f1c14005eb0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 shutdown_connections 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x557b28a0e940 0x557b28a10d30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b2898d460 0x557b28991c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x557b28997b30 0x557b28992270 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 --2- 172.21.15.146:0/3176617995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b2898cb40 0x557b289a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 >> 172.21.15.146:0/3176617995 conn(0x557b28988e60 msgr2=0x557b289a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1bffe00640 0 client.26942 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:03.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 shutdown_connections 2024-04-24T08:29:03.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 1 -- 172.21.15.146:0/3176617995 wait complete. 2024-04-24T08:29:03.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.183+0000 7f1c23bfe740 20 client.26942 trim_cache size 0 max 16384 2024-04-24T08:29:03.193 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:29:03.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1c000a3da0 msgr2=0x7f1c000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1c000a3da0 0x7f1c000a6260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0057c0 tx=0x7f1c0c0056d0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c1093f0 msgr2=0x7f1c1c1a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c1093f0 0x7f1c1c1a0bc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1c04005fd0 tx=0x7f1c0400ec20 comp rx=0 tx=0).stop 2024-04-24T08:29:03.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 shutdown_connections 2024-04-24T08:29:03.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f1c000a3da0 0x7f1c000a6260 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c1c19a7f0 0x7f1c1c19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c1c1093f0 0x7f1c1c1a0bc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 --2- 172.21.15.146:0/935552750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c1c104320 0x7f1c1c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 >> 172.21.15.146:0/935552750 conn(0x7f1c1c0ffb40 msgr2=0x7f1c1c101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 shutdown_connections 2024-04-24T08:29:03.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.184+0000 7f1c1b200640 1 -- 172.21.15.146:0/935552750 wait complete. 2024-04-24T08:29:03.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:29:03.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.413+0000 7f207e000640 1 -- 172.21.15.146:0/2562044215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078109520 msgr2=0x7f20781110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.413+0000 7f207e000640 1 --2- 172.21.15.146:0/2562044215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078109520 0x7f20781110f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2070009be0 tx=0x7f207002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.414+0000 7f207e000640 1 -- 172.21.15.146:0/2562044215 shutdown_connections 2024-04-24T08:29:03.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.414+0000 7f207e000640 1 --2- 172.21.15.146:0/2562044215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078109520 0x7f20781110f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.414+0000 7f207e000640 1 --2- 172.21.15.146:0/2562044215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2078104e00 0x7f2078108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.414+0000 7f207e000640 1 --2- 172.21.15.146:0/2562044215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2078104450 0x7f2078104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.414+0000 7f207e000640 1 -- 172.21.15.146:0/2562044215 >> 172.21.15.146:0/2562044215 conn(0x7f20780ffc70 msgr2=0x7f2078102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.414+0000 7f207e000640 1 -- 172.21.15.146:0/2562044215 shutdown_connections 2024-04-24T08:29:03.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.414+0000 7f207e000640 1 -- 172.21.15.146:0/2562044215 wait complete. 2024-04-24T08:29:03.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 Processor -- start 2024-04-24T08:29:03.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 -- start start 2024-04-24T08:29:03.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078104450 0x7f20781a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2078109520 0x7f20781a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f207819a8a0 0x7f207819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20781145c0 con 0x7f207819a8a0 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2078114440 con 0x7f2078104450 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.415+0000 7f207e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2078114740 con 0x7f2078109520 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078104450 0x7f20781a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078104450 0x7f20781a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33112/0 (socket says 172.21.15.146:33112) 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 -- 172.21.15.146:0/518268203 learned_addr learned my addr 172.21.15.146:0/518268203 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077e00640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f207819a8a0 0x7f207819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2076a00640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2078109520 0x7f20781a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 -- 172.21.15.146:0/518268203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2078109520 msgr2=0x7f20781a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2078109520 0x7f20781a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 -- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f207819a8a0 msgr2=0x7f207819ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f207819a8a0 0x7f207819ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.416+0000 7f2077400640 1 -- 172.21.15.146:0/518268203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207819b3c0 con 0x7f2078104450 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.417+0000 7f2077400640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078104450 0x7f20781a06f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f206800e9c0 tx=0x7f206800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.417+0000 7f206fe00640 1 -- 172.21.15.146:0/518268203 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206800cde0 con 0x7f2078104450 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.417+0000 7f206fe00640 1 -- 172.21.15.146:0/518268203 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20680035f0 con 0x7f2078104450 2024-04-24T08:29:03.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.417+0000 7f206fe00640 1 -- 172.21.15.146:0/518268203 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068010690 con 0x7f2078104450 2024-04-24T08:29:03.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.417+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207819b6b0 con 0x7f2078104450 2024-04-24T08:29:03.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.417+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20781a7950 con 0x7f2078104450 2024-04-24T08:29:03.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.420+0000 7f2080e2d740 1 Processor -- start 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2080e2d740 1 -- start start 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f206fe00640 1 -- 172.21.15.146:0/518268203 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f20680037a0 con 0x7f2078104450 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ac0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311acb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311acdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2080e2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f311a261d0 con 0x55f311acb940 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2080e2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f3119cc450 con 0x55f311ac1270 2024-04-24T08:29:03.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2080e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f311a22190 con 0x55f311ac0950 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ac0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ac0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57696/0 (socket says 172.21.15.146:57696) 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2077e00640 1 -- 172.21.15.146:0/2060770453 learned_addr learned my addr 172.21.15.146:0/2060770453 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2077400640 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311acb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.421+0000 7f2076a00640 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311acdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2077e00640 1 -- 172.21.15.146:0/2060770453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 msgr2=0x55f311acb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2077e00640 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311acb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2077e00640 1 -- 172.21.15.146:0/2060770453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 msgr2=0x55f311acdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2077e00640 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311acdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2077e00640 1 -- 172.21.15.146:0/2060770453 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f311a89310 con 0x55f311ac0950 2024-04-24T08:29:03.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2077e00640 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ac0d30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2070009be0 tx=0x7f2070030a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c608ac65aec721c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2067e00640 1 -- 172.21.15.146:0/2060770453 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070041030 con 0x55f311ac0950 2024-04-24T08:29:03.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2067e00640 1 -- 172.21.15.146:0/2060770453 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2070030c60 con 0x55f311ac0950 2024-04-24T08:29:03.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f2067e00640 1 -- 172.21.15.146:0/2060770453 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070036040 con 0x55f311ac0950 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.422+0000 7f206fe00640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f205c0a3da0 0x7f205c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f206fe00640 1 -- 172.21.15.146:0/518268203 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f20680d0b70 con 0x7f2078104450 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2077400640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f205c0a3da0 0x7f205c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2080e2d740 1 -- 172.21.15.146:0/2060770453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 msgr2=0x55f311ac0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2080e2d740 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ac0d30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2070009be0 tx=0x7f2070030a30 comp rx=0 tx=0).stop 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2080e2d740 1 -- 172.21.15.146:0/2060770453 shutdown_connections 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2080e2d740 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311acdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2080e2d740 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311acb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2080e2d740 1 --2- 172.21.15.146:0/2060770453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ac0d30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.423+0000 7f2080e2d740 1 -- 172.21.15.146:0/2060770453 >> 172.21.15.146:0/2060770453 conn(0x55f311abcc70 msgr2=0x55f311abd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.424+0000 7f2080e2d740 1 -- 172.21.15.146:0/2060770453 shutdown_connections 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.424+0000 7f2077400640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f205c0a3da0 0x7f205c0a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55f311ac1f10 tx=0x7f20680d4620 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.424+0000 7f2080e2d740 1 -- 172.21.15.146:0/2060770453 wait complete. 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.424+0000 7f2080e2d740 1 -- start start 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.424+0000 7f2080e2d740 1 Processor -- start 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311acb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311ac1270 0x55f311ac8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311acb940 0x55f311ac8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2080e2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f311ace770 con 0x55f311ac1270 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2080e2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f3119df810 con 0x55f311acb940 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2080e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f311ace8f0 con 0x55f311ac0950 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2076a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311ac1270 0x55f311ac8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311acb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2080e2d740 10 client.0 _ll_register_callbacks cb 0x55f3119ad140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:03.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2076a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311ac1270 0x55f311ac8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40418/0 (socket says 172.21.15.146:40418) 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311acb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57712/0 (socket says 172.21.15.146:57712) 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2076a00640 1 -- 172.21.15.146:0/3811761457 learned_addr learned my addr 172.21.15.146:0/3811761457 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.425+0000 7f2077e00640 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311acb940 0x55f311ac8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2076a00640 1 -- 172.21.15.146:0/3811761457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 msgr2=0x55f311acb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2076a00640 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311acb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2076a00640 1 -- 172.21.15.146:0/3811761457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311acb940 msgr2=0x55f311ac8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2076a00640 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311acb940 0x55f311ac8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2076a00640 1 -- 172.21.15.146:0/3811761457 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3119f03f0 con 0x55f311ac1270 2024-04-24T08:29:03.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2077e00640 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311acb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:03.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2076a00640 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311ac1270 0x55f311ac8640 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2060012700 tx=0x7f2060012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d99c3b43d995649 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.426+0000 7f2064c00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:03.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.427+0000 7f2064c00640 1 -- 172.21.15.146:0/3811761457 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060004300 con 0x55f311ac1270 2024-04-24T08:29:03.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.427+0000 7f2064c00640 1 -- 172.21.15.146:0/3811761457 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f20600119a0 con 0x55f311ac1270 2024-04-24T08:29:03.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.427+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2070030c60 con 0x55f311ac1270 2024-04-24T08:29:03.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.427+0000 7f2064c00640 1 -- 172.21.15.146:0/3811761457 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f20600044a0 con 0x55f311ac1270 2024-04-24T08:29:03.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.427+0000 7f2080e2d740 10 client.16803 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f3117383f0 con 0x55f311ac1270 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2064c00640 1 -- 172.21.15.146:0/3811761457 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2070030c60 con 0x55f311ac1270 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2064c00640 1 -- 172.21.15.146:0/3811761457 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f206000f460 con 0x55f311ac1270 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2080e2d740 10 client.16803 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2080e2d740 10 client.16803 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2080e2d740 20 client.16803 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2080e2d740 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311adbb50 0x55f311addf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2080e2d740 4 client.16803 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:03.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.428+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f311a33bf0 con 0x55f311adbb50 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.429+0000 7f2077e00640 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311adbb50 0x55f311addf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.429+0000 7f2077e00640 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311adbb50 0x55f311addf60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f857825e9108ef8 server_cookie=c7f7b7df10b5a111 in_seq=0 out_seq=0 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.429+0000 7f2064c00640 10 client.16803 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2064c00640 1 -- 172.21.15.146:0/3811761457 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f311a33bf0 con 0x55f311adbb50 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2064c00640 10 client.16803 handle_command_reply: tid=0 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 client.16803 shutdown 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311ac1270 msgr2=0x55f311ac8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311ac1270 0x55f311ac8640 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2060012700 tx=0x7f2060012bc0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 shutdown_connections 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311adbb50 0x55f311addf60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311acb940 0x55f311ac8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311ac1270 0x55f311ac8640 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 --2- 172.21.15.146:0/3811761457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311acb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2064c00640 0 client.16803 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.430+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 >> 172.21.15.146:0/3811761457 conn(0x55f311abcc70 msgr2=0x55f311ac2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.431+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 shutdown_connections 2024-04-24T08:29:03.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.431+0000 7f2080e2d740 1 -- 172.21.15.146:0/3811761457 wait complete. 2024-04-24T08:29:03.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.431+0000 7f2080e2d740 20 client.16803 trim_cache size 0 max 16384 2024-04-24T08:29:03.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 Processor -- start 2024-04-24T08:29:03.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 -- start start 2024-04-24T08:29:03.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ad7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311ab7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311abd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20781b28c0 con 0x55f311acb940 2024-04-24T08:29:03.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f20781b0190 con 0x55f311ac1270 2024-04-24T08:29:03.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2080e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20781ae000 con 0x55f311ac0950 2024-04-24T08:29:03.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.446+0000 7f2077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ad7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2076a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311ab7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ad7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57726/0 (socket says 172.21.15.146:57726) 2024-04-24T08:29:03.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2076a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311ab7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33130/0 (socket says 172.21.15.146:33130) 2024-04-24T08:29:03.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 -- 172.21.15.146:0/673155291 learned_addr learned my addr 172.21.15.146:0/673155291 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311abd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.449 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 -- 172.21.15.146:0/673155291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 msgr2=0x55f311ab7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311ab7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 -- 172.21.15.146:0/673155291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 msgr2=0x55f311abd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311abd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 -- 172.21.15.146:0/673155291 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3118027d0 con 0x55f311ac0950 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2076a00640 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311ab7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.447+0000 7f2077e00640 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ad7660 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2070000f80 tx=0x7f207002a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=347bdb5e0d0f3c4e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2067400640 1 -- 172.21.15.146:0/673155291 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070036040 con 0x55f311ac0950 2024-04-24T08:29:03.450 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2067400640 1 -- 172.21.15.146:0/673155291 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2070003b20 con 0x55f311ac0950 2024-04-24T08:29:03.451 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2067400640 1 -- 172.21.15.146:0/673155291 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070041040 con 0x55f311ac0950 2024-04-24T08:29:03.454 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2080e2d740 1 -- 172.21.15.146:0/673155291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 msgr2=0x55f311ad7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2080e2d740 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ad7660 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2070000f80 tx=0x7f207002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2080e2d740 1 -- 172.21.15.146:0/673155291 shutdown_connections 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2080e2d740 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311abd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2080e2d740 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac1270 0x55f311ab7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.448+0000 7f2080e2d740 1 --2- 172.21.15.146:0/673155291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac0950 0x55f311ad7660 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.449+0000 7f2080e2d740 1 -- 172.21.15.146:0/673155291 >> 172.21.15.146:0/673155291 conn(0x55f311abcc70 msgr2=0x55f311ac7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.449+0000 7f2080e2d740 1 -- 172.21.15.146:0/673155291 shutdown_connections 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.449+0000 7f2080e2d740 1 -- 172.21.15.146:0/673155291 wait complete. 2024-04-24T08:29:03.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.449+0000 7f2080e2d740 1 -- start start 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.449+0000 7f2080e2d740 1 Processor -- start 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac0950 0x55f311ad3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac1270 0x55f311ac58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2080e2d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311ac5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2080e2d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f20781ae650 con 0x55f311acb940 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2080e2d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f20781afac0 con 0x55f311ac0950 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2080e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20781aed20 con 0x55f311ac1270 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2080e2d740 10 client.0 _ll_register_callbacks cb 0x7f2060009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac1270 0x55f311ac58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac1270 0x55f311ac58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57736/0 (socket says 172.21.15.146:57736) 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2077e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac0950 0x55f311ad3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 -- 172.21.15.146:0/1691660979 learned_addr learned my addr 172.21.15.146:0/1691660979 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2077e00640 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311ac5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 -- 172.21.15.146:0/1691660979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac0950 msgr2=0x55f311ad3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac0950 0x55f311ad3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 -- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 msgr2=0x55f311ac5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311ac5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2076a00640 1 -- 172.21.15.146:0/1691660979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f311ac6340 con 0x55f311ac1270 2024-04-24T08:29:03.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.450+0000 7f2077e00640 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311ac5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2076a00640 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac1270 0x55f311ac58c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2060000c00 tx=0x7f2060005900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cefd5e34fc75bf21 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2064c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2064c00640 1 -- 172.21.15.146:0/1691660979 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060029040 con 0x55f311ac1270 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2064c00640 1 -- 172.21.15.146:0/1691660979 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f206000f460 con 0x55f311ac1270 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2070003b20 con 0x55f311ac1270 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2064c00640 1 -- 172.21.15.146:0/1691660979 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f206003c050 con 0x55f311ac1270 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2080e2d740 10 client.26947 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:03.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.451+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f311acce80 con 0x55f311ac1270 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2064c00640 1 -- 172.21.15.146:0/1691660979 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2070003b20 con 0x55f311ac1270 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2064c00640 1 -- 172.21.15.146:0/1691660979 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2060005040 con 0x55f311ac1270 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2080e2d740 10 client.26947 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2080e2d740 10 client.26947 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2080e2d740 20 client.26947 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2080e2d740 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311b42520 0x55f311b44940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2080e2d740 4 client.26947 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f311a2da60 con 0x55f311b42520 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.452+0000 7f2077e00640 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311b42520 0x55f311b44940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2077e00640 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311b42520 0x55f311b44940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d4a2fb74ddd00d4 server_cookie=fb9ccc5977e8b299 in_seq=0 out_seq=0 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2064c00640 10 client.26947 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2064c00640 1 -- 172.21.15.146:0/1691660979 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f311a2da60 con 0x55f311b42520 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2064c00640 10 client.26947 handle_command_reply: tid=0 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 client.26947 shutdown 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac1270 msgr2=0x55f311ac58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac1270 0x55f311ac58c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2060000c00 tx=0x7f2060005900 comp rx=0 tx=0).stop 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 shutdown_connections 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55f311b42520 0x55f311b44940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f311ac1270 0x55f311ac58c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f311acb940 0x55f311ac5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2064c00640 0 client.26947 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 --2- 172.21.15.146:0/1691660979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f311ac0950 0x55f311ad3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 >> 172.21.15.146:0/1691660979 conn(0x55f311abcc70 msgr2=0x55f311abd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 shutdown_connections 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.453+0000 7f2080e2d740 1 -- 172.21.15.146:0/1691660979 wait complete. 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f2080e2d740 20 client.26947 trim_cache size 0 max 16384 2024-04-24T08:29:03.461 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:29:03.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f205c0a3da0 msgr2=0x7f205c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f205c0a3da0 0x7f205c0a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55f311ac1f10 tx=0x7f20680d4620 comp rx=0 tx=0).stop 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078104450 msgr2=0x7f20781a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078104450 0x7f20781a06f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f206800e9c0 tx=0x7f206800ee80 comp rx=0 tx=0).stop 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 shutdown_connections 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f205c0a3da0 0x7f205c0a6260 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2078109520 0x7f20781a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f207819a8a0 0x7f207819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 --2- 172.21.15.146:0/518268203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2078104450 0x7f20781a06f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 >> 172.21.15.146:0/518268203 conn(0x7f20780ffc70 msgr2=0x7f2078101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.454+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 shutdown_connections 2024-04-24T08:29:03.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.455+0000 7f207e000640 1 -- 172.21.15.146:0/518268203 wait complete. 2024-04-24T08:29:03.470 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:03 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3923552603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:29:03.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:29:03.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 -- 172.21.15.146:0/1951035083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108580 msgr2=0x7f357c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 --2- 172.21.15.146:0/1951035083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108580 0x7f357c108a00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3564009930 tx=0x7f356402a420 comp rx=0 tx=0).stop 2024-04-24T08:29:03.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 -- 172.21.15.146:0/1951035083 shutdown_connections 2024-04-24T08:29:03.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 --2- 172.21.15.146:0/1951035083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f357c108f40 0x7f357c1153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 --2- 172.21.15.146:0/1951035083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108580 0x7f357c108a00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 --2- 172.21.15.146:0/1951035083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c10d500 0x7f357c10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 -- 172.21.15.146:0/1951035083 >> 172.21.15.146:0/1951035083 conn(0x7f357c077e20 msgr2=0x7f357c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 -- 172.21.15.146:0/1951035083 shutdown_connections 2024-04-24T08:29:03.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.667+0000 7f3581a00640 1 -- 172.21.15.146:0/1951035083 wait complete. 2024-04-24T08:29:03.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.668+0000 7f3581a00640 1 Processor -- start 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581a00640 1 -- start start 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c108580 0x7f357c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108f40 0x7f357c19f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f357c10d500 0x7f357c1a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f357c1188d0 con 0x7f357c108f40 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f357c118750 con 0x7f357c10d500 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f357c118a50 con 0x7f357c108580 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3573400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108f40 0x7f357c19f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f357be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c108580 0x7f357c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f357be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c108580 0x7f357c19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57764/0 (socket says 172.21.15.146:57764) 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f357be00640 1 -- 172.21.15.146:0/413111117 learned_addr learned my addr 172.21.15.146:0/413111117 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.669+0000 7f3581000640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f357c10d500 0x7f357c1a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f357be00640 1 -- 172.21.15.146:0/413111117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f357c10d500 msgr2=0x7f357c1a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f357be00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f357c10d500 0x7f357c1a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f357be00640 1 -- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108f40 msgr2=0x7f357c19f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f357be00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108f40 0x7f357c19f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f357be00640 1 -- 172.21.15.146:0/413111117 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f357c1a3d50 con 0x7f357c108580 2024-04-24T08:29:03.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f357be00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c108580 0x7f357c19edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f356c00e9c0 tx=0x7f356c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f3579600640 1 -- 172.21.15.146:0/413111117 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c00cd80 con 0x7f357c108580 2024-04-24T08:29:03.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f357c1a4040 con 0x7f357c108580 2024-04-24T08:29:03.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f3579600640 1 -- 172.21.15.146:0/413111117 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356c0035f0 con 0x7f357c108580 2024-04-24T08:29:03.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f3579600640 1 -- 172.21.15.146:0/413111117 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c010690 con 0x7f357c108580 2024-04-24T08:29:03.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.670+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f357c1ab8d0 con 0x7f357c108580 2024-04-24T08:29:03.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.674+0000 7f3585f85740 1 Processor -- start 2024-04-24T08:29:03.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3585f85740 1 -- start start 2024-04-24T08:29:03.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3585f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad8b40 0x556676ad8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3585f85740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad9460 0x556676ae35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3579600640 1 -- 172.21.15.146:0/413111117 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f356c0037a0 con 0x7f357c108580 2024-04-24T08:29:03.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3585f85740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3585f85740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556676a3e1d0 con 0x556676ae3b30 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3585f85740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5566769e4450 con 0x556676ad9460 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.675+0000 7f3585f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556676a3a190 con 0x556676ad8b40 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f357be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad9460 0x556676ae35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad8b40 0x556676ad8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad8b40 0x556676ad8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57768/0 (socket says 172.21.15.146:57768) 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 -- 172.21.15.146:0/3906258003 learned_addr learned my addr 172.21.15.146:0/3906258003 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3573400640 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 -- 172.21.15.146:0/3906258003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad9460 msgr2=0x556676ae35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad9460 0x556676ae35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 -- 172.21.15.146:0/3906258003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 msgr2=0x556676ae5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 -- 172.21.15.146:0/3906258003 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556676aa1310 con 0x556676ad8b40 2024-04-24T08:29:03.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f357be00640 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad9460 0x556676ae35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:03.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.676+0000 7f3581000640 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad8b40 0x556676ad8f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3574009fc0 tx=0x7f35740118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b24c470f126bfb6a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f3571600640 1 -- 172.21.15.146:0/3906258003 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357401e070 con 0x556676ad8b40 2024-04-24T08:29:03.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f3571600640 1 -- 172.21.15.146:0/3906258003 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3574003c30 con 0x556676ad8b40 2024-04-24T08:29:03.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f3571600640 1 -- 172.21.15.146:0/3906258003 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357400fa40 con 0x556676ad8b40 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f3579600640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f35600a3d20 0x7f35600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f3579600640 1 -- 172.21.15.146:0/413111117 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f356c0d0be0 con 0x7f357c108580 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f357be00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f35600a3d20 0x7f35600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f3585f85740 1 -- 172.21.15.146:0/3906258003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad8b40 msgr2=0x556676ad8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.677+0000 7f3585f85740 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad8b40 0x556676ad8f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3574009fc0 tx=0x7f35740118d0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 -- 172.21.15.146:0/3906258003 shutdown_connections 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad9460 0x556676ae35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 --2- 172.21.15.146:0/3906258003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad8b40 0x556676ad8f20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 -- 172.21.15.146:0/3906258003 >> 172.21.15.146:0/3906258003 conn(0x556676ad4e60 msgr2=0x556676ad5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 -- 172.21.15.146:0/3906258003 shutdown_connections 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 -- 172.21.15.146:0/3906258003 wait complete. 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f357be00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f35600a3d20 0x7f35600a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x556676ada100 tx=0x7f356c0d4690 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 -- start start 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.678+0000 7f3585f85740 1 Processor -- start 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3585f85740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676ae3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3585f85740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ad9460 0x556676ae0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3585f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ae3b30 0x556676ae0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3585f85740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556676ae69c0 con 0x556676ad9460 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3585f85740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5566769f7810 con 0x556676ad8b40 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3585f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556676ae6b40 con 0x556676ae3b30 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3585f85740 10 client.0 _ll_register_callbacks cb 0x5566769c5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3573400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ad9460 0x556676ae0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3573400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ad9460 0x556676ae0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40474/0 (socket says 172.21.15.146:40474) 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3581000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676ae3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3573400640 1 -- 172.21.15.146:0/3435248511 learned_addr learned my addr 172.21.15.146:0/3435248511 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.679+0000 7f3581000640 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ae3b30 0x556676ae0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3573400640 1 -- 172.21.15.146:0/3435248511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ae3b30 msgr2=0x556676ae0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3573400640 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ae3b30 0x556676ae0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3573400640 1 -- 172.21.15.146:0/3435248511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 msgr2=0x556676ae3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3573400640 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676ae3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3573400640 1 -- 172.21.15.146:0/3435248511 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556676a083f0 con 0x556676ad9460 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3581000640 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ae3b30 0x556676ae0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:03.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3573400640 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ad9460 0x556676ae0860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3564000c00 tx=0x7f35640317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94eac99ab10afc1a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f356a000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:03.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f356a000640 1 -- 172.21.15.146:0/3435248511 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3564035040 con 0x556676ad9460 2024-04-24T08:29:03.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f356a000640 1 -- 172.21.15.146:0/3435248511 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f3564004240 con 0x556676ad9460 2024-04-24T08:29:03.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3574003c30 con 0x556676ad9460 2024-04-24T08:29:03.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f356a000640 1 -- 172.21.15.146:0/3435248511 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3564004440 con 0x556676ad9460 2024-04-24T08:29:03.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3585f85740 10 client.16821 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:03.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.680+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5566767503f0 con 0x556676ad9460 2024-04-24T08:29:03.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f356a000640 1 -- 172.21.15.146:0/3435248511 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3574003c30 con 0x556676ad9460 2024-04-24T08:29:03.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f356a000640 1 -- 172.21.15.146:0/3435248511 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3564004c20 con 0x556676ad9460 2024-04-24T08:29:03.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f3585f85740 10 client.16821 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:03.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f3585f85740 10 client.16821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:03.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f3585f85740 20 client.16821 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:03.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f3585f85740 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676af3ce0 0x556676af60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f3585f85740 4 client.16821 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:03.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.681+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556676a4bbf0 con 0x556676af3ce0 2024-04-24T08:29:03.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.682+0000 7f3581000640 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676af3ce0 0x556676af60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.682+0000 7f3581000640 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676af3ce0 0x556676af60d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6599f0b524ade3c8 server_cookie=4886df50fb445168 in_seq=0 out_seq=0 2024-04-24T08:29:03.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.682+0000 7f356a000640 10 client.16821 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:03.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f356a000640 1 -- 172.21.15.146:0/3435248511 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556676a4bbf0 con 0x556676af3ce0 2024-04-24T08:29:03.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f356a000640 10 client.16821 handle_command_reply: tid=0 2024-04-24T08:29:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 client.16821 shutdown 2024-04-24T08:29:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ad9460 msgr2=0x556676ae0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ad9460 0x556676ae0860 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3564000c00 tx=0x7f35640317e0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 shutdown_connections 2024-04-24T08:29:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676af3ce0 0x556676af60d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ae3b30 0x556676ae0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ad9460 0x556676ae0860 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 --2- 172.21.15.146:0/3435248511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676ae3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f356a000640 0 client.16821 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:03.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 >> 172.21.15.146:0/3435248511 conn(0x556676ad4e60 msgr2=0x556676adac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.683+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 shutdown_connections 2024-04-24T08:29:03.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.684+0000 7f3585f85740 1 -- 172.21.15.146:0/3435248511 wait complete. 2024-04-24T08:29:03.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.684+0000 7f3585f85740 20 client.16821 trim_cache size 0 max 16384 2024-04-24T08:29:03.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 Processor -- start 2024-04-24T08:29:03.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 -- start start 2024-04-24T08:29:03.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aef880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676ad3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f357c1b6a80 con 0x556676ae3b30 2024-04-24T08:29:03.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f357c1b4350 con 0x556676ad8b40 2024-04-24T08:29:03.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.698+0000 7f3585f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f357c1b21c0 con 0x556676ad9460 2024-04-24T08:29:03.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aef880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33202/0 (socket says 172.21.15.146:33202) 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 -- 172.21.15.146:0/3522052107 learned_addr learned my addr 172.21.15.146:0/3522052107 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3573400640 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676ad3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 -- 172.21.15.146:0/3522052107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 msgr2=0x556676ad3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676ad3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 -- 172.21.15.146:0/3522052107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 msgr2=0x556676ae30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 -- 172.21.15.146:0/3522052107 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556676ae1310 con 0x556676ad8b40 2024-04-24T08:29:03.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.699+0000 7f3581000640 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aef880 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f357400f5f0 tx=0x7f3574009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=708fbeac1820dd6c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.700+0000 7f3570c00640 1 -- 172.21.15.146:0/3522052107 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357401e040 con 0x556676ad8b40 2024-04-24T08:29:03.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.700+0000 7f3570c00640 1 -- 172.21.15.146:0/3522052107 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f357401f040 con 0x556676ad8b40 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.700+0000 7f3570c00640 1 -- 172.21.15.146:0/3522052107 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574004070 con 0x556676ad8b40 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.700+0000 7f3585f85740 1 -- 172.21.15.146:0/3522052107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 msgr2=0x556676aef880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.700+0000 7f3585f85740 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aef880 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f357400f5f0 tx=0x7f3574009500 comp rx=0 tx=0).stop 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 -- 172.21.15.146:0/3522052107 shutdown_connections 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676ad3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676ae30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 --2- 172.21.15.146:0/3522052107 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aef880 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 -- 172.21.15.146:0/3522052107 >> 172.21.15.146:0/3522052107 conn(0x556676ad4e60 msgr2=0x556676ad36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 -- 172.21.15.146:0/3522052107 shutdown_connections 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 -- 172.21.15.146:0/3522052107 wait complete. 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 -- start start 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.701+0000 7f3585f85740 1 Processor -- start 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3585f85740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aebd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3585f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676aec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3585f85740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676adcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3585f85740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f357c1b2810 con 0x556676ae3b30 2024-04-24T08:29:03.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3585f85740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f357c1b3c80 con 0x556676ad8b40 2024-04-24T08:29:03.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3585f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f357c1b2ee0 con 0x556676ad9460 2024-04-24T08:29:03.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3585f85740 10 client.0 _ll_register_callbacks cb 0x7f3564009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:03.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3581000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676adcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3573400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676aec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3581000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aebd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3581000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aebd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33218/0 (socket says 172.21.15.146:33218) 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3581000640 1 -- 172.21.15.146:0/3261933464 learned_addr learned my addr 172.21.15.146:0/3261933464 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3573400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676aec2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57798/0 (socket says 172.21.15.146:57798) 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3573400640 1 -- 172.21.15.146:0/3261933464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 msgr2=0x556676aebd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.702+0000 7f3573400640 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aebd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f3573400640 1 -- 172.21.15.146:0/3261933464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 msgr2=0x556676adcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f3573400640 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676adcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f3573400640 1 -- 172.21.15.146:0/3261933464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55667681a7d0 con 0x556676ad9460 2024-04-24T08:29:03.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f3573400640 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676aec2d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3564003760 tx=0x7f3564003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7658fd29bf476dc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f356a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f356a000640 1 -- 172.21.15.146:0/3261933464 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3564035050 con 0x556676ad9460 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f357401f040 con 0x556676ad9460 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f356a000640 1 -- 172.21.15.146:0/3261933464 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f3564003e60 con 0x556676ad9460 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f356a000640 1 -- 172.21.15.146:0/3261933464 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3564053040 con 0x556676ad9460 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f3585f85740 10 client.26965 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.703+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556676ade2f0 con 0x556676ad9460 2024-04-24T08:29:03.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f356a000640 1 -- 172.21.15.146:0/3261933464 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f357401f040 con 0x556676ad9460 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f356a000640 1 -- 172.21.15.146:0/3261933464 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3564046960 con 0x556676ad9460 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f3585f85740 10 client.26965 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f3585f85740 10 client.26965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f3585f85740 20 client.26965 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f3585f85740 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676b5b220 0x556676b5d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f3585f85740 4 client.26965 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556676a45a60 con 0x556676b5b220 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.704+0000 7f3581000640 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676b5b220 0x556676b5d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:03.713 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3581000640 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676b5b220 0x556676b5d610 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=233ab97f3be64325 server_cookie=6fa88f63081ad665 in_seq=0 out_seq=0 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f356a000640 10 client.26965 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f356a000640 1 -- 172.21.15.146:0/3261933464 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556676a45a60 con 0x556676b5b220 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f356a000640 10 client.26965 handle_command_reply: tid=0 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 client.26965 shutdown 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 msgr2=0x556676aec2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676aec2d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3564003760 tx=0x7f3564003b90 comp rx=0 tx=0).stop 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 shutdown_connections 2024-04-24T08:29:03.714 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x556676b5b220 0x556676b5d610 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556676ad9460 0x556676aec2d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556676ae3b30 0x556676adcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f356a000640 0 client.26965 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 --2- 172.21.15.146:0/3261933464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556676ad8b40 0x556676aebd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 >> 172.21.15.146:0/3261933464 conn(0x556676ad4e60 msgr2=0x556676ad3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 shutdown_connections 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 1 -- 172.21.15.146:0/3261933464 wait complete. 2024-04-24T08:29:03.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.705+0000 7f3585f85740 20 client.26965 trim_cache size 0 max 16384 2024-04-24T08:29:03.716 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f35600a3d20 msgr2=0x7f35600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f35600a3d20 0x7f35600a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x556676ada100 tx=0x7f356c0d4690 comp rx=0 tx=0).stop 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c108580 msgr2=0x7f357c19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c108580 0x7f357c19edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f356c00e9c0 tx=0x7f356c00ee80 comp rx=0 tx=0).stop 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 shutdown_connections 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f35600a3d20 0x7f35600a61e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f357c10d500 0x7f357c1a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.717 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f357c108f40 0x7f357c19f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 --2- 172.21.15.146:0/413111117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f357c108580 0x7f357c19edb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:03.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 >> 172.21.15.146:0/413111117 conn(0x7f357c077e20 msgr2=0x7f357c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:03.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 shutdown_connections 2024-04-24T08:29:03.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:03.706+0000 7f3581a00640 1 -- 172.21.15.146:0/413111117 wait complete. 2024-04-24T08:29:03.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:03 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3923552603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:29:03.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:03 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3923552603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:29:04.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:04 smithi153 ceph-mon[32746]: pgmap v241: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:29:04.726 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:29:04.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:29:04.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:29:04.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:04 smithi146 ceph-mon[27805]: pgmap v241: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:29:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:04 smithi177 ceph-mon[33046]: pgmap v241: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:29:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 -- 172.21.15.146:0/2302970185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 msgr2=0x7f9fa410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 --2- 172.21.15.146:0/2302970185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa410ea40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009930 tx=0x7f9f9802a420 comp rx=0 tx=0).stop 2024-04-24T08:29:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 -- 172.21.15.146:0/2302970185 shutdown_connections 2024-04-24T08:29:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 --2- 172.21.15.146:0/2302970185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa4108610 0x7f9fa4108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 --2- 172.21.15.146:0/2302970185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fa4108fd0 0x7f9fa4115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 --2- 172.21.15.146:0/2302970185 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa410ea40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 -- 172.21.15.146:0/2302970185 >> 172.21.15.146:0/2302970185 conn(0x7f9fa4076540 msgr2=0x7f9fa4078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 -- 172.21.15.146:0/2302970185 shutdown_connections 2024-04-24T08:29:04.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.948+0000 7f9fab400640 1 -- 172.21.15.146:0/2302970185 wait complete. 2024-04-24T08:29:04.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.949+0000 7f9fab400640 1 Processor -- start 2024-04-24T08:29:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.949+0000 7f9fab400640 1 -- start start 2024-04-24T08:29:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fab400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa4108610 0x7f9fa419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fab400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fa4108fd0 0x7f9fa419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa41a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fab400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41189e0 con 0x7f9fa410e660 2024-04-24T08:29:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fab400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4118860 con 0x7f9fa4108fd0 2024-04-24T08:29:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fab400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4118b60 con 0x7f9fa4108610 2024-04-24T08:29:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa41a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fa4108fd0 0x7f9fa419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa4108610 0x7f9fa419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa41a37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40510/0 (socket says 172.21.15.146:40510) 2024-04-24T08:29:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa9400640 1 -- 172.21.15.146:0/3102260596 learned_addr learned my addr 172.21.15.146:0/3102260596 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa3e00640 1 -- 172.21.15.146:0/3102260596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa4108610 msgr2=0x7f9fa419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa4108610 0x7f9fa419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa3e00640 1 -- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 msgr2=0x7f9fa41a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa41a37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa3e00640 1 -- 172.21.15.146:0/3102260596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa41a3ef0 con 0x7f9fa4108fd0 2024-04-24T08:29:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.950+0000 7f9fa9400640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa41a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.951+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fa4108fd0 0x7f9fa419f440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000e9f0 tx=0x7f9f9000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.951+0000 7f9fa1600640 1 -- 172.21.15.146:0/3102260596 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9000ccc0 con 0x7f9fa4108fd0 2024-04-24T08:29:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.951+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa41a41e0 con 0x7f9fa4108fd0 2024-04-24T08:29:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.951+0000 7f9fa1600640 1 -- 172.21.15.146:0/3102260596 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f90005020 con 0x7f9fa4108fd0 2024-04-24T08:29:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.951+0000 7f9fa1600640 1 -- 172.21.15.146:0/3102260596 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f900049e0 con 0x7f9fa4108fd0 2024-04-24T08:29:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.951+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa41aba70 con 0x7f9fa4108fd0 2024-04-24T08:29:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.954+0000 7f9fa1600640 1 -- 172.21.15.146:0/3102260596 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f9f90005560 con 0x7f9fa4108fd0 2024-04-24T08:29:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.954+0000 7f9fae1b1740 1 Processor -- start 2024-04-24T08:29:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fae1b1740 1 -- start start 2024-04-24T08:29:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f75470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f759b0 0x558609f7fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f82470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558609edae70 con 0x558609f75090 2024-04-24T08:29:04.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558609e81380 con 0x558609f80080 2024-04-24T08:29:04.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558609ed6e30 con 0x558609f759b0 2024-04-24T08:29:04.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fa9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f75470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fa9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f75470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40522/0 (socket says 172.21.15.146:40522) 2024-04-24T08:29:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fa9400640 1 -- 172.21.15.146:0/349534024 learned_addr learned my addr 172.21.15.146:0/349534024 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f759b0 0x558609f7fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.955+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f82470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9fa9400640 1 -- 172.21.15.146:0/349534024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f759b0 msgr2=0x558609f7fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9fa9400640 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f759b0 0x558609f7fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9fa9400640 1 -- 172.21.15.146:0/349534024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 msgr2=0x558609f82470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9fa9400640 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f82470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9fa9400640 1 -- 172.21.15.146:0/349534024 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558609cb57e0 con 0x558609f75090 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f759b0 0x558609f7fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9fa9400640 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f75470 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00e830 tx=0x7f9f9c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d671227361587c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.956+0000 7f9f95c00640 1 -- 172.21.15.146:0/349534024 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c004d20 con 0x558609f75090 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.957+0000 7f9f95c00640 1 -- 172.21.15.146:0/349534024 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c005020 con 0x558609f75090 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.957+0000 7f9f95c00640 1 -- 172.21.15.146:0/349534024 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c01a600 con 0x558609f75090 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.957+0000 7f9fa1600640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9f8c0a3d20 0x7f9f8c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.957+0000 7f9fae1b1740 1 -- 172.21.15.146:0/349534024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 msgr2=0x558609f75470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.957+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f75470 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00e830 tx=0x7f9f9c00ecf0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 -- 172.21.15.146:0/349534024 shutdown_connections 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f759b0 0x558609f7fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f82470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fa1600640 1 -- 172.21.15.146:0/3102260596 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f9f900d07a0 con 0x7f9fa4108fd0 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/349534024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f75470 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 -- 172.21.15.146:0/349534024 >> 172.21.15.146:0/349534024 conn(0x558609f713b0 msgr2=0x558609f71790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9f8c0a3d20 0x7f9f8c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 -- 172.21.15.146:0/349534024 shutdown_connections 2024-04-24T08:29:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 -- 172.21.15.146:0/349534024 wait complete. 2024-04-24T08:29:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fae1b1740 1 -- start start 2024-04-24T08:29:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.958+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9f8c0a3d20 0x7f9f8c0a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9f98004770 tx=0x7f9f980058d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 1 Processor -- start 2024-04-24T08:29:04.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f7d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f759b0 0x558609f7d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f80080 0x558609f7daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558609f82e70 con 0x558609f75090 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558609e94660 con 0x558609f759b0 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558609f82ff0 con 0x558609f80080 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fae1b1740 10 client.0 _ll_register_callbacks cb 0x558609e82aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f7d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f7d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40538/0 (socket says 172.21.15.146:40538) 2024-04-24T08:29:04.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.959+0000 7f9fa8a00640 1 -- 172.21.15.146:0/2816287019 learned_addr learned my addr 172.21.15.146:0/2816287019 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa9400640 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f80080 0x558609f7daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f759b0 0x558609f7d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa8a00640 1 -- 172.21.15.146:0/2816287019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f80080 msgr2=0x558609f7daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f80080 0x558609f7daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa8a00640 1 -- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f759b0 msgr2=0x558609f7d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f759b0 0x558609f7d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa8a00640 1 -- 172.21.15.146:0/2816287019 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558609d15f70 con 0x558609f75090 2024-04-24T08:29:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f7d070 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9f98000c00 tx=0x7f9f98009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e67224d508550d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9f8a000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.960+0000 7f9f8a000640 1 -- 172.21.15.146:0/2816287019 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9804e070 con 0x558609f75090 2024-04-24T08:29:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9f8a000640 1 -- 172.21.15.146:0/2816287019 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f9f98003ac0 con 0x558609f75090 2024-04-24T08:29:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f9c005020 con 0x558609f75090 2024-04-24T08:29:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9f8a000640 1 -- 172.21.15.146:0/2816287019 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9f9805a050 con 0x558609f75090 2024-04-24T08:29:04.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9fae1b1740 10 client.16839 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:04.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558609bad170 con 0x558609f75090 2024-04-24T08:29:04.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9f8a000640 1 -- 172.21.15.146:0/2816287019 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c005020 con 0x558609f75090 2024-04-24T08:29:04.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9f8a000640 1 -- 172.21.15.146:0/2816287019 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9f9803d450 con 0x558609f75090 2024-04-24T08:29:04.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9fae1b1740 10 client.16839 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:04.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9fae1b1740 10 client.16839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:04.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9fae1b1740 20 client.16839 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:04.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.961+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609f90400 0x558609f92890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.962+0000 7f9fae1b1740 4 client.16839 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.962+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558609ee8890 con 0x558609f90400 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.962+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609f90400 0x558609f92890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.962+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609f90400 0x558609f92890 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f817adcdd4cfcda server_cookie=4c26e4586a189fcb in_seq=0 out_seq=0 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.962+0000 7f9f8a000640 10 client.16839 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.963+0000 7f9f8a000640 1 -- 172.21.15.146:0/2816287019 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558609ee8890 con 0x558609f90400 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.963+0000 7f9f8a000640 10 client.16839 handle_command_reply: tid=0 2024-04-24T08:29:04.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.963+0000 7f9fae1b1740 1 client.16839 shutdown 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.963+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 msgr2=0x558609f7d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.963+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f7d070 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9f98000c00 tx=0x7f9f98009260 comp rx=0 tx=0).stop 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 shutdown_connections 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609f90400 0x558609f92890 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f80080 0x558609f7daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f759b0 0x558609f7d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9f8a000640 0 client.16839 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2816287019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f75090 0x558609f7d070 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 >> 172.21.15.146:0/2816287019 conn(0x558609f713b0 msgr2=0x558609f771b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 shutdown_connections 2024-04-24T08:29:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2816287019 wait complete. 2024-04-24T08:29:04.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.964+0000 7f9fae1b1740 20 client.16839 trim_cache size 0 max 16384 2024-04-24T08:29:04.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 Processor -- start 2024-04-24T08:29:04.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 -- start start 2024-04-24T08:29:04.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f777f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f7d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f98160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41b6cc0 con 0x558609f759b0 2024-04-24T08:29:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41b4590 con 0x558609f80080 2024-04-24T08:29:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41b2400 con 0x558609f75090 2024-04-24T08:29:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.983+0000 7f9fa3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f7d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f777f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f7d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40554/0 (socket says 172.21.15.146:40554) 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa3e00640 1 -- 172.21.15.146:0/2257230439 learned_addr learned my addr 172.21.15.146:0/2257230439 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa9400640 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f98160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa8a00640 1 -- 172.21.15.146:0/2257230439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 msgr2=0x558609f98160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f98160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa8a00640 1 -- 172.21.15.146:0/2257230439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 msgr2=0x558609f7d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f7d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa8a00640 1 -- 172.21.15.146:0/2257230439 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558609f82470 con 0x558609f75090 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f7d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:04.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f777f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9f980096c0 tx=0x7f9f980317f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f101d37f99c52ab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:04.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9f95200640 1 -- 172.21.15.146:0/2257230439 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9804e040 con 0x558609f75090 2024-04-24T08:29:04.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9f95200640 1 -- 172.21.15.146:0/2257230439 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f98063070 con 0x558609f75090 2024-04-24T08:29:04.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.984+0000 7f9f95200640 1 -- 172.21.15.146:0/2257230439 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f98035040 con 0x558609f75090 2024-04-24T08:29:04.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2257230439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 msgr2=0x558609f777f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:04.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f777f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9f980096c0 tx=0x7f9f980317f0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2257230439 shutdown_connections 2024-04-24T08:29:04.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f98160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f7d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/2257230439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f777f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2257230439 >> 172.21.15.146:0/2257230439 conn(0x558609f713b0 msgr2=0x558609f7d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:04.993 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2257230439 shutdown_connections 2024-04-24T08:29:04.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.985+0000 7f9fae1b1740 1 -- 172.21.15.146:0/2257230439 wait complete. 2024-04-24T08:29:04.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 -- start start 2024-04-24T08:29:04.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 Processor -- start 2024-04-24T08:29:04.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f88240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f88780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f85210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.994 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41b2a50 con 0x558609f759b0 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41b3ec0 con 0x558609f80080 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41b3120 con 0x558609f75090 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.986+0000 7f9fae1b1740 10 client.0 _ll_register_callbacks cb 0x7f9f98009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f88780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f88240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f88780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40556/0 (socket says 172.21.15.146:40556) 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 -- 172.21.15.146:0/1851593329 learned_addr learned my addr 172.21.15.146:0/1851593329 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:04.995 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f88240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:57866/0 (socket says 172.21.15.146:57866) 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa9400640 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f85210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 -- 172.21.15.146:0/1851593329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 msgr2=0x558609f88240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f88240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 -- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 msgr2=0x558609f85210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f85210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 -- 172.21.15.146:0/1851593329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558609f85a40 con 0x558609f759b0 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f88240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9fa3e00640 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f88780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9f900cdf90 tx=0x7f9f90087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1672985ca117b8ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:04.996 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.987+0000 7f9f8a000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:04.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9f8a000640 1 -- 172.21.15.146:0/1851593329 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9001c040 con 0x558609f759b0 2024-04-24T08:29:04.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9f8a000640 1 -- 172.21.15.146:0/1851593329 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f9f900d3e00 con 0x558609f759b0 2024-04-24T08:29:04.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f98063070 con 0x558609f759b0 2024-04-24T08:29:04.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9f8a000640 1 -- 172.21.15.146:0/1851593329 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9f900a0ae0 con 0x558609f759b0 2024-04-24T08:29:04.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 10 client.16845 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:04.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558609ff7560 con 0x558609f759b0 2024-04-24T08:29:04.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9f8a000640 1 -- 172.21.15.146:0/1851593329 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f98063070 con 0x558609f759b0 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9f8a000640 1 -- 172.21.15.146:0/1851593329 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9f9009c990 con 0x558609f759b0 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 10 client.16845 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 20 client.16845 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609ff7c60 0x558609ffa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 4 client.16845 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.988+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558609ee2700 con 0x558609ff7c60 2024-04-24T08:29:04.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.989+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609ff7c60 0x558609ffa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.989+0000 7f9f8a000640 10 client.16845 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.989+0000 7f9fa8a00640 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609ff7c60 0x558609ffa080 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6fd03299dc5588 server_cookie=54a05067ff7dd377 in_seq=0 out_seq=0 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9f8a000640 1 -- 172.21.15.146:0/1851593329 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558609ee2700 con 0x558609ff7c60 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9f8a000640 10 client.16845 handle_command_reply: tid=0 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 client.16845 shutdown 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 msgr2=0x558609f88780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f88780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9f900cdf90 tx=0x7f9f90087040 comp rx=0 tx=0).stop 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 shutdown_connections 2024-04-24T08:29:04.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558609ff7c60 0x558609ffa080 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558609f80080 0x558609f85210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558609f759b0 0x558609f88780 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9f8a000640 0 client.16845 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 --2- 172.21.15.146:0/1851593329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558609f75090 0x558609f88240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 >> 172.21.15.146:0/1851593329 conn(0x558609f713b0 msgr2=0x558609f7d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 shutdown_connections 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 1 -- 172.21.15.146:0/1851593329 wait complete. 2024-04-24T08:29:05.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.991+0000 7f9fae1b1740 20 client.16845 trim_cache size 0 max 16384 2024-04-24T08:29:05.001 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:29:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:29:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "2cfac76d-cf68-432b-974e-db10718adbce", 2024-04-24T08:29:05.001 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:29:05.001 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:29:05.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9f8c0a3d20 msgr2=0x7f9f8c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9f8c0a3d20 0x7f9f8c0a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9f98004770 tx=0x7f9f980058d0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fa4108fd0 msgr2=0x7f9fa419f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.002 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fa4108fd0 0x7f9fa419f440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9f9000e9f0 tx=0x7f9f9000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 shutdown_connections 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9f8c0a3d20 0x7f9f8c0a61e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fa410e660 0x7f9fa41a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9fa4108fd0 0x7f9fa419f440 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 --2- 172.21.15.146:0/3102260596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9fa4108610 0x7f9fa419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 >> 172.21.15.146:0/3102260596 conn(0x7f9fa4076540 msgr2=0x7f9fa4077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 shutdown_connections 2024-04-24T08:29:05.003 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:04.992+0000 7f9fab400640 1 -- 172.21.15.146:0/3102260596 wait complete. 2024-04-24T08:29:05.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:29:05.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:29:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 -- 172.21.15.146:0/2589236876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc100de0 msgr2=0x7f3fdc1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/2589236876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc100de0 0x7f3fdc1011c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009900 tx=0x7f3fd002a400 comp rx=0 tx=0).stop 2024-04-24T08:29:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 -- 172.21.15.146:0/2589236876 shutdown_connections 2024-04-24T08:29:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/2589236876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc101700 0x7f3fdc10e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/2589236876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fdc10e990 0x7f3fdc110d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/2589236876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc100de0 0x7f3fdc1011c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 -- 172.21.15.146:0/2589236876 >> 172.21.15.146:0/2589236876 conn(0x7f3fdc077280 msgr2=0x7f3fdc077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 -- 172.21.15.146:0/2589236876 shutdown_connections 2024-04-24T08:29:05.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.331+0000 7f3fe3a00640 1 -- 172.21.15.146:0/2589236876 wait complete. 2024-04-24T08:29:05.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 Processor -- start 2024-04-24T08:29:05.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 -- start start 2024-04-24T08:29:05.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc100de0 0x7f3fdc1a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fdc101700 0x7f3fdc1a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc10e990 0x7f3fdc19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3fdc114650 con 0x7f3fdc101700 2024-04-24T08:29:05.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3fdc1144d0 con 0x7f3fdc10e990 2024-04-24T08:29:05.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe3a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3fdc1147d0 con 0x7f3fdc100de0 2024-04-24T08:29:05.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fe1a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc10e990 0x7f3fdc19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.333+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fdc101700 0x7f3fdc1a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc10e990 0x7f3fdc19a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33280/0 (socket says 172.21.15.146:33280) 2024-04-24T08:29:05.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 -- 172.21.15.146:0/3425583163 learned_addr learned my addr 172.21.15.146:0/3425583163 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:05.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1000640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc100de0 0x7f3fdc1a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 -- 172.21.15.146:0/3425583163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc100de0 msgr2=0x7f3fdc1a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc100de0 0x7f3fdc1a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 -- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fdc101700 msgr2=0x7f3fdc1a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fdc101700 0x7f3fdc1a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 -- 172.21.15.146:0/3425583163 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fdc19b0b0 con 0x7f3fdc10e990 2024-04-24T08:29:05.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe1a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc10e990 0x7f3fdc19a930 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400cce0 tx=0x7f3fd400a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:05.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fd9600640 1 -- 172.21.15.146:0/3425583163 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400f450 con 0x7f3fdc10e990 2024-04-24T08:29:05.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fd9600640 1 -- 172.21.15.146:0/3425583163 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd400fad0 con 0x7f3fdc10e990 2024-04-24T08:29:05.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.334+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fdc19b3a0 con 0x7f3fdc10e990 2024-04-24T08:29:05.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.335+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fdc1a75f0 con 0x7f3fdc10e990 2024-04-24T08:29:05.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.335+0000 7f3fd9600640 1 -- 172.21.15.146:0/3425583163 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4018bd0 con 0x7f3fdc10e990 2024-04-24T08:29:05.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fd9600640 1 -- 172.21.15.146:0/3425583163 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3fd400fc80 con 0x7f3fdc10e990 2024-04-24T08:29:05.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 Processor -- start 2024-04-24T08:29:05.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 -- start start 2024-04-24T08:29:05.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690f4180 0x560b690fe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b69100c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560b690591f0 con 0x560b690fe850 2024-04-24T08:29:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560b68fff350 con 0x560b690f3860 2024-04-24T08:29:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.338+0000 7f3fe6766740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560b690551b0 con 0x560b690f4180 2024-04-24T08:29:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33290/0 (socket says 172.21.15.146:33290) 2024-04-24T08:29:05.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1011488479 learned_addr learned my addr 172.21.15.146:0/1011488479 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690f4180 0x560b690fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b69100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fd9600640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3fc40a3da0 0x7f3fc40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1011488479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690f4180 msgr2=0x560b690fe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690f4180 0x560b690fe310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1011488479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 msgr2=0x560b69100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b69100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1011488479 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b69053f50 con 0x560b690f3860 2024-04-24T08:29:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fd9600640 1 -- 172.21.15.146:0/3425583163 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f3fd40d0bd0 con 0x7f3fdc10e990 2024-04-24T08:29:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.339+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f3c40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8011440 tx=0x7f3fc8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ccfdfadb3b44dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.340+0000 7f3fe1a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3fc40a3da0 0x7f3fc40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.340+0000 7f3fce000640 1 -- 172.21.15.146:0/1011488479 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8004030 con 0x560b690f3860 2024-04-24T08:29:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.340+0000 7f3fce000640 1 -- 172.21.15.146:0/1011488479 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fc80041d0 con 0x560b690f3860 2024-04-24T08:29:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.340+0000 7f3fce000640 1 -- 172.21.15.146:0/1011488479 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc800e7e0 con 0x560b690f3860 2024-04-24T08:29:05.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 -- 172.21.15.146:0/1011488479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 msgr2=0x560b690f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f3c40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8011440 tx=0x7f3fc8005850 comp rx=0 tx=0).stop 2024-04-24T08:29:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe1a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3fc40a3da0 0x7f3fc40a6260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400ab20 tx=0x7f3fd40072a0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 -- 172.21.15.146:0/1011488479 shutdown_connections 2024-04-24T08:29:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690f4180 0x560b690fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b69100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1011488479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f3c40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 -- 172.21.15.146:0/1011488479 >> 172.21.15.146:0/1011488479 conn(0x560b690efb80 msgr2=0x560b690eff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 -- 172.21.15.146:0/1011488479 shutdown_connections 2024-04-24T08:29:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 -- 172.21.15.146:0/1011488479 wait complete. 2024-04-24T08:29:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.341+0000 7f3fe6766740 1 -- start start 2024-04-24T08:29:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 1 Processor -- start 2024-04-24T08:29:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690f4180 0x560b690fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fe850 0x560b690fc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560b69101640 con 0x560b690f4180 2024-04-24T08:29:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560b69012690 con 0x560b690f3860 2024-04-24T08:29:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560b691017c0 con 0x560b690fe850 2024-04-24T08:29:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690fb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:33304/0 (socket says 172.21.15.146:33304) 2024-04-24T08:29:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1154760308 learned_addr learned my addr 172.21.15.146:0/1154760308 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:05.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe6766740 10 client.0 _ll_register_callbacks cb 0x560b6904e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690f4180 0x560b690fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.342+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fe850 0x560b690fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fe1000640 1 -- 172.21.15.146:0/1154760308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fe850 msgr2=0x560b690fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fe850 0x560b690fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fe1000640 1 -- 172.21.15.146:0/1154760308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 msgr2=0x560b690fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690fb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fe1000640 1 -- 172.21.15.146:0/1154760308 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b69011920 con 0x560b690f4180 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fe850 0x560b690fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690f4180 0x560b690fbc30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3fd002a9f0 tx=0x7f3fd00043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64688a59050537f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.343+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1154760308 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0035040 con 0x560b690f4180 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1154760308 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f3fd0005450 con 0x560b690f4180 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fc80041d0 con 0x560b690f4180 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1154760308 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3fd0005650 con 0x560b690f4180 2024-04-24T08:29:05.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 10 client.16854 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560b68e357d0 con 0x560b690f4180 2024-04-24T08:29:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1154760308 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fc80041d0 con 0x560b690f4180 2024-04-24T08:29:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1154760308 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3fd0003ae0 con 0x560b690f4180 2024-04-24T08:29:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 10 client.16854 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 10 client.16854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 20 client.16854 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:05.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b6910eba0 0x560b69111000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 4 client.16854 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.344+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b69066c10 con 0x560b6910eba0 2024-04-24T08:29:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.345+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b6910eba0 0x560b69111000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.345+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b6910eba0 0x560b69111000 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e355a1b008818833 server_cookie=ada30d9ea11cca9d in_seq=0 out_seq=0 2024-04-24T08:29:05.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.345+0000 7f3fc2a00640 10 client.16854 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:05.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1154760308 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b69066c10 con 0x560b6910eba0 2024-04-24T08:29:05.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fc2a00640 10 client.16854 handle_command_reply: tid=0 2024-04-24T08:29:05.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 client.16854 shutdown 2024-04-24T08:29:05.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690f4180 msgr2=0x560b690fbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690f4180 0x560b690fbc30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3fd002a9f0 tx=0x7f3fd00043d0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 shutdown_connections 2024-04-24T08:29:05.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b6910eba0 0x560b69111000 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fe850 0x560b690fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fc2a00640 0 client.16854 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:05.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690f4180 0x560b690fbc30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1154760308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 >> 172.21.15.146:0/1154760308 conn(0x560b690efb80 msgr2=0x560b690f5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 shutdown_connections 2024-04-24T08:29:05.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 1 -- 172.21.15.146:0/1154760308 wait complete. 2024-04-24T08:29:05.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.346+0000 7f3fe6766740 20 client.16854 trim_cache size 0 max 16384 2024-04-24T08:29:05.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.365+0000 7f3fe6766740 1 Processor -- start 2024-04-24T08:29:05.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe6766740 1 -- start start 2024-04-24T08:29:05.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b6910a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690fa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe6766740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc0110a0 con 0x560b690fe850 2024-04-24T08:29:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe6766740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc00e970 con 0x560b690f3860 2024-04-24T08:29:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe6766740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc00c7e0 con 0x560b690fdf30 2024-04-24T08:29:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690fa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690fa850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40578/0 (socket says 172.21.15.146:40578) 2024-04-24T08:29:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fe1000640 1 -- 172.21.15.146:0/2208305938 learned_addr learned my addr 172.21.15.146:0/2208305938 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.366+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b6910a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fe1000640 1 -- 172.21.15.146:0/2208305938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 msgr2=0x560b690fbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fe1000640 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690fbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fe1000640 1 -- 172.21.15.146:0/2208305938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 msgr2=0x560b6910a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fe1000640 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b6910a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fe1000640 1 -- 172.21.15.146:0/2208305938 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b690f3fb0 con 0x560b690fe850 2024-04-24T08:29:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690fbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fe1000640 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690fa850 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009850 tx=0x7f3fd0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5199e61a573d6d76 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fcd600640 1 -- 172.21.15.146:0/2208305938 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0035070 con 0x560b690fe850 2024-04-24T08:29:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fcd600640 1 -- 172.21.15.146:0/2208305938 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd0041070 con 0x560b690fe850 2024-04-24T08:29:05.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.367+0000 7f3fcd600640 1 -- 172.21.15.146:0/2208305938 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0053070 con 0x560b690fe850 2024-04-24T08:29:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 -- 172.21.15.146:0/2208305938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 msgr2=0x560b690fa850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690fa850 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009850 tx=0x7f3fd0001e70 comp rx=0 tx=0).stop 2024-04-24T08:29:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 -- 172.21.15.146:0/2208305938 shutdown_connections 2024-04-24T08:29:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690fbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690fa850 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 --2- 172.21.15.146:0/2208305938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b6910a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 -- 172.21.15.146:0/2208305938 >> 172.21.15.146:0/2208305938 conn(0x560b690efb80 msgr2=0x560b690fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 -- 172.21.15.146:0/2208305938 shutdown_connections 2024-04-24T08:29:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 -- 172.21.15.146:0/2208305938 wait complete. 2024-04-24T08:29:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.368+0000 7f3fe6766740 1 -- start start 2024-04-24T08:29:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 1 Processor -- start 2024-04-24T08:29:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690f8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690f9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc00ce30 con 0x560b690fe850 2024-04-24T08:29:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc00e2a0 con 0x560b690f3860 2024-04-24T08:29:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc00d500 con 0x560b690fdf30 2024-04-24T08:29:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe6766740 10 client.0 _ll_register_callbacks cb 0x7f3fd0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690f9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fdbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690f9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40582/0 (socket says 172.21.15.146:40582) 2024-04-24T08:29:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1616079420 learned_addr learned my addr 172.21.15.146:0/1616079420 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.369+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690f8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1616079420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 msgr2=0x560b690f8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690f8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 msgr2=0x560b690f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:05.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fdbe00640 1 -- 172.21.15.146:0/1616079420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b68d6b3f0 con 0x560b690fe850 2024-04-24T08:29:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fdbe00640 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690f9010 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009230 tx=0x7f3fc8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b603714e6006b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1616079420 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8028040 con 0x560b690fe850 2024-04-24T08:29:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1616079420 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8542+0+0 (secure 0 0 0) 0x7f3fc8031620 con 0x560b690fe850 2024-04-24T08:29:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.370+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fd0041070 con 0x560b690fe850 2024-04-24T08:29:05.380 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1616079420 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3fc8031820 con 0x560b690fe850 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 10 client.16860 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560b68d2b0f0 con 0x560b690fe850 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1616079420 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd0041070 con 0x560b690fe850 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1616079420 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3fc80049f0 con 0x560b690fe850 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 10 client.16860 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 10 client.16860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 20 client.16860 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:05.381 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b69175760 0x560b69177b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 4 client.16860 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:29:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560b69060a80 con 0x560b69175760 2024-04-24T08:29:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.371+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b69175760 0x560b69177b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe1000640 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b69175760 0x560b69177b50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81ad3b12a14fdb68 server_cookie=2ebcb08f10e4e993 in_seq=0 out_seq=0 2024-04-24T08:29:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fc2a00640 10 client.16860 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:05.382 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fc2a00640 1 -- 172.21.15.146:0/1616079420 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560b69060a80 con 0x560b69175760 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fc2a00640 10 client.16860 handle_command_reply: tid=0 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 client.16860 shutdown 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 msgr2=0x560b690f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690f9010 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009230 tx=0x7f3fc8005850 comp rx=0 tx=0).stop 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 shutdown_connections 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560b69175760 0x560b69177b50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560b690fdf30 0x560b690f8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560b690fe850 0x560b690f9010 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.383 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fc2a00640 0 client.16860 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:05.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 --2- 172.21.15.146:0/1616079420 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560b690f3860 0x560b690f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 >> 172.21.15.146:0/1616079420 conn(0x560b690efb80 msgr2=0x560b690fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 shutdown_connections 2024-04-24T08:29:05.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.372+0000 7f3fe6766740 1 -- 172.21.15.146:0/1616079420 wait complete. 2024-04-24T08:29:05.384 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe6766740 20 client.16860 trim_cache size 0 max 16384 2024-04-24T08:29:05.385 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:29:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:29:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:29:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "2cfac76d-cf68-432b-974e-db10718adbce": { 2024-04-24T08:29:05.385 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:29:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "2cfac76d-cf68-432b-974e-db10718adbce", 2024-04-24T08:29:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:29:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:29:05.386 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:29:05.386 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:29:05.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3fc40a3da0 msgr2=0x7f3fc40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3fc40a3da0 0x7f3fc40a6260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400ab20 tx=0x7f3fd40072a0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc10e990 msgr2=0x7f3fdc19a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc10e990 0x7f3fdc19a930 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3fd400cce0 tx=0x7f3fd400a620 comp rx=0 tx=0).stop 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 shutdown_connections 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3fc40a3da0 0x7f3fc40a6260 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fdc10e990 0x7f3fdc19a930 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fdc101700 0x7f3fdc1a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 --2- 172.21.15.146:0/3425583163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3fdc100de0 0x7f3fdc1a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 >> 172.21.15.146:0/3425583163 conn(0x7f3fdc077280 msgr2=0x7f3fdc10fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:05.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 shutdown_connections 2024-04-24T08:29:05.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:05.373+0000 7f3fe3a00640 1 -- 172.21.15.146:0/3425583163 wait complete. 2024-04-24T08:29:05.391 INFO:tasks.cephfs.filesystem:scrub status for tag:2cfac76d-cf68-432b-974e-db10718adbce - {'path': '/', 'tag': '2cfac76d-cf68-432b-974e-db10718adbce', 'options': 'recursive,force'} 2024-04-24T08:29:05.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:05 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:29:05.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:05 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:05 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:29:05.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:05 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:29:05.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:05 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:05 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:29:05.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:05 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:05 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:29:05.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:05 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:05 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:29:05.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:05 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:29:05.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:05 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:05 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:29:05.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:05 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:05 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:29:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:05 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:05 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:29:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:05 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:29:05.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:05 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:05.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:05 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:29:05.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:05 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:06 smithi153 ceph-mon[32746]: pgmap v242: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:29:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:06 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:06.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:06 smithi146 ceph-mon[27805]: pgmap v242: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:29:06.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:06 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:06.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:06 smithi177 ceph-mon[33046]: pgmap v242: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:29:06.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:06 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:08 smithi146 ceph-mon[27805]: pgmap v243: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:29:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:08 smithi146 ceph-mon[27805]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T08:29:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:08 smithi177 ceph-mon[33046]: pgmap v243: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:29:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:08 smithi177 ceph-mon[33046]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T08:29:08.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:08 smithi153 ceph-mon[32746]: pgmap v243: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:29:08.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:08 smithi153 ceph-mon[32746]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T08:29:10.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:10 smithi146 ceph-mon[27805]: pgmap v245: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:29:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:10 smithi177 ceph-mon[33046]: pgmap v245: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:29:10.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:10 smithi153 ceph-mon[32746]: pgmap v245: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:29:12.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:12 smithi146 ceph-mon[27805]: pgmap v246: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:29:12.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:12 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:29:12.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:12 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:29:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:12 smithi177 ceph-mon[33046]: pgmap v246: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:29:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:12 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:29:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:12 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:29:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:12 smithi153 ceph-mon[32746]: pgmap v246: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:29:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:12 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:29:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:12 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:29:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:14 smithi177 ceph-mon[33046]: pgmap v247: 97 pgs: 97 active+clean; 74 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:14 smithi153 ceph-mon[32746]: pgmap v247: 97 pgs: 97 active+clean; 74 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:14.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:14 smithi146 ceph-mon[27805]: pgmap v247: 97 pgs: 97 active+clean; 74 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:16 smithi153 ceph-mon[32746]: pgmap v248: 97 pgs: 97 active+clean; 74 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:16 smithi153 ceph-mon[32746]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T08:29:17.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:16 smithi146 ceph-mon[27805]: pgmap v248: 97 pgs: 97 active+clean; 74 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:17.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:16 smithi146 ceph-mon[27805]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T08:29:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:16 smithi177 ceph-mon[33046]: pgmap v248: 97 pgs: 97 active+clean; 74 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:16 smithi177 ceph-mon[33046]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T08:29:19.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:18 smithi153 ceph-mon[32746]: pgmap v250: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2024-04-24T08:29:19.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:18 smithi146 ceph-mon[27805]: pgmap v250: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2024-04-24T08:29:19.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:18 smithi177 ceph-mon[33046]: pgmap v250: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2024-04-24T08:29:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:20 smithi153 ceph-mon[32746]: pgmap v251: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:21.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:20 smithi146 ceph-mon[27805]: pgmap v251: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:20 smithi177 ceph-mon[33046]: pgmap v251: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:23.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:22 smithi153 ceph-mon[32746]: pgmap v252: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:23.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:22 smithi146 ceph-mon[27805]: pgmap v252: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:23.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:22 smithi177 ceph-mon[33046]: pgmap v252: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:29:25.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:24 smithi153 ceph-mon[32746]: pgmap v253: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:29:25.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:24 smithi146 ceph-mon[27805]: pgmap v253: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:29:25.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:24 smithi177 ceph-mon[33046]: pgmap v253: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:29:27.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:26 smithi153 ceph-mon[32746]: pgmap v254: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:29:27.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:26 smithi146 ceph-mon[27805]: pgmap v254: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:29:27.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:26 smithi177 ceph-mon[33046]: pgmap v254: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:29:29.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:28 smithi153 ceph-mon[32746]: pgmap v255: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:29:29.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:28 smithi146 ceph-mon[27805]: pgmap v255: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:29:29.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:28 smithi177 ceph-mon[33046]: pgmap v255: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:29:31.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:30 smithi153 ceph-mon[32746]: pgmap v256: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:31.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:30 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:29:31.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:30 smithi146 ceph-mon[27805]: pgmap v256: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:31.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:30 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:29:31.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:30 smithi177 ceph-mon[33046]: pgmap v256: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:31.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:30 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:29:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:32 smithi153 ceph-mon[32746]: pgmap v257: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:32 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:29:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:32 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:29:33.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:32 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:29:33.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:32 smithi146 ceph-mon[27805]: pgmap v257: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:33.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:32 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:29:33.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:32 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:29:33.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:32 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:29:33.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:32 smithi177 ceph-mon[33046]: pgmap v257: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:32 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:29:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:32 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:29:33.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:32 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:29:35.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:34 smithi153 ceph-mon[32746]: pgmap v258: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:35.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:34 smithi146 ceph-mon[27805]: pgmap v258: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:35.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:34 smithi177 ceph-mon[33046]: pgmap v258: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:35.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:29:35.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:29:35.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.603+0000 7f93a0a00640 1 -- 172.21.15.146:0/260261714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1011a0 msgr2=0x7f939c110f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.603+0000 7f93a0a00640 1 --2- 172.21.15.146:0/260261714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1011a0 0x7f939c110f90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9394009ca0 tx=0x7f939402a8b0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.603+0000 7f93a0a00640 1 -- 172.21.15.146:0/260261714 shutdown_connections 2024-04-24T08:29:35.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.603+0000 7f93a0a00640 1 --2- 172.21.15.146:0/260261714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f939c1007e0 0x7f939c100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.603+0000 7f93a0a00640 1 --2- 172.21.15.146:0/260261714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1011a0 0x7f939c110f90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.603+0000 7f93a0a00640 1 --2- 172.21.15.146:0/260261714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c102d30 0x7f939c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.603+0000 7f93a0a00640 1 -- 172.21.15.146:0/260261714 >> 172.21.15.146:0/260261714 conn(0x7f939c076090 msgr2=0x7f939c0764a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:35.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.604+0000 7f93a0a00640 1 -- 172.21.15.146:0/260261714 shutdown_connections 2024-04-24T08:29:35.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.604+0000 7f93a0a00640 1 -- 172.21.15.146:0/260261714 wait complete. 2024-04-24T08:29:35.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.604+0000 7f93a0a00640 1 Processor -- start 2024-04-24T08:29:35.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f93a0a00640 1 -- start start 2024-04-24T08:29:35.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f93a0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1007e0 0x7f939c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f93a0a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f939c1011a0 0x7f939c1a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f93a0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c102d30 0x7f939c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f93a0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f939c114520 con 0x7f939c1007e0 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f93a0a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1143a0 con 0x7f939c102d30 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f93a0a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1146a0 con 0x7f939c1011a0 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f939a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1007e0 0x7f939c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f9393e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f939c1011a0 0x7f939c1a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f939a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1007e0 0x7f939c1a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40060/0 (socket says 172.21.15.146:40060) 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f939a800640 1 -- 172.21.15.146:0/3186812008 learned_addr learned my addr 172.21.15.146:0/3186812008 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:35.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.605+0000 7f939b200640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c102d30 0x7f939c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f939a800640 1 -- 172.21.15.146:0/3186812008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f939c1011a0 msgr2=0x7f939c1a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f939a800640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f939c1011a0 0x7f939c1a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f939a800640 1 -- 172.21.15.146:0/3186812008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c102d30 msgr2=0x7f939c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f939a800640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c102d30 0x7f939c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f939a800640 1 -- 172.21.15.146:0/3186812008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c19af70 con 0x7f939c1007e0 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f939a800640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1007e0 0x7f939c1a0630 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f938c00d950 tx=0x7f938c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f9393400640 1 -- 172.21.15.146:0/3186812008 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c017030 con 0x7f939c1007e0 2024-04-24T08:29:35.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f9393400640 1 -- 172.21.15.146:0/3186812008 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f938c004020 con 0x7f939c1007e0 2024-04-24T08:29:35.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939c19b260 con 0x7f939c1007e0 2024-04-24T08:29:35.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f9393400640 1 -- 172.21.15.146:0/3186812008 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c0047a0 con 0x7f939c1007e0 2024-04-24T08:29:35.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.606+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939c1a7480 con 0x7f939c1007e0 2024-04-24T08:29:35.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 Processor -- start 2024-04-24T08:29:35.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 -- start start 2024-04-24T08:29:35.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f9393400640 1 -- 172.21.15.146:0/3186812008 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f938c0041d0 con 0x7f939c1007e0 2024-04-24T08:29:35.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda90a30 0x564bcda90e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda9ba20 0x564bcda9de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564bcd9f61f0 con 0x564bcda91350 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564bcd99c350 con 0x564bcda90a30 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.611+0000 7f93a4219740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564bcd9f21b0 con 0x564bcda9ba20 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda90a30 0x564bcda90e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda90a30 0x564bcda90e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48088/0 (socket says 172.21.15.146:48088) 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 -- 172.21.15.146:0/367744495 learned_addr learned my addr 172.21.15.146:0/367744495 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:35.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f9393e00640 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda9ba20 0x564bcda9de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 -- 172.21.15.146:0/367744495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda9ba20 msgr2=0x564bcda9de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda9ba20 0x564bcda9de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 -- 172.21.15.146:0/367744495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 msgr2=0x564bcda9b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 -- 172.21.15.146:0/367744495 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bcd9f0f50 con 0x564bcda90a30 2024-04-24T08:29:35.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f9393e00640 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda9ba20 0x564bcda9de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:35.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939a800640 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:35.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.612+0000 7f939b200640 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda90a30 0x564bcda90e10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9394005fa0 tx=0x7f93940093c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3ee10255516a118 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.613+0000 7f938b400640 1 -- 172.21.15.146:0/367744495 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394030760 con 0x564bcda90a30 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.613+0000 7f938b400640 1 -- 172.21.15.146:0/367744495 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9394036030 con 0x564bcda90a30 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.613+0000 7f938b400640 1 -- 172.21.15.146:0/367744495 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93940044a0 con 0x564bcda90a30 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.613+0000 7f9393400640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93800a3d20 0x7f93800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.613+0000 7f939a800640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93800a3d20 0x7f93800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.613+0000 7f9393400640 1 -- 172.21.15.146:0/3186812008 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f938c0d0100 con 0x7f939c1007e0 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 -- 172.21.15.146:0/367744495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda90a30 msgr2=0x564bcda90e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda90a30 0x564bcda90e10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9394005fa0 tx=0x7f93940093c0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f939a800640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93800a3d20 0x7f93800a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f938c0cd8f0 tx=0x7f938c0d3c50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 -- 172.21.15.146:0/367744495 shutdown_connections 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda9ba20 0x564bcda9de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 --2- 172.21.15.146:0/367744495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda90a30 0x564bcda90e10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 -- 172.21.15.146:0/367744495 >> 172.21.15.146:0/367744495 conn(0x564bcda8cd50 msgr2=0x564bcda8d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 -- 172.21.15.146:0/367744495 shutdown_connections 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 -- 172.21.15.146:0/367744495 wait complete. 2024-04-24T08:29:35.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.614+0000 7f93a4219740 1 -- start start 2024-04-24T08:29:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 1 Processor -- start 2024-04-24T08:29:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcda98f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcda99440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564bcda9e7b0 con 0x564bcda90a30 2024-04-24T08:29:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564bcd9af690 con 0x564bcda9ba20 2024-04-24T08:29:35.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564bcda9e930 con 0x564bcda91350 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f93a4219740 10 client.0 _ll_register_callbacks cb 0x564bcd9eb650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f9393e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcda98f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f939b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f9393e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcda98f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40604/0 (socket says 172.21.15.146:40604) 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.615+0000 7f9393e00640 1 -- 172.21.15.146:0/1733138700 learned_addr learned my addr 172.21.15.146:0/1733138700 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f939b200640 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcda99440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f9393e00640 1 -- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 msgr2=0x564bcda99440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f9393e00640 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcda99440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f9393e00640 1 -- 172.21.15.146:0/1733138700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 msgr2=0x564bcda989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f9393e00640 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f9393e00640 1 -- 172.21.15.146:0/1733138700 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bcd9ae920 con 0x564bcda91350 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f939b200640 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcda99440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f9393e00640 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcda98f00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f938400bf30 tx=0x7f93840128e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed438996a4f18e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f937fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f937fe00640 1 -- 172.21.15.146:0/1733138700 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384004220 con 0x564bcda91350 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9394036030 con 0x564bcda91350 2024-04-24T08:29:35.624 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.616+0000 7f937fe00640 1 -- 172.21.15.146:0/1733138700 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f93840043c0 con 0x564bcda91350 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f937fe00640 1 -- 172.21.15.146:0/1733138700 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9384005020 con 0x564bcda91350 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 10 client.27013 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564bcd7d27d0 con 0x564bcda91350 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f937fe00640 1 -- 172.21.15.146:0/1733138700 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9394036030 con 0x564bcda91350 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f937fe00640 1 -- 172.21.15.146:0/1733138700 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9384010760 con 0x564bcda91350 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 10 client.27013 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 10 client.27013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:35.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 20 client.27013 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdaabe10 0x564bcdaae250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 4 client.27013 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.617+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bcda03c10 con 0x564bcdaabe10 2024-04-24T08:29:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.618+0000 7f939b200640 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdaabe10 0x564bcdaae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.618+0000 7f939b200640 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdaabe10 0x564bcdaae250 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e1eada7e7bb8b1c server_cookie=bed2ff57a1aaa85e in_seq=0 out_seq=0 2024-04-24T08:29:35.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.618+0000 7f937fe00640 10 client.27013 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:35.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.619+0000 7f937fe00640 1 -- 172.21.15.146:0/1733138700 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564bcda03c10 con 0x564bcdaabe10 2024-04-24T08:29:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.619+0000 7f937fe00640 10 client.27013 handle_command_reply: tid=0 2024-04-24T08:29:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.619+0000 7f93a4219740 1 client.27013 shutdown 2024-04-24T08:29:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.619+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 msgr2=0x564bcda98f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.619+0000 7f93a4219740 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcda98f00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f938400bf30 tx=0x7f93840128e0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 shutdown_connections 2024-04-24T08:29:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdaabe10 0x564bcdaae250 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.628 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcda98f00 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcda99440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 --2- 172.21.15.146:0/1733138700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f937fe00640 0 client.27013 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:35.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 >> 172.21.15.146:0/1733138700 conn(0x564bcda8cd50 msgr2=0x564bcda8bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:35.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 shutdown_connections 2024-04-24T08:29:35.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 1 -- 172.21.15.146:0/1733138700 wait complete. 2024-04-24T08:29:35.629 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.620+0000 7f93a4219740 20 client.27013 trim_cache size 0 max 16384 2024-04-24T08:29:35.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.638+0000 7f93a4219740 1 Processor -- start 2024-04-24T08:29:35.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f93a4219740 1 -- start start 2024-04-24T08:29:35.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f93a4219740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda87e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f93a4219740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcdaa7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f93a4219740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcdab3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f93a4219740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1b2680 con 0x564bcda90a30 2024-04-24T08:29:35.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f93a4219740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1aff50 con 0x564bcda9ba20 2024-04-24T08:29:35.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f93a4219740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1addc0 con 0x564bcda91350 2024-04-24T08:29:35.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f9393e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcdaa7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f939b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda87e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f9393e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcdaa7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40614/0 (socket says 172.21.15.146:40614) 2024-04-24T08:29:35.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.639+0000 7f9393e00640 1 -- 172.21.15.146:0/1192637413 learned_addr learned my addr 172.21.15.146:0/1192637413 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:35.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f9393e00640 1 -- 172.21.15.146:0/1192637413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 msgr2=0x564bcdab3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f9393e00640 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcdab3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f9393e00640 1 -- 172.21.15.146:0/1192637413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 msgr2=0x564bcda87e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f9393e00640 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda87e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f9393e00640 1 -- 172.21.15.146:0/1192637413 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bcda97970 con 0x564bcda91350 2024-04-24T08:29:35.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f939b200640 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda87e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:35.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f9393e00640 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcdaa7670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9384001eb0 tx=0x7f9384010e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d8dd4ad9ed4e444 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f938aa00640 1 -- 172.21.15.146:0/1192637413 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384000e60 con 0x564bcda91350 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f938aa00640 1 -- 172.21.15.146:0/1192637413 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9384010960 con 0x564bcda91350 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.640+0000 7f938aa00640 1 -- 172.21.15.146:0/1192637413 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384028070 con 0x564bcda91350 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 -- 172.21.15.146:0/1192637413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 msgr2=0x564bcdaa7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcdaa7670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9384001eb0 tx=0x7f9384010e10 comp rx=0 tx=0).stop 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 -- 172.21.15.146:0/1192637413 shutdown_connections 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda91350 0x564bcdaa7670 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcdab3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 --2- 172.21.15.146:0/1192637413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda90a30 0x564bcda87e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 -- 172.21.15.146:0/1192637413 >> 172.21.15.146:0/1192637413 conn(0x564bcda8cd50 msgr2=0x564bcda99210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 -- 172.21.15.146:0/1192637413 shutdown_connections 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 -- 172.21.15.146:0/1192637413 wait complete. 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.641+0000 7f93a4219740 1 -- start start 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 1 Processor -- start 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda90a30 0x564bcda9f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcdaa01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1ae410 con 0x564bcda91350 2024-04-24T08:29:35.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1af880 con 0x564bcda9ba20 2024-04-24T08:29:35.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f939c1aeae0 con 0x564bcda90a30 2024-04-24T08:29:35.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f93a4219740 10 client.0 _ll_register_callbacks cb 0x7f938400bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:35.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f939b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda90a30 0x564bcda9f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f939b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda90a30 0x564bcda9f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40626/0 (socket says 172.21.15.146:40626) 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f9393e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f939b200640 1 -- 172.21.15.146:0/1908783804 learned_addr learned my addr 172.21.15.146:0/1908783804 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.642+0000 7f939b200640 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcdaa01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f939b200640 1 -- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 msgr2=0x564bcdaa01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f939b200640 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcdaa01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f939b200640 1 -- 172.21.15.146:0/1908783804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 msgr2=0x564bcda9fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f939b200640 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f939b200640 1 -- 172.21.15.146:0/1908783804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bcd7083f0 con 0x564bcda90a30 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f939b200640 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda90a30 0x564bcda9f740 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9394003ee0 tx=0x7f939402a8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=553a9f9aa29f902e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f937fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f937fe00640 1 -- 172.21.15.146:0/1908783804 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93940306a0 con 0x564bcda90a30 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f937fe00640 1 -- 172.21.15.146:0/1908783804 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f9394003970 con 0x564bcda90a30 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9384010960 con 0x564bcda90a30 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f937fe00640 1 -- 172.21.15.146:0/1908783804 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9394003b70 con 0x564bcda90a30 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f93a4219740 10 client.27019 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.643+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564bcd6c80f0 con 0x564bcda90a30 2024-04-24T08:29:35.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f937fe00640 1 -- 172.21.15.146:0/1908783804 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9384010960 con 0x564bcda90a30 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f937fe00640 1 -- 172.21.15.146:0/1908783804 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9394050040 con 0x564bcda90a30 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f93a4219740 10 client.27019 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f93a4219740 10 client.27019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f93a4219740 20 client.27019 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f93a4219740 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdb1b080 0x564bcdb1d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f93a4219740 4 client.27019 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564bcd9fda80 con 0x564bcdb1b080 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.644+0000 7f9393e00640 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdb1b080 0x564bcdb1d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f9393e00640 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdb1b080 0x564bcdb1d470 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=367a5d025cf7e567 server_cookie=1f6057d95ece1335 in_seq=0 out_seq=0 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f937fe00640 10 client.27019 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f937fe00640 1 -- 172.21.15.146:0/1908783804 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564bcd9fda80 con 0x564bcdb1b080 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f937fe00640 10 client.27019 handle_command_reply: tid=0 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 client.27019 shutdown 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda90a30 msgr2=0x564bcda9f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda90a30 0x564bcda9f740 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9394003ee0 tx=0x7f939402a8b0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 shutdown_connections 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564bcdb1b080 0x564bcdb1d470 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564bcda9ba20 0x564bcdaa01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564bcda91350 0x564bcda9fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 --2- 172.21.15.146:0/1908783804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564bcda90a30 0x564bcda9f740 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f937fe00640 0 client.27019 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 >> 172.21.15.146:0/1908783804 conn(0x564bcda8cd50 msgr2=0x564bcda92180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 shutdown_connections 2024-04-24T08:29:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 1 -- 172.21.15.146:0/1908783804 wait complete. 2024-04-24T08:29:35.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.645+0000 7f93a4219740 20 client.27019 trim_cache size 0 max 16384 2024-04-24T08:29:35.657 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:29:35.657 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:29:35.657 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:29:35.657 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:29:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93800a3d20 msgr2=0x7f93800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93800a3d20 0x7f93800a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f938c0cd8f0 tx=0x7f938c0d3c50 comp rx=0 tx=0).stop 2024-04-24T08:29:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1007e0 msgr2=0x7f939c1a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1007e0 0x7f939c1a0630 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f938c00d950 tx=0x7f938c00de10 comp rx=0 tx=0).stop 2024-04-24T08:29:35.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 shutdown_connections 2024-04-24T08:29:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93800a3d20 0x7f93800a61e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f939c1011a0 0x7f939c1a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f939c102d30 0x7f939c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 --2- 172.21.15.146:0/3186812008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f939c1007e0 0x7f939c1a0630 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 >> 172.21.15.146:0/3186812008 conn(0x7f939c076090 msgr2=0x7f939c0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 shutdown_connections 2024-04-24T08:29:35.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.646+0000 7f93a0a00640 1 -- 172.21.15.146:0/3186812008 wait complete. 2024-04-24T08:29:35.665 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:29:35.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:29:35.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 -- 172.21.15.146:0/2209909867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2681084a0 msgr2=0x7fe268108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 --2- 172.21.15.146:0/2209909867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2681084a0 0x7fe268108900 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe254009900 tx=0x7fe25402a480 comp rx=0 tx=0).stop 2024-04-24T08:29:35.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 -- 172.21.15.146:0/2209909867 shutdown_connections 2024-04-24T08:29:35.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 --2- 172.21.15.146:0/2209909867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe268108e40 0x7fe2681152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 --2- 172.21.15.146:0/2209909867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe2681084a0 0x7fe268108900 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 --2- 172.21.15.146:0/2209909867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe26810d3c0 0x7fe26810d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 -- 172.21.15.146:0/2209909867 >> 172.21.15.146:0/2209909867 conn(0x7fe268077b80 msgr2=0x7fe268079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:35.867 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 -- 172.21.15.146:0/2209909867 shutdown_connections 2024-04-24T08:29:35.868 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.865+0000 7fe267200640 1 -- 172.21.15.146:0/2209909867 wait complete. 2024-04-24T08:29:35.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.866+0000 7fe267200640 1 Processor -- start 2024-04-24T08:29:35.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.866+0000 7fe267200640 1 -- start start 2024-04-24T08:29:35.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe267200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2681084a0 0x7fe2681a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe267200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe268108e40 0x7fe2681a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe267200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26810d3c0 0x7fe26819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe267200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe268118850 con 0x7fe268108e40 2024-04-24T08:29:35.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe267200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe2681186d0 con 0x7fe26810d3c0 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe267200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe2681189d0 con 0x7fe2681084a0 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe265e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2681084a0 0x7fe2681a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe265e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2681084a0 0x7fe2681a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40638/0 (socket says 172.21.15.146:40638) 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe265e00640 1 -- 172.21.15.146:0/1956453595 learned_addr learned my addr 172.21.15.146:0/1956453595 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe265400640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe268108e40 0x7fe2681a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.867+0000 7fe266800640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26810d3c0 0x7fe26819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe265e00640 1 -- 172.21.15.146:0/1956453595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26810d3c0 msgr2=0x7fe26819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe265e00640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26810d3c0 0x7fe26819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe265e00640 1 -- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe268108e40 msgr2=0x7fe2681a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe265e00640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe268108e40 0x7fe2681a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe265e00640 1 -- 172.21.15.146:0/1956453595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26819f270 con 0x7fe2681084a0 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe265e00640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2681084a0 0x7fe2681a4940 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe25800e9f0 tx=0x7fe25800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe25ea00640 1 -- 172.21.15.146:0/1956453595 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25800cd80 con 0x7fe2681084a0 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe25ea00640 1 -- 172.21.15.146:0/1956453595 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe258005500 con 0x7fe2681084a0 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26819f560 con 0x7fe2681084a0 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.868+0000 7fe25ea00640 1 -- 172.21.15.146:0/1956453595 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258010690 con 0x7fe2681084a0 2024-04-24T08:29:35.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.869+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2681ab800 con 0x7fe2681084a0 2024-04-24T08:29:35.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.870+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe268069190 con 0x7fe2681084a0 2024-04-24T08:29:35.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.872+0000 7fe25ea00640 1 -- 172.21.15.146:0/1956453595 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fe2580046c0 con 0x7fe2681084a0 2024-04-24T08:29:35.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.873+0000 7fe25ea00640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:35.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.873+0000 7fe25ea00640 1 -- 172.21.15.146:0/1956453595 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7fe2580d0bd0 con 0x7fe2681084a0 2024-04-24T08:29:35.874 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.873+0000 7fe265400640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:35.875 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.874+0000 7fe265400640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe254000c00 tx=0x7fe254005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:35.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:35.875+0000 7fe25ea00640 1 -- 172.21.15.146:0/1956453595 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe25809d200 con 0x7fe2681084a0 2024-04-24T08:29:35.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:35 smithi146 ceph-mon[27805]: pgmap v259: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:36.123 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.121+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe26819ffe0 con 0x7fe2681084a0 2024-04-24T08:29:36.124 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.123+0000 7fe25ea00640 1 -- 172.21.15.146:0/1956453595 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7fe25809cb00 con 0x7fe2681084a0 2024-04-24T08:29:36.124 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:29:36.124 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:29:36.124 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:29:36.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe24c0a3d20 msgr2=0x7fe24c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe254000c00 tx=0x7fe254005b50 comp rx=0 tx=0).stop 2024-04-24T08:29:36.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2681084a0 msgr2=0x7fe2681a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2681084a0 0x7fe2681a4940 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe25800e9f0 tx=0x7fe25800edf0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 shutdown_connections 2024-04-24T08:29:36.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe26810d3c0 0x7fe26819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe268108e40 0x7fe2681a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 --2- 172.21.15.146:0/1956453595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2681084a0 0x7fe2681a4940 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.125+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 >> 172.21.15.146:0/1956453595 conn(0x7fe268077b80 msgr2=0x7fe2680781a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.126+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 shutdown_connections 2024-04-24T08:29:36.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.126+0000 7fe267200640 1 -- 172.21.15.146:0/1956453595 wait complete. 2024-04-24T08:29:36.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:29:36.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:35 smithi153 ceph-mon[32746]: pgmap v259: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:36.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:35 smithi177 ceph-mon[33046]: pgmap v259: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:36.354 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.353+0000 7fc28fe00640 1 -- 172.21.15.146:0/2641444162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc290104320 msgr2=0x7fc290104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.353+0000 7fc28fe00640 1 --2- 172.21.15.146:0/2641444162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc290104320 0x7fc290104700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc280009900 tx=0x7fc28002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.353+0000 7fc28fe00640 1 -- 172.21.15.146:0/2641444162 shutdown_connections 2024-04-24T08:29:36.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.353+0000 7fc28fe00640 1 --2- 172.21.15.146:0/2641444162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2901093f0 0x7fc290110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.353+0000 7fc28fe00640 1 --2- 172.21.15.146:0/2641444162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc290104cd0 0x7fc290108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.353+0000 7fc28fe00640 1 --2- 172.21.15.146:0/2641444162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc290104320 0x7fc290104700 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.353+0000 7fc28fe00640 1 -- 172.21.15.146:0/2641444162 >> 172.21.15.146:0/2641444162 conn(0x7fc2900ffb40 msgr2=0x7fc290101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.354+0000 7fc28fe00640 1 -- 172.21.15.146:0/2641444162 shutdown_connections 2024-04-24T08:29:36.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.354+0000 7fc28fe00640 1 -- 172.21.15.146:0/2641444162 wait complete. 2024-04-24T08:29:36.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 Processor -- start 2024-04-24T08:29:36.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 -- start start 2024-04-24T08:29:36.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc290104320 0x7fc2901a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2901093f0 0x7fc2901a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc29019a7e0 0x7fc29019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc290114480 con 0x7fc29019a7e0 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc290114300 con 0x7fc2901093f0 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc290114600 con 0x7fc290104320 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc29019a7e0 0x7fc29019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc29019a7e0 0x7fc29019ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40126/0 (socket says 172.21.15.146:40126) 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28f400640 1 -- 172.21.15.146:0/1654934559 learned_addr learned my addr 172.21.15.146:0/1654934559 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.355+0000 7fc28ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc290104320 0x7fc2901a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc28e000640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2901093f0 0x7fc2901a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc28f400640 1 -- 172.21.15.146:0/1654934559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc290104320 msgr2=0x7fc2901a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc28f400640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc290104320 0x7fc2901a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc28f400640 1 -- 172.21.15.146:0/1654934559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2901093f0 msgr2=0x7fc2901a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc28f400640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2901093f0 0x7fc2901a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc28f400640 1 -- 172.21.15.146:0/1654934559 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29019b370 con 0x7fc29019a7e0 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc28f400640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc29019a7e0 0x7fc29019ac90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc28800eaa0 tx=0x7fc28800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc287400640 1 -- 172.21.15.146:0/1654934559 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28800cdb0 con 0x7fc29019a7e0 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.356+0000 7fc287400640 1 -- 172.21.15.146:0/1654934559 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc288005020 con 0x7fc29019a7e0 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.357+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc29019b660 con 0x7fc29019a7e0 2024-04-24T08:29:36.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.357+0000 7fc287400640 1 -- 172.21.15.146:0/1654934559 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2880116b0 con 0x7fc29019a7e0 2024-04-24T08:29:36.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.357+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2901a77f0 con 0x7fc29019a7e0 2024-04-24T08:29:36.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.360+0000 7fc2989a0740 1 Processor -- start 2024-04-24T08:29:36.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.360+0000 7fc287400640 1 -- 172.21.15.146:0/1654934559 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fc288005560 con 0x7fc29019a7e0 2024-04-24T08:29:36.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc2989a0740 1 -- start start 2024-04-24T08:29:36.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f65f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f705f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f72f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc2989a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560e93ecb1d0 con 0x560e93f65b40 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc2989a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e93e71450 con 0x560e93f70b30 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc2989a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560e93ec7190 con 0x560e93f66460 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc28e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f65f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc28e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f65f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40132/0 (socket says 172.21.15.146:40132) 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc28e000640 1 -- 172.21.15.146:0/1531443835 learned_addr learned my addr 172.21.15.146:0/1531443835 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.361+0000 7fc28ea00640 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28e000640 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f72f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28e000640 1 -- 172.21.15.146:0/1531443835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 msgr2=0x560e93f705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28e000640 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f705f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28e000640 1 -- 172.21.15.146:0/1531443835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 msgr2=0x560e93f72f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28e000640 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f72f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28e000640 1 -- 172.21.15.146:0/1531443835 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e93f2e310 con 0x560e93f65b40 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28ea00640 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc28e000640 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f65f20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0135b0 tx=0x7fc27c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117d5405b0396dae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc287400640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc2740a3d20 0x7fc2740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc27be00640 1 -- 172.21.15.146:0/1531443835 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c004050 con 0x560e93f65b40 2024-04-24T08:29:36.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc27be00640 1 -- 172.21.15.146:0/1531443835 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc27c0041f0 con 0x560e93f65b40 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc27be00640 1 -- 172.21.15.146:0/1531443835 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c00f760 con 0x560e93f65b40 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.362+0000 7fc287400640 1 -- 172.21.15.146:0/1654934559 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7fc2880d09a0 con 0x7fc29019a7e0 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc28ea00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc2740a3d20 0x7fc2740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 -- 172.21.15.146:0/1531443835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 msgr2=0x560e93f65f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f65f20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0135b0 tx=0x7fc27c005850 comp rx=0 tx=0).stop 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 -- 172.21.15.146:0/1531443835 shutdown_connections 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f72f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 --2- 172.21.15.146:0/1531443835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f65f20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 -- 172.21.15.146:0/1531443835 >> 172.21.15.146:0/1531443835 conn(0x560e93f61e60 msgr2=0x560e93f62240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 -- 172.21.15.146:0/1531443835 shutdown_connections 2024-04-24T08:29:36.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc28ea00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc2740a3d20 0x7fc2740a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc280009230 tx=0x7fc2800058d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.363+0000 7fc2989a0740 1 -- 172.21.15.146:0/1531443835 wait complete. 2024-04-24T08:29:36.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.364+0000 7fc2989a0740 1 -- start start 2024-04-24T08:29:36.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.364+0000 7fc2989a0740 1 Processor -- start 2024-04-24T08:29:36.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.364+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f65b40 0x560e93f705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.364+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.364+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f70b30 0x560e93f6dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc2989a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560e93f739c0 con 0x560e93f70b30 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc2989a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560e93e84810 con 0x560e93f65b40 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc2989a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560e93f73b40 con 0x560e93f66460 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc2989a0740 10 client.0 _ll_register_callbacks cb 0x560e93e52140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f65b40 0x560e93f705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f70b30 0x560e93f6dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f65b40 0x560e93f705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48158/0 (socket says 172.21.15.146:48158) 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 -- 172.21.15.146:0/3940933938 learned_addr learned my addr 172.21.15.146:0/3940933938 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28e000640 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 -- 172.21.15.146:0/3940933938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 msgr2=0x560e93f6d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 -- 172.21.15.146:0/3940933938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f70b30 msgr2=0x560e93f6dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f70b30 0x560e93f6dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28ea00640 1 -- 172.21.15.146:0/3940933938 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e93e953f0 con 0x560e93f65b40 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.365+0000 7fc28e000640 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc28ea00640 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f65b40 0x560e93f705a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc280009900 tx=0x7fc280009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62f2cc44d8b7c70a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc278c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc278c00640 1 -- 172.21.15.146:0/3940933938 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28004e070 con 0x560e93f65b40 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc278c00640 1 -- 172.21.15.146:0/3940933938 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7fc280003d40 con 0x560e93f65b40 2024-04-24T08:29:36.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc27c0041f0 con 0x560e93f65b40 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc278c00640 1 -- 172.21.15.146:0/3940933938 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc28005a050 con 0x560e93f65b40 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc2989a0740 10 client.27044 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560e93bdd3f0 con 0x560e93f65b40 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.366+0000 7fc278c00640 1 -- 172.21.15.146:0/3940933938 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc27c0041f0 con 0x560e93f65b40 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc278c00640 1 -- 172.21.15.146:0/3940933938 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc28003d610 con 0x560e93f65b40 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc2989a0740 10 client.27044 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc2989a0740 10 client.27044 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:29:36.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc2989a0740 20 client.27044 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93f80c90 0x560e93f83110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc2989a0740 4 client.27044 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e93ed8bf0 con 0x560e93f80c90 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.367+0000 7fc28e000640 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93f80c90 0x560e93f83110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.368+0000 7fc28e000640 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93f80c90 0x560e93f83110 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2f2c54bad22ce2be server_cookie=b7955bfd7f421947 in_seq=0 out_seq=0 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.368+0000 7fc278c00640 10 client.27044 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.368+0000 7fc278c00640 1 -- 172.21.15.146:0/3940933938 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e93ed8bf0 con 0x560e93f80c90 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.368+0000 7fc278c00640 10 client.27044 handle_command_reply: tid=0 2024-04-24T08:29:36.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 client.27044 shutdown 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f65b40 msgr2=0x560e93f705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f65b40 0x560e93f705a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc280009900 tx=0x7fc280009260 comp rx=0 tx=0).stop 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 shutdown_connections 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93f80c90 0x560e93f83110 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f70b30 0x560e93f6dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3940933938 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f65b40 0x560e93f705a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc278c00640 0 client.27044 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:36.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 >> 172.21.15.146:0/3940933938 conn(0x560e93f61e60 msgr2=0x560e93f67c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 shutdown_connections 2024-04-24T08:29:36.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 1 -- 172.21.15.146:0/3940933938 wait complete. 2024-04-24T08:29:36.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.369+0000 7fc2989a0740 20 client.27044 trim_cache size 0 max 16384 2024-04-24T08:29:36.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 Processor -- start 2024-04-24T08:29:36.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 -- start start 2024-04-24T08:29:36.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.385 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901b2600 con 0x560e93f65b40 2024-04-24T08:29:36.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901afed0 con 0x560e93f70b30 2024-04-24T08:29:36.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.384+0000 7fc2989a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901add40 con 0x560e93f66460 2024-04-24T08:29:36.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40680/0 (socket says 172.21.15.146:40680) 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 -- 172.21.15.146:0/3661492132 learned_addr learned my addr 172.21.15.146:0/3661492132 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28f400640 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28ea00640 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 -- 172.21.15.146:0/3661492132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 msgr2=0x560e93f700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 -- 172.21.15.146:0/3661492132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 msgr2=0x560e93f682a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f682a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 -- 172.21.15.146:0/3661492132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e93ca77d0 con 0x560e93f66460 2024-04-24T08:29:36.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28ea00640 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:36.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.385+0000 7fc28e000640 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc27c003dc0 tx=0x7fc27c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1868419267dbd83 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.386+0000 7fc27b400640 1 -- 172.21.15.146:0/3661492132 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c0037f0 con 0x560e93f66460 2024-04-24T08:29:36.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.386+0000 7fc27b400640 1 -- 172.21.15.146:0/3661492132 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc27c003990 con 0x560e93f66460 2024-04-24T08:29:36.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.386+0000 7fc27b400640 1 -- 172.21.15.146:0/3661492132 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c02c4a0 con 0x560e93f66460 2024-04-24T08:29:36.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 -- 172.21.15.146:0/3661492132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 msgr2=0x560e93f6d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc27c003dc0 tx=0x7fc27c005f90 comp rx=0 tx=0).stop 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 -- 172.21.15.146:0/3661492132 shutdown_connections 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f6d170 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3661492132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 -- 172.21.15.146:0/3661492132 >> 172.21.15.146:0/3661492132 conn(0x560e93f61e60 msgr2=0x560e93f60f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 -- 172.21.15.146:0/3661492132 shutdown_connections 2024-04-24T08:29:36.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 -- 172.21.15.146:0/3661492132 wait complete. 2024-04-24T08:29:36.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 -- start start 2024-04-24T08:29:36.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.387+0000 7fc2989a0740 1 Processor -- start 2024-04-24T08:29:36.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f78da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f792e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc2989a0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f73d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.394 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc2989a0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901ae390 con 0x560e93f65b40 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc2989a0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901af800 con 0x560e93f70b30 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc2989a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901aea60 con 0x560e93f66460 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc2989a0740 10 client.0 _ll_register_callbacks cb 0x7fc280009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc28ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f78da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc28f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f73d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc28f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f73d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48166/0 (socket says 172.21.15.146:48166) 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc28e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f792e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.388+0000 7fc28f400640 1 -- 172.21.15.146:0/3042192632 learned_addr learned my addr 172.21.15.146:0/3042192632 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.395 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc28f400640 1 -- 172.21.15.146:0/3042192632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 msgr2=0x560e93f792e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc28f400640 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f792e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc28f400640 1 -- 172.21.15.146:0/3042192632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 msgr2=0x560e93f78da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc28f400640 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f78da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc28f400640 1 -- 172.21.15.146:0/3042192632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e93f74310 con 0x560e93f70b30 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc28e000640 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f792e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc28f400640 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f73d50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc28809c680 tx=0x7fc2880a0db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40faa9650d24ef72 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc278c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:29:36.396 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc278c00640 1 -- 172.21.15.146:0/3042192632 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28801f040 con 0x560e93f70b30 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc278c00640 1 -- 172.21.15.146:0/3042192632 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7fc2880872b0 con 0x560e93f70b30 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc27c003990 con 0x560e93f70b30 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc278c00640 1 -- 172.21.15.146:0/3042192632 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc28809bdd0 con 0x560e93f70b30 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc2989a0740 10 client.27050 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.389+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560e93fe7b70 con 0x560e93f70b30 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc278c00640 1 -- 172.21.15.146:0/3042192632 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc27c003990 con 0x560e93f70b30 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc278c00640 1 -- 172.21.15.146:0/3042192632 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc28800f040 con 0x560e93f70b30 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc2989a0740 10 client.27050 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:36.397 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc2989a0740 10 client.27050 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:29:36.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc2989a0740 20 client.27050 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:36.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93fe82c0 0x560e93fea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc2989a0740 4 client.27050 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:29:36.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e93ed2a60 con 0x560e93fe82c0 2024-04-24T08:29:36.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.390+0000 7fc28ea00640 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93fe82c0 0x560e93fea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc28ea00640 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93fe82c0 0x560e93fea6e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4ec45ccac03fc96d server_cookie=1735f3b85150364b in_seq=0 out_seq=0 2024-04-24T08:29:36.398 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc278c00640 10 client.27050 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc278c00640 1 -- 172.21.15.146:0/3042192632 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e93ed2a60 con 0x560e93fe82c0 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc278c00640 10 client.27050 handle_command_reply: tid=0 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 client.27050 shutdown 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 msgr2=0x560e93f73d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f73d50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc28809c680 tx=0x7fc2880a0db0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 shutdown_connections 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x560e93fe82c0 0x560e93fea6e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.399 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560e93f66460 0x560e93f792e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560e93f70b30 0x560e93f73d50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 --2- 172.21.15.146:0/3042192632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e93f65b40 0x560e93f78da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc278c00640 0 client.27050 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:29:36.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 >> 172.21.15.146:0/3042192632 conn(0x560e93f61e60 msgr2=0x560e93f67290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 shutdown_connections 2024-04-24T08:29:36.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.391+0000 7fc2989a0740 1 -- 172.21.15.146:0/3042192632 wait complete. 2024-04-24T08:29:36.400 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc2989a0740 20 client.27050 trim_cache size 0 max 16384 2024-04-24T08:29:36.401 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc2740a3d20 msgr2=0x7fc2740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc2740a3d20 0x7fc2740a61e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc280009230 tx=0x7fc2800058d0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc29019a7e0 msgr2=0x7fc29019ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc29019a7e0 0x7fc29019ac90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc28800eaa0 tx=0x7fc28800ef60 comp rx=0 tx=0).stop 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 shutdown_connections 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fc2740a3d20 0x7fc2740a61e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc29019a7e0 0x7fc29019ac90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.402 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2901093f0 0x7fc2901a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 --2- 172.21.15.146:0/1654934559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc290104320 0x7fc2901a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 >> 172.21.15.146:0/1654934559 conn(0x7fc2900ffb40 msgr2=0x7fc290101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.392+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 shutdown_connections 2024-04-24T08:29:36.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.393+0000 7fc28fe00640 1 -- 172.21.15.146:0/1654934559 wait complete. 2024-04-24T08:29:36.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:29:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 -- 172.21.15.146:0/3616377372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 msgr2=0x7fba101153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 --2- 172.21.15.146:0/3616377372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 0x7fba101153b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba08009900 tx=0x7fba0802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 -- 172.21.15.146:0/3616377372 shutdown_connections 2024-04-24T08:29:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 --2- 172.21.15.146:0/3616377372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 0x7fba10108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 --2- 172.21.15.146:0/3616377372 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 0x7fba101153b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 --2- 172.21.15.146:0/3616377372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1010d500 0x7fba1010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 -- 172.21.15.146:0/3616377372 >> 172.21.15.146:0/3616377372 conn(0x7fba10077e20 msgr2=0x7fba10106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 -- 172.21.15.146:0/3616377372 shutdown_connections 2024-04-24T08:29:36.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.616+0000 7fba15000640 1 -- 172.21.15.146:0/3616377372 wait complete. 2024-04-24T08:29:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.617+0000 7fba15000640 1 Processor -- start 2024-04-24T08:29:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.617+0000 7fba15000640 1 -- start start 2024-04-24T08:29:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba15000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 0x7fba101a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba15000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 0x7fba101a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba15000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1010d500 0x7fba1019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba15000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fba10118920 con 0x7fba1010d500 2024-04-24T08:29:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba15000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba101187a0 con 0x7fba10108f40 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba15000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba10118aa0 con 0x7fba10108580 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 0x7fba101a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 0x7fba101a4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40696/0 (socket says 172.21.15.146:40696) 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1010d500 0x7fba1019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0f400640 1 -- 172.21.15.146:0/2227432295 learned_addr learned my addr 172.21.15.146:0/2227432295 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba06a00640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 0x7fba101a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0f400640 1 -- 172.21.15.146:0/2227432295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 msgr2=0x7fba101a4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0f400640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 0x7fba101a4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0f400640 1 -- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1010d500 msgr2=0x7fba1019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.618+0000 7fba0f400640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1010d500 0x7fba1019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.619+0000 7fba0f400640 1 -- 172.21.15.146:0/2227432295 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1019f280 con 0x7fba10108580 2024-04-24T08:29:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.619+0000 7fba0f400640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 0x7fba101a4900 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fba0000e9c0 tx=0x7fba0000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.619+0000 7fba0cc00640 1 -- 172.21.15.146:0/2227432295 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0000cde0 con 0x7fba10108580 2024-04-24T08:29:36.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.619+0000 7fba0cc00640 1 -- 172.21.15.146:0/2227432295 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba00010910 con 0x7fba10108580 2024-04-24T08:29:36.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.619+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1019f570 con 0x7fba10108580 2024-04-24T08:29:36.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.619+0000 7fba0cc00640 1 -- 172.21.15.146:0/2227432295 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00018a00 con 0x7fba10108580 2024-04-24T08:29:36.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.619+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba101ab7c0 con 0x7fba10108580 2024-04-24T08:29:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 Processor -- start 2024-04-24T08:29:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 -- start start 2024-04-24T08:29:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba0cc00640 1 -- 172.21.15.146:0/2227432295 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fba0001f020 con 0x7fba10108580 2024-04-24T08:29:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b63b40 0x556398b63f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.632 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b6e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b70f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556398ac91d0 con 0x556398b63b40 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556398a6f450 con 0x556398b6eb30 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba19459740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556398ac5190 con 0x556398b64460 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba0fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b63b40 0x556398b63f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba0fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b63b40 0x556398b63f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40198/0 (socket says 172.21.15.146:40198) 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba0fe00640 1 -- 172.21.15.146:0/3899923296 learned_addr learned my addr 172.21.15.146:0/3899923296 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba0f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b6e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.623+0000 7fba06a00640 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b70f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba0fe00640 1 -- 172.21.15.146:0/3899923296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 msgr2=0x556398b6e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba0fe00640 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b6e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba0fe00640 1 -- 172.21.15.146:0/3899923296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 msgr2=0x556398b70f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba0fe00640 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b70f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba0fe00640 1 -- 172.21.15.146:0/3899923296 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556398b2c310 con 0x556398b63b40 2024-04-24T08:29:36.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba06a00640 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b70f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba0fe00640 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b63b40 0x556398b63f20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fba08009900 tx=0x7fba08031490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79712f5805f64e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.624+0000 7fba0cc00640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb9f40a3da0 0x7fb9f40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba04c00640 1 -- 172.21.15.146:0/3899923296 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08040070 con 0x556398b63b40 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba0cc00640 1 -- 172.21.15.146:0/2227432295 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7fba000d09b0 con 0x7fba10108580 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba04c00640 1 -- 172.21.15.146:0/3899923296 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba08004400 con 0x556398b63b40 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba04c00640 1 -- 172.21.15.146:0/3899923296 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08035040 con 0x556398b63b40 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba0f400640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb9f40a3da0 0x7fb9f40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba19459740 1 -- 172.21.15.146:0/3899923296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b63b40 msgr2=0x556398b63f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba19459740 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b63b40 0x556398b63f20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fba08009900 tx=0x7fba08031490 comp rx=0 tx=0).stop 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba0f400640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb9f40a3da0 0x7fb9f40a6260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x556398b65100 tx=0x7fba000d3f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba19459740 1 -- 172.21.15.146:0/3899923296 shutdown_connections 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba19459740 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b6e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba19459740 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b70f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba19459740 1 --2- 172.21.15.146:0/3899923296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b63b40 0x556398b63f20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.625+0000 7fba19459740 1 -- 172.21.15.146:0/3899923296 >> 172.21.15.146:0/3899923296 conn(0x556398b5fe60 msgr2=0x556398b60240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.626+0000 7fba19459740 1 -- 172.21.15.146:0/3899923296 shutdown_connections 2024-04-24T08:29:36.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.626+0000 7fba19459740 1 -- 172.21.15.146:0/3899923296 wait complete. 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.626+0000 7fba19459740 1 -- start start 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.626+0000 7fba19459740 1 Processor -- start 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba19459740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba19459740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba19459740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b6eb30 0x556398b6c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba19459740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x556398b71960 con 0x556398b64460 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba19459740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x556398a82810 con 0x556398b63b40 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba19459740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556398b71ae0 con 0x556398b6eb30 2024-04-24T08:29:36.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba19459740 10 client.0 _ll_register_callbacks cb 0x556398a50140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40212/0 (socket says 172.21.15.146:40212) 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 -- 172.21.15.146:0/2359623452 learned_addr learned my addr 172.21.15.146:0/2359623452 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b6eb30 0x556398b6c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 -- 172.21.15.146:0/2359623452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b6eb30 msgr2=0x556398b6c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b6eb30 0x556398b6c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 -- 172.21.15.146:0/2359623452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 msgr2=0x556398b6e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba06a00640 1 -- 172.21.15.146:0/2359623452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556398a933f0 con 0x556398b64460 2024-04-24T08:29:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.627+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b6eb30 0x556398b6c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fba06a00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6b8e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8011d10 tx=0x7fb9f800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b351add6a1b76dd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fb9fd600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fb9fd600640 1 -- 172.21.15.146:0/2359623452 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8003970 con 0x556398b64460 2024-04-24T08:29:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fb9fd600640 1 -- 172.21.15.146:0/2359623452 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7fb9f8003b10 con 0x556398b64460 2024-04-24T08:29:36.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba08004400 con 0x556398b64460 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fb9fd600640 1 -- 172.21.15.146:0/2359623452 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9f8003d10 con 0x556398b64460 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fba19459740 10 client.16914 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.628+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5563987db3f0 con 0x556398b64460 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fb9fd600640 1 -- 172.21.15.146:0/2359623452 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba08004400 con 0x556398b64460 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fb9fd600640 1 -- 172.21.15.146:0/2359623452 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb9f8005020 con 0x556398b64460 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fba19459740 10 client.16914 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fba19459740 10 client.16914 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:29:36.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fba19459740 20 client.16914 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fba19459740 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398b7ee00 0x556398b81240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fba19459740 4 client.16914 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556398ad6bf0 con 0x556398b7ee00 2024-04-24T08:29:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.629+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398b7ee00 0x556398b81240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.630+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398b7ee00 0x556398b81240 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6f99c0606fe4b81d server_cookie=5740b7be56bcf8aa in_seq=0 out_seq=0 2024-04-24T08:29:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.630+0000 7fb9fd600640 10 client.16914 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:36.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.630+0000 7fb9fd600640 1 -- 172.21.15.146:0/2359623452 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556398ad6bf0 con 0x556398b7ee00 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.630+0000 7fb9fd600640 10 client.16914 handle_command_reply: tid=0 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 client.16914 shutdown 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 msgr2=0x556398b6b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6b8e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8011d10 tx=0x7fb9f800f790 comp rx=0 tx=0).stop 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 shutdown_connections 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398b7ee00 0x556398b81240 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b6eb30 0x556398b6c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6b8e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fb9fd600640 0 client.16914 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 --2- 172.21.15.146:0/2359623452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 >> 172.21.15.146:0/2359623452 conn(0x556398b5fe60 msgr2=0x556398b65b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 shutdown_connections 2024-04-24T08:29:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 1 -- 172.21.15.146:0/2359623452 wait complete. 2024-04-24T08:29:36.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.631+0000 7fba19459740 20 client.16914 trim_cache size 0 max 16384 2024-04-24T08:29:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 Processor -- start 2024-04-24T08:29:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 -- start start 2024-04-24T08:29:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b63b40 0x556398b662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b86b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b69b0 con 0x556398b64460 2024-04-24T08:29:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b4280 con 0x556398b6eb30 2024-04-24T08:29:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.646+0000 7fba19459740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b20f0 con 0x556398b63b40 2024-04-24T08:29:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40224/0 (socket says 172.21.15.146:40224) 2024-04-24T08:29:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 -- 172.21.15.146:0/2568832564 learned_addr learned my addr 172.21.15.146:0/2568832564 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b63b40 0x556398b662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 -- 172.21.15.146:0/2568832564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b63b40 msgr2=0x556398b662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b63b40 0x556398b662a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b86b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 -- 172.21.15.146:0/2568832564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 msgr2=0x556398b86b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b86b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 -- 172.21.15.146:0/2568832564 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563988a57d0 con 0x556398b64460 2024-04-24T08:29:36.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba0fe00640 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b86b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:29:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.647+0000 7fba06a00640 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6e120 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8005630 tx=0x7fb9f8005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9621ac1220cb58be server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.648+0000 7fb9ffe00640 1 -- 172.21.15.146:0/2568832564 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80050c0 con 0x556398b64460 2024-04-24T08:29:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.648+0000 7fb9ffe00640 1 -- 172.21.15.146:0/2568832564 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f8003970 con 0x556398b64460 2024-04-24T08:29:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.648+0000 7fb9ffe00640 1 -- 172.21.15.146:0/2568832564 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8022b60 con 0x556398b64460 2024-04-24T08:29:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 -- 172.21.15.146:0/2568832564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 msgr2=0x556398b6e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6e120 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8005630 tx=0x7fb9f8005d30 comp rx=0 tx=0).stop 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 -- 172.21.15.146:0/2568832564 shutdown_connections 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b6eb30 0x556398b86b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b64460 0x556398b6e120 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 --2- 172.21.15.146:0/2568832564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b63b40 0x556398b662a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 -- 172.21.15.146:0/2568832564 >> 172.21.15.146:0/2568832564 conn(0x556398b5fe60 msgr2=0x556398b5e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 -- 172.21.15.146:0/2568832564 shutdown_connections 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 -- 172.21.15.146:0/2568832564 wait complete. 2024-04-24T08:29:36.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 -- start start 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.649+0000 7fba19459740 1 Processor -- start 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba19459740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba19459740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba19459740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b6eb30 0x556398b752f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba19459740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b2740 con 0x556398b6eb30 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba19459740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b3bb0 con 0x556398b63b40 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba19459740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b2e10 con 0x556398b64460 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba19459740 10 client.0 _ll_register_callbacks cb 0x7fb9f8009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:36.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba0fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba06a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba0fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48204/0 (socket says 172.21.15.146:48204) 2024-04-24T08:29:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba0fe00640 1 -- 172.21.15.146:0/3549106241 learned_addr learned my addr 172.21.15.146:0/3549106241 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.650+0000 7fba0fe00640 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b6eb30 0x556398b752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fba06a00640 1 -- 172.21.15.146:0/3549106241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 msgr2=0x556398b6fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fba06a00640 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fba06a00640 1 -- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b6eb30 msgr2=0x556398b752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fba06a00640 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b6eb30 0x556398b752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fba06a00640 1 -- 172.21.15.146:0/3549106241 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556398b75830 con 0x556398b64460 2024-04-24T08:29:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fba06a00640 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b701e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x556398a501c0 tx=0x7fb9f8005ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f76ac69ca883904f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fb9fd600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fb9fd600640 1 -- 172.21.15.146:0/3549106241 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8003b20 con 0x556398b64460 2024-04-24T08:29:36.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9f8003970 con 0x556398b64460 2024-04-24T08:29:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.651+0000 7fb9fd600640 1 -- 172.21.15.146:0/3549106241 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7fb9f8017040 con 0x556398b64460 2024-04-24T08:29:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fb9fd600640 1 -- 172.21.15.146:0/3549106241 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb9f802b440 con 0x556398b64460 2024-04-24T08:29:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 10 client.27061 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556398be5a30 con 0x556398b64460 2024-04-24T08:29:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fb9fd600640 1 -- 172.21.15.146:0/3549106241 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f8003970 con 0x556398b64460 2024-04-24T08:29:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fb9fd600640 1 -- 172.21.15.146:0/3549106241 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb9f803b5e0 con 0x556398b64460 2024-04-24T08:29:36.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 10 client.27061 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 10 client.27061 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 20 client.27061 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398be60b0 0x556398be84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 4 client.27061 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.652+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556398ad0a60 con 0x556398be60b0 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.653+0000 7fba0fe00640 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398be60b0 0x556398be84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.653+0000 7fb9fd600640 10 client.27061 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:36.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.653+0000 7fba0fe00640 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398be60b0 0x556398be84d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3d391e0bed113b0 server_cookie=e8cb0bcb31bdc90 in_seq=0 out_seq=0 2024-04-24T08:29:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.653+0000 7fb9fd600640 1 -- 172.21.15.146:0/3549106241 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556398ad0a60 con 0x556398be60b0 2024-04-24T08:29:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.653+0000 7fb9fd600640 10 client.27061 handle_command_reply: tid=0 2024-04-24T08:29:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.653+0000 7fba19459740 1 client.27061 shutdown 2024-04-24T08:29:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 msgr2=0x556398b701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b701e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x556398a501c0 tx=0x7fb9f8005ba0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 shutdown_connections 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x556398be60b0 0x556398be84d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556398b64460 0x556398b701e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fb9fd600640 0 client.27061 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556398b6eb30 0x556398b752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 --2- 172.21.15.146:0/3549106241 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x556398b63b40 0x556398b6fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 >> 172.21.15.146:0/3549106241 conn(0x556398b5fe60 msgr2=0x556398b5ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 shutdown_connections 2024-04-24T08:29:36.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 1 -- 172.21.15.146:0/3549106241 wait complete. 2024-04-24T08:29:36.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.654+0000 7fba19459740 20 client.27061 trim_cache size 0 max 16384 2024-04-24T08:29:36.664 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:29:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb9f40a3da0 msgr2=0x7fb9f40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb9f40a3da0 0x7fb9f40a6260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x556398b65100 tx=0x7fba000d3f90 comp rx=0 tx=0).stop 2024-04-24T08:29:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 msgr2=0x7fba101a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 0x7fba101a4900 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fba0000e9c0 tx=0x7fba0000ee80 comp rx=0 tx=0).stop 2024-04-24T08:29:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 shutdown_connections 2024-04-24T08:29:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb9f40a3da0 0x7fb9f40a6260 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.665 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1010d500 0x7fba1019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba10108f40 0x7fba101a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 --2- 172.21.15.146:0/2227432295 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba10108580 0x7fba101a4900 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 >> 172.21.15.146:0/2227432295 conn(0x7fba10077e20 msgr2=0x7fba100782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 shutdown_connections 2024-04-24T08:29:36.666 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.655+0000 7fba15000640 1 -- 172.21.15.146:0/2227432295 wait complete. 2024-04-24T08:29:36.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:29:36.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 -- 172.21.15.146:0/710444836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 msgr2=0x7f576c0746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.872 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 --2- 172.21.15.146:0/710444836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 0x7f576c0746b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f575c009900 tx=0x7f575c02a480 comp rx=0 tx=0).stop 2024-04-24T08:29:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 -- 172.21.15.146:0/710444836 shutdown_connections 2024-04-24T08:29:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 --2- 172.21.15.146:0/710444836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c10e8e0 0x7f576c110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 --2- 172.21.15.146:0/710444836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f576c074bf0 0x7f576c10e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 --2- 172.21.15.146:0/710444836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 0x7f576c0746b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 -- 172.21.15.146:0/710444836 >> 172.21.15.146:0/710444836 conn(0x7f576c079450 msgr2=0x7f576c079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 -- 172.21.15.146:0/710444836 shutdown_connections 2024-04-24T08:29:36.873 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.871+0000 7f5771200640 1 -- 172.21.15.146:0/710444836 wait complete. 2024-04-24T08:29:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.872+0000 7f5771200640 1 Processor -- start 2024-04-24T08:29:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.872+0000 7f5771200640 1 -- start start 2024-04-24T08:29:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5771200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 0x7f576c1a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5771200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c074bf0 0x7f576c1a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.876 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5771200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f576c10e8e0 0x7f576c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5771200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f576c114620 con 0x7f576c10e8e0 2024-04-24T08:29:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5771200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1144a0 con 0x7f576c074bf0 2024-04-24T08:29:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5771200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1147a0 con 0x7f576c0742d0 2024-04-24T08:29:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5761400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c074bf0 0x7f576c1a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5761400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c074bf0 0x7f576c1a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48218/0 (socket says 172.21.15.146:48218) 2024-04-24T08:29:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5761400640 1 -- 172.21.15.146:0/922453599 learned_addr learned my addr 172.21.15.146:0/922453599 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.877 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f5769e00640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 0x7f576c1a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.873+0000 7f576a800640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f576c10e8e0 0x7f576c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5761400640 1 -- 172.21.15.146:0/922453599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 msgr2=0x7f576c1a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5761400640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 0x7f576c1a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5761400640 1 -- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f576c10e8e0 msgr2=0x7f576c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5761400640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f576c10e8e0 0x7f576c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5761400640 1 -- 172.21.15.146:0/922453599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576c19b0c0 con 0x7f576c074bf0 2024-04-24T08:29:36.878 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f576a800640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f576c10e8e0 0x7f576c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:36.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5761400640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c074bf0 0x7f576c1a0cd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f575400e9f0 tx=0x7f575400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5763400640 1 -- 172.21.15.146:0/922453599 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575400cd80 con 0x7f576c074bf0 2024-04-24T08:29:36.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5763400640 1 -- 172.21.15.146:0/922453599 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5754010910 con 0x7f576c074bf0 2024-04-24T08:29:36.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576c19b3b0 con 0x7f576c074bf0 2024-04-24T08:29:36.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5763400640 1 -- 172.21.15.146:0/922453599 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754018990 con 0x7f576c074bf0 2024-04-24T08:29:36.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.874+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f576c1a7650 con 0x7f576c074bf0 2024-04-24T08:29:36.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.877+0000 7f577406e740 1 Processor -- start 2024-04-24T08:29:36.887 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f577406e740 1 -- start start 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f577406e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c974d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f5763400640 1 -- 172.21.15.146:0/922453599 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f5754010ac0 con 0x7f576c074bf0 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f577406e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c975270 0x555e9c97f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f577406e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c97f940 0x555e9c981d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f577406e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555e9c8da1d0 con 0x555e9c975270 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f577406e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555e9c880450 con 0x555e9c974950 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f577406e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555e9c8d6190 con 0x555e9c97f940 2024-04-24T08:29:36.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f5769e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c975270 0x555e9c97f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f5769e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c975270 0x555e9c97f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40264/0 (socket says 172.21.15.146:40264) 2024-04-24T08:29:36.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f5761400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c97f940 0x555e9c981d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f5769e00640 1 -- 172.21.15.146:0/82392290 learned_addr learned my addr 172.21.15.146:0/82392290 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.878+0000 7f576a800640 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c974d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f5769e00640 1 -- 172.21.15.146:0/82392290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c97f940 msgr2=0x555e9c981d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f5769e00640 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c97f940 0x555e9c981d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.889 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f5769e00640 1 -- 172.21.15.146:0/82392290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 msgr2=0x555e9c974d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f5769e00640 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c974d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f5769e00640 1 -- 172.21.15.146:0/82392290 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e9c93d310 con 0x555e9c975270 2024-04-24T08:29:36.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f5761400640 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c97f940 0x555e9c981d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:36.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f576a800640 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c974d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:36.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f5769e00640 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c975270 0x555e9c97f400 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f575c009900 tx=0x7f575c005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac9391d36680bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.879+0000 7f575b400640 1 -- 172.21.15.146:0/82392290 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c035040 con 0x555e9c975270 2024-04-24T08:29:36.890 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f575b400640 1 -- 172.21.15.146:0/82392290 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575c005520 con 0x555e9c975270 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f575b400640 1 -- 172.21.15.146:0/82392290 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c0035f0 con 0x555e9c975270 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f5763400640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f5763400640 1 -- 172.21.15.146:0/922453599 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f57540d0c40 con 0x7f576c074bf0 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f576a800640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 -- 172.21.15.146:0/82392290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c975270 msgr2=0x555e9c97f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c975270 0x555e9c97f400 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f575c009900 tx=0x7f575c005de0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 -- 172.21.15.146:0/82392290 shutdown_connections 2024-04-24T08:29:36.891 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c97f940 0x555e9c981d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c975270 0x555e9c97f400 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 --2- 172.21.15.146:0/82392290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c974d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 -- 172.21.15.146:0/82392290 >> 172.21.15.146:0/82392290 conn(0x555e9c970c70 msgr2=0x555e9c971050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.880+0000 7f577406e740 1 -- 172.21.15.146:0/82392290 shutdown_connections 2024-04-24T08:29:36.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.881+0000 7f577406e740 1 -- 172.21.15.146:0/82392290 wait complete. 2024-04-24T08:29:36.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.881+0000 7f576a800640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f57500a3d20 0x7f57500a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f576c06b8d0 tx=0x7f5764009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.881+0000 7f577406e740 1 -- start start 2024-04-24T08:29:36.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.881+0000 7f577406e740 1 Processor -- start 2024-04-24T08:29:36.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f577406e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c97f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f577406e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c97c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f577406e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c97cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f577406e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555e9c982770 con 0x555e9c97f940 2024-04-24T08:29:36.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f577406e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555e9c893810 con 0x555e9c975270 2024-04-24T08:29:36.893 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f577406e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555e9c9828f0 con 0x555e9c974950 2024-04-24T08:29:36.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f577406e740 10 client.0 _ll_register_callbacks cb 0x555e9c861140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:36.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5761400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c97c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5769e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c97f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5761400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c97c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48238/0 (socket says 172.21.15.146:48238) 2024-04-24T08:29:36.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f576a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c97cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5761400640 1 -- 172.21.15.146:0/67406761 learned_addr learned my addr 172.21.15.146:0/67406761 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.894 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5761400640 1 -- 172.21.15.146:0/67406761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 msgr2=0x555e9c97f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5761400640 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c97f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5761400640 1 -- 172.21.15.146:0/67406761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 msgr2=0x555e9c97cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.882+0000 7f5761400640 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c97cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f5761400640 1 -- 172.21.15.146:0/67406761 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e9c8a43f0 con 0x555e9c975270 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f5769e00640 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c97f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f5761400640 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c97c650 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f57540ce380 tx=0x7f5754098f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a58a3f1a98438427 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f574fe00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:29:36.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f574fe00640 1 -- 172.21.15.146:0/67406761 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57540ce7a0 con 0x555e9c975270 2024-04-24T08:29:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f575c005520 con 0x555e9c975270 2024-04-24T08:29:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f574fe00640 1 -- 172.21.15.146:0/67406761 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f5754018b30 con 0x555e9c975270 2024-04-24T08:29:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f574fe00640 1 -- 172.21.15.146:0/67406761 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f57540d4020 con 0x555e9c975270 2024-04-24T08:29:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f577406e740 10 client.27077 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.883+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555e9c5ec3f0 con 0x555e9c975270 2024-04-24T08:29:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f574fe00640 1 -- 172.21.15.146:0/67406761 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575c005520 con 0x555e9c975270 2024-04-24T08:29:36.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f574fe00640 1 -- 172.21.15.146:0/67406761 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f57540d7070 con 0x555e9c975270 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f577406e740 10 client.27077 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f577406e740 10 client.27077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f577406e740 20 client.27077 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f577406e740 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c98fa90 0x555e9c991ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f577406e740 4 client.27077 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e9c8e7bf0 con 0x555e9c98fa90 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.884+0000 7f5769e00640 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c98fa90 0x555e9c991ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.897 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.885+0000 7f5769e00640 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c98fa90 0x555e9c991ec0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1492bfd68a968e42 server_cookie=f83c37c291f4767c in_seq=0 out_seq=0 2024-04-24T08:29:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.885+0000 7f574fe00640 10 client.27077 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f574fe00640 1 -- 172.21.15.146:0/67406761 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e9c8e7bf0 con 0x555e9c98fa90 2024-04-24T08:29:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f574fe00640 10 client.27077 handle_command_reply: tid=0 2024-04-24T08:29:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 client.27077 shutdown 2024-04-24T08:29:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 msgr2=0x555e9c97c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c97c650 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f57540ce380 tx=0x7f5754098f40 comp rx=0 tx=0).stop 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 shutdown_connections 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c98fa90 0x555e9c991ec0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c97cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f574fe00640 0 client.27077 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c97c650 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 --2- 172.21.15.146:0/67406761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c97f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 >> 172.21.15.146:0/67406761 conn(0x555e9c970c70 msgr2=0x555e9c96fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 shutdown_connections 2024-04-24T08:29:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 1 -- 172.21.15.146:0/67406761 wait complete. 2024-04-24T08:29:36.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.886+0000 7f577406e740 20 client.27077 trim_cache size 0 max 16384 2024-04-24T08:29:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.901+0000 7f577406e740 1 Processor -- start 2024-04-24T08:29:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.901+0000 7f577406e740 1 -- start start 2024-04-24T08:29:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.901+0000 7f577406e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c970570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.901+0000 7f577406e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c975270 0x555e9c97ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.901+0000 7f577406e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c9977d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.901+0000 7f577406e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1b2840 con 0x555e9c97f940 2024-04-24T08:29:36.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.901+0000 7f577406e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1b0110 con 0x555e9c974950 2024-04-24T08:29:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f577406e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1adf80 con 0x555e9c975270 2024-04-24T08:29:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f576a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c9977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f576a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c9977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40274/0 (socket says 172.21.15.146:40274) 2024-04-24T08:29:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f576a800640 1 -- 172.21.15.146:0/1102088564 learned_addr learned my addr 172.21.15.146:0/1102088564 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f5769e00640 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c970570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f5761400640 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c975270 0x555e9c97ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f5769e00640 1 -- 172.21.15.146:0/1102088564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c975270 msgr2=0x555e9c97ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f5769e00640 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c975270 0x555e9c97ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f5769e00640 1 -- 172.21.15.146:0/1102088564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 msgr2=0x555e9c9977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f5769e00640 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c9977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f5769e00640 1 -- 172.21.15.146:0/1102088564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e9c9750a0 con 0x555e9c974950 2024-04-24T08:29:36.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.902+0000 7f576a800640 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c9977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:36.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f5769e00640 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c970570 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f575c031550 tx=0x7f575c004be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35a34f5b92d17d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f575aa00640 1 -- 172.21.15.146:0/1102088564 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c059070 con 0x555e9c974950 2024-04-24T08:29:36.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f575aa00640 1 -- 172.21.15.146:0/1102088564 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575c0042c0 con 0x555e9c974950 2024-04-24T08:29:36.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f575aa00640 1 -- 172.21.15.146:0/1102088564 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c035040 con 0x555e9c974950 2024-04-24T08:29:36.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f577406e740 1 -- 172.21.15.146:0/1102088564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 msgr2=0x555e9c970570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f577406e740 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c970570 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f575c031550 tx=0x7f575c004be0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f577406e740 1 -- 172.21.15.146:0/1102088564 shutdown_connections 2024-04-24T08:29:36.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f577406e740 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c975270 0x555e9c97ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f577406e740 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c9977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.903+0000 7f577406e740 1 --2- 172.21.15.146:0/1102088564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c974950 0x555e9c970570 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.904+0000 7f577406e740 1 -- 172.21.15.146:0/1102088564 >> 172.21.15.146:0/1102088564 conn(0x555e9c970c70 msgr2=0x555e9c971050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.904+0000 7f577406e740 1 -- 172.21.15.146:0/1102088564 shutdown_connections 2024-04-24T08:29:36.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.904+0000 7f577406e740 1 -- 172.21.15.146:0/1102088564 wait complete. 2024-04-24T08:29:36.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.904+0000 7f577406e740 1 -- start start 2024-04-24T08:29:36.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.904+0000 7f577406e740 1 Processor -- start 2024-04-24T08:29:36.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f577406e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c980510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f577406e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c980a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f577406e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c986ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f577406e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1ae5d0 con 0x555e9c97f940 2024-04-24T08:29:36.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f577406e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1afa40 con 0x555e9c975270 2024-04-24T08:29:36.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f577406e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f576c1aeca0 con 0x555e9c974950 2024-04-24T08:29:36.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f577406e740 10 client.0 _ll_register_callbacks cb 0x7f57540d43e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:36.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c980510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f576a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c986ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c980510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40780/0 (socket says 172.21.15.146:40780) 2024-04-24T08:29:36.914 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5761400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c980a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 -- 172.21.15.146:0/2002803981 learned_addr learned my addr 172.21.15.146:0/2002803981 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:36.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 -- 172.21.15.146:0/2002803981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 msgr2=0x555e9c980a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c980a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 -- 172.21.15.146:0/2002803981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 msgr2=0x555e9c986ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:36.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c986ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.905+0000 7f5769e00640 1 -- 172.21.15.146:0/2002803981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e9c6b67d0 con 0x555e9c974950 2024-04-24T08:29:36.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f5769e00640 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c980510 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f575c005e20 tx=0x7f575c0040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf145df027f80a36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:36.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f574fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:36.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f574fe00640 1 -- 172.21.15.146:0/2002803981 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c059070 con 0x555e9c974950 2024-04-24T08:29:36.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f575c0042c0 con 0x555e9c974950 2024-04-24T08:29:36.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f574fe00640 1 -- 172.21.15.146:0/2002803981 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f575c0315b0 con 0x555e9c974950 2024-04-24T08:29:36.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f574fe00640 1 -- 172.21.15.146:0/2002803981 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f575c035040 con 0x555e9c974950 2024-04-24T08:29:36.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f577406e740 10 client.27079 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:36.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555e9c988340 con 0x555e9c974950 2024-04-24T08:29:36.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.906+0000 7f574fe00640 1 -- 172.21.15.146:0/2002803981 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575c0042c0 con 0x555e9c974950 2024-04-24T08:29:36.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f574fe00640 1 -- 172.21.15.146:0/2002803981 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f575c003d00 con 0x555e9c974950 2024-04-24T08:29:36.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f577406e740 10 client.27079 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:36.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f577406e740 10 client.27079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:36.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f577406e740 20 client.27079 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:36.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f577406e740 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c9f6090 0x555e9c9f8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:36.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f577406e740 4 client.27079 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555e9c8e1a60 con 0x555e9c9f6090 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f5761400640 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c9f6090 0x555e9c9f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f5761400640 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c9f6090 0x555e9c9f8480 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2c02779dfc2d3b server_cookie=7c0f07a98d45bafe in_seq=0 out_seq=0 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.907+0000 7f574fe00640 10 client.27079 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f574fe00640 1 -- 172.21.15.146:0/2002803981 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555e9c8e1a60 con 0x555e9c9f6090 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f574fe00640 10 client.27079 handle_command_reply: tid=0 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 client.27079 shutdown 2024-04-24T08:29:36.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 msgr2=0x555e9c980510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c980510 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f575c005e20 tx=0x7f575c0040e0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 shutdown_connections 2024-04-24T08:29:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x555e9c9f6090 0x555e9c9f8480 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555e9c97f940 0x555e9c986ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555e9c975270 0x555e9c980a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 --2- 172.21.15.146:0/2002803981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555e9c974950 0x555e9c980510 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f574fe00640 0 client.27079 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:36.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 >> 172.21.15.146:0/2002803981 conn(0x555e9c970c70 msgr2=0x555e9c97aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 shutdown_connections 2024-04-24T08:29:36.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 1 -- 172.21.15.146:0/2002803981 wait complete. 2024-04-24T08:29:36.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.908+0000 7f577406e740 20 client.27079 trim_cache size 0 max 16384 2024-04-24T08:29:36.920 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:29:36.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f57500a3d20 msgr2=0x7f57500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f57500a3d20 0x7f57500a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f576c06b8d0 tx=0x7f5764009500 comp rx=0 tx=0).stop 2024-04-24T08:29:36.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c074bf0 msgr2=0x7f576c1a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:36.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c074bf0 0x7f576c1a0cd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f575400e9f0 tx=0x7f575400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 shutdown_connections 2024-04-24T08:29:36.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f576c10e8e0 0x7f576c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f576c074bf0 0x7f576c1a0cd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 --2- 172.21.15.146:0/922453599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576c0742d0 0x7f576c1a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:36.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 >> 172.21.15.146:0/922453599 conn(0x7f576c079450 msgr2=0x7f576c0fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:36.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 shutdown_connections 2024-04-24T08:29:36.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:36.909+0000 7f5771200640 1 -- 172.21.15.146:0/922453599 wait complete. 2024-04-24T08:29:37.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:36 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1956453595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:29:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:36 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1956453595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:29:37.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:36 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1956453595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:29:37.929 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:29:37.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:29:37.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:29:38.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 -- 172.21.15.146:0/3179307174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90601044b0 msgr2=0x7f9060104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 --2- 172.21.15.146:0/3179307174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90601044b0 0x7f9060104890 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9050009860 tx=0x7f905002a340 comp rx=0 tx=0).stop 2024-04-24T08:29:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 -- 172.21.15.146:0/3179307174 shutdown_connections 2024-04-24T08:29:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 --2- 172.21.15.146:0/3179307174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060109580 0x7f9060111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 --2- 172.21.15.146:0/3179307174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9060104e60 0x7f9060108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 --2- 172.21.15.146:0/3179307174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90601044b0 0x7f9060104890 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 -- 172.21.15.146:0/3179307174 >> 172.21.15.146:0/3179307174 conn(0x7f90600ffd90 msgr2=0x7f90601021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 -- 172.21.15.146:0/3179307174 shutdown_connections 2024-04-24T08:29:38.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.148+0000 7f9066200640 1 -- 172.21.15.146:0/3179307174 wait complete. 2024-04-24T08:29:38.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.149+0000 7f9066200640 1 Processor -- start 2024-04-24T08:29:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.149+0000 7f9066200640 1 -- start start 2024-04-24T08:29:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f9066200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90601044b0 0x7f90601a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f9066200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060109580 0x7f90601a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f9066200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f906019a9b0 0x7f906019ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f9066200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9060114610 con 0x7f90601044b0 2024-04-24T08:29:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f9066200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9060114490 con 0x7f906019a9b0 2024-04-24T08:29:38.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f9066200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9060114790 con 0x7f9060109580 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f905f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90601044b0 0x7f90601a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f9056a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060109580 0x7f90601a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f905f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90601044b0 0x7f90601a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40292/0 (socket says 172.21.15.146:40292) 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f905f400640 1 -- 172.21.15.146:0/548879702 learned_addr learned my addr 172.21.15.146:0/548879702 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.150+0000 7f905fe00640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f906019a9b0 0x7f906019ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905f400640 1 -- 172.21.15.146:0/548879702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060109580 msgr2=0x7f90601a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905f400640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060109580 0x7f90601a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905f400640 1 -- 172.21.15.146:0/548879702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f906019a9b0 msgr2=0x7f906019ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905f400640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f906019a9b0 0x7f906019ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905f400640 1 -- 172.21.15.146:0/548879702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906019b4d0 con 0x7f90601044b0 2024-04-24T08:29:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f9056a00640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060109580 0x7f90601a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905f400640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90601044b0 0x7f90601a07b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9050035040 tx=0x7f9050033b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905cc00640 1 -- 172.21.15.146:0/548879702 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9050037040 con 0x7f90601044b0 2024-04-24T08:29:38.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905cc00640 1 -- 172.21.15.146:0/548879702 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9050040070 con 0x7f90601044b0 2024-04-24T08:29:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906019b760 con 0x7f90601044b0 2024-04-24T08:29:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f905cc00640 1 -- 172.21.15.146:0/548879702 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90500039b0 con 0x7f90601044b0 2024-04-24T08:29:38.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.151+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90601a7900 con 0x7f90601044b0 2024-04-24T08:29:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f905cc00640 1 -- 172.21.15.146:0/548879702 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f90500305e0 con 0x7f90601044b0 2024-04-24T08:29:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 Processor -- start 2024-04-24T08:29:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 -- start start 2024-04-24T08:29:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558973624090 0x558973624470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x55897362eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973631470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558973589e70 con 0x5589736249b0 2024-04-24T08:29:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558973530380 con 0x55897362f080 2024-04-24T08:29:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.155+0000 7f9068f88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558973585e30 con 0x558973624090 2024-04-24T08:29:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558973624090 0x558973624470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f9056a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973631470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558973624090 0x558973624470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40802/0 (socket says 172.21.15.146:40802) 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 -- 172.21.15.146:0/4040042027 learned_addr learned my addr 172.21.15.146:0/4040042027 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905f400640 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x55897362eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 -- 172.21.15.146:0/4040042027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 msgr2=0x558973631470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973631470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 -- 172.21.15.146:0/4040042027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 msgr2=0x55897362eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x55897362eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 -- 172.21.15.146:0/4040042027 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589733647e0 con 0x558973624090 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f9056a00640 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973631470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:38.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905cc00640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f90400a3d20 0x7f90400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.156+0000 7f905fe00640 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558973624090 0x558973624470 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f905800e480 tx=0x7f90580095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4789de03ba77477d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.157+0000 7f905cc00640 1 -- 172.21.15.146:0/548879702 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f90500f06f0 con 0x7f90601044b0 2024-04-24T08:29:38.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.157+0000 7f905f400640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f90400a3d20 0x7f90400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.157+0000 7f904fe00640 1 -- 172.21.15.146:0/4040042027 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058003970 con 0x558973624090 2024-04-24T08:29:38.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.157+0000 7f904fe00640 1 -- 172.21.15.146:0/4040042027 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905800d7d0 con 0x558973624090 2024-04-24T08:29:38.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.157+0000 7f904fe00640 1 -- 172.21.15.146:0/4040042027 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058004490 con 0x558973624090 2024-04-24T08:29:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 -- 172.21.15.146:0/4040042027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558973624090 msgr2=0x558973624470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f905f400640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f90400a3d20 0x7f90400a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x558973625650 tx=0x7f90500f6530 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558973624090 0x558973624470 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f905800e480 tx=0x7f90580095c0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 -- 172.21.15.146:0/4040042027 shutdown_connections 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973631470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x55897362eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 --2- 172.21.15.146:0/4040042027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558973624090 0x558973624470 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 -- 172.21.15.146:0/4040042027 >> 172.21.15.146:0/4040042027 conn(0x5589736203b0 msgr2=0x558973620790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 -- 172.21.15.146:0/4040042027 shutdown_connections 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 -- 172.21.15.146:0/4040042027 wait complete. 2024-04-24T08:29:38.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 -- start start 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.158+0000 7f9068f88740 1 Processor -- start 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9068f88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x55897362c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9068f88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x55897362c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9068f88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x55897362ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9068f88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558973631e70 con 0x558973624090 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9068f88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558973543660 con 0x55897362f080 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9068f88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558973631ff0 con 0x5589736249b0 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9068f88740 10 client.0 _ll_register_callbacks cb 0x558973531aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:38.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9056a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x55897362c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f905fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x55897362c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9056a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x55897362c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40804/0 (socket says 172.21.15.146:40804) 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f9056a00640 1 -- 172.21.15.146:0/1337412935 learned_addr learned my addr 172.21.15.146:0/1337412935 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.159+0000 7f905fe00640 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x55897362ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9056a00640 1 -- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 msgr2=0x55897362ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9056a00640 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x55897362ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9056a00640 1 -- 172.21.15.146:0/1337412935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 msgr2=0x55897362c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9056a00640 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x55897362c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9056a00640 1 -- 172.21.15.146:0/1337412935 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589733c4f70 con 0x5589736249b0 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f905fe00640 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x55897362ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9056a00640 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x55897362c550 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9048012d50 tx=0x7f904800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=632ec41c19e0b213 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f904cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f904cc00640 1 -- 172.21.15.146:0/1337412935 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90480120d0 con 0x5589736249b0 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f904cc00640 1 -- 172.21.15.146:0/1337412935 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f9048004030 con 0x5589736249b0 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f905800d7d0 con 0x5589736249b0 2024-04-24T08:29:38.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f904cc00640 1 -- 172.21.15.146:0/1337412935 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9048012270 con 0x5589736249b0 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9068f88740 10 client.27091 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.160+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55897325c170 con 0x5589736249b0 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f904cc00640 1 -- 172.21.15.146:0/1337412935 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905800d7d0 con 0x5589736249b0 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f904cc00640 1 -- 172.21.15.146:0/1337412935 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f904801c460 con 0x5589736249b0 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f9068f88740 10 client.27091 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f9068f88740 10 client.27091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f9068f88740 20 client.27091 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f9068f88740 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55897363f4b0 0x5589736418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f9068f88740 4 client.27091 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:38.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.161+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558973597890 con 0x55897363f4b0 2024-04-24T08:29:38.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.162+0000 7f905fe00640 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55897363f4b0 0x5589736418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.162+0000 7f905fe00640 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55897363f4b0 0x5589736418c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=410572e1955de2fa server_cookie=db965b8a6a51c744 in_seq=0 out_seq=0 2024-04-24T08:29:38.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.162+0000 7f904cc00640 10 client.27091 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f904cc00640 1 -- 172.21.15.146:0/1337412935 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558973597890 con 0x55897363f4b0 2024-04-24T08:29:38.172 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f904cc00640 10 client.27091 handle_command_reply: tid=0 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 client.27091 shutdown 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 msgr2=0x55897362c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x55897362c550 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9048012d50 tx=0x7f904800f790 comp rx=0 tx=0).stop 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 shutdown_connections 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55897363f4b0 0x5589736418c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x55897362c550 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x55897362ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f904cc00640 0 client.27091 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 --2- 172.21.15.146:0/1337412935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x55897362c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 >> 172.21.15.146:0/1337412935 conn(0x5589736203b0 msgr2=0x5589736261b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 shutdown_connections 2024-04-24T08:29:38.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.163+0000 7f9068f88740 1 -- 172.21.15.146:0/1337412935 wait complete. 2024-04-24T08:29:38.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.164+0000 7f9068f88740 20 client.27091 trim_cache size 0 max 16384 2024-04-24T08:29:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 Processor -- start 2024-04-24T08:29:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 -- start start 2024-04-24T08:29:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x5589736267f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x5589736471d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973647710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f90601b2850 con 0x558973624090 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90601b0120 con 0x55897362f080 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f9068f88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f90601adf90 con 0x5589736249b0 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f905fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x5589736267f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f905fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x5589736267f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40322/0 (socket says 172.21.15.146:40322) 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.183+0000 7f905fe00640 1 -- 172.21.15.146:0/171810552 learned_addr learned my addr 172.21.15.146:0/171810552 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f9056a00640 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x5589736471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f905fe00640 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973647710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f905fe00640 1 -- 172.21.15.146:0/171810552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 msgr2=0x5589736471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f905fe00640 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x5589736471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f905fe00640 1 -- 172.21.15.146:0/171810552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 msgr2=0x558973647710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f905fe00640 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973647710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f905fe00640 1 -- 172.21.15.146:0/171810552 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589736247e0 con 0x558973624090 2024-04-24T08:29:38.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.184+0000 7f905fe00640 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x5589736267f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f905800dda0 tx=0x7f90580040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb737e604911bd75 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f904f400640 1 -- 172.21.15.146:0/171810552 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905800d7d0 con 0x558973624090 2024-04-24T08:29:38.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f904f400640 1 -- 172.21.15.146:0/171810552 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905802a070 con 0x558973624090 2024-04-24T08:29:38.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f904f400640 1 -- 172.21.15.146:0/171810552 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90580054a0 con 0x558973624090 2024-04-24T08:29:38.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 -- 172.21.15.146:0/171810552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 msgr2=0x5589736267f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x5589736267f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f905800dda0 tx=0x7f90580040f0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 -- 172.21.15.146:0/171810552 shutdown_connections 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5589736249b0 0x5589736471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55897362f080 0x558973647710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 --2- 172.21.15.146:0/171810552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558973624090 0x5589736267f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 -- 172.21.15.146:0/171810552 >> 172.21.15.146:0/171810552 conn(0x5589736203b0 msgr2=0x55897361ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.185+0000 7f9068f88740 1 -- 172.21.15.146:0/171810552 shutdown_connections 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.186+0000 7f9068f88740 1 -- 172.21.15.146:0/171810552 wait complete. 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.186+0000 7f9068f88740 1 -- start start 2024-04-24T08:29:38.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.186+0000 7f9068f88740 1 Processor -- start 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9068f88740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558973624090 0x558973637240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9068f88740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x558973637780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9068f88740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55897362f080 0x558973634300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9068f88740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f90601ae5e0 con 0x5589736249b0 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9068f88740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90601afa50 con 0x558973624090 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9068f88740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f90601aecb0 con 0x55897362f080 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9068f88740 10 client.0 _ll_register_callbacks cb 0x7f9048009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9056a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x558973637780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f905fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558973624090 0x558973637240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f905fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558973624090 0x558973637240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48318/0 (socket says 172.21.15.146:48318) 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f905fe00640 1 -- 172.21.15.146:0/2076261454 learned_addr learned my addr 172.21.15.146:0/2076261454 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f905fe00640 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55897362f080 0x558973634300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9056a00640 1 -- 172.21.15.146:0/2076261454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55897362f080 msgr2=0x558973634300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9056a00640 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55897362f080 0x558973634300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9056a00640 1 -- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558973624090 msgr2=0x558973637240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.187+0000 7f9056a00640 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558973624090 0x558973637240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f9056a00640 1 -- 172.21.15.146:0/2076261454 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558973634b30 con 0x5589736249b0 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f905fe00640 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558973624090 0x558973637240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f9056a00640 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x558973637780 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f904801c900 tx=0x7f904801dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60751e1fd04992a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f904cc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f904cc00640 1 -- 172.21.15.146:0/2076261454 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048029040 con 0x5589736249b0 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f904cc00640 1 -- 172.21.15.146:0/2076261454 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7f9048005040 con 0x5589736249b0 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f905802a070 con 0x5589736249b0 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f904cc00640 1 -- 172.21.15.146:0/2076261454 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f90480120d0 con 0x5589736249b0 2024-04-24T08:29:38.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f9068f88740 10 client.16956 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.188+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5589736aeec0 con 0x5589736249b0 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f904cc00640 1 -- 172.21.15.146:0/2076261454 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905802a070 con 0x5589736249b0 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f904cc00640 1 -- 172.21.15.146:0/2076261454 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f90480035f0 con 0x5589736249b0 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f9068f88740 10 client.16956 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f9068f88740 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f9068f88740 20 client.16956 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f9068f88740 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5589736af670 0x5589736b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f9068f88740 4 client.16956 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:29:38.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.189+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558973591700 con 0x5589736af670 2024-04-24T08:29:38.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.190+0000 7f905fe00640 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5589736af670 0x5589736b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.190+0000 7f905fe00640 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5589736af670 0x5589736b1a90 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d1021b8c7ebda5a server_cookie=d17864c728625de0 in_seq=0 out_seq=0 2024-04-24T08:29:38.197 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.190+0000 7f904cc00640 10 client.16956 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f904cc00640 1 -- 172.21.15.146:0/2076261454 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558973591700 con 0x5589736af670 2024-04-24T08:29:38.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f904cc00640 10 client.16956 handle_command_reply: tid=0 2024-04-24T08:29:38.198 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 client.16956 shutdown 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 msgr2=0x558973637780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x558973637780 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f904801c900 tx=0x7f904801dd10 comp rx=0 tx=0).stop 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 shutdown_connections 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5589736af670 0x5589736b1a90 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55897362f080 0x558973634300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5589736249b0 0x558973637780 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 --2- 172.21.15.146:0/2076261454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558973624090 0x558973637240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.199 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.191+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 >> 172.21.15.146:0/2076261454 conn(0x5589736203b0 msgr2=0x55897361f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f904cc00640 0 client.16956 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 shutdown_connections 2024-04-24T08:29:38.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f9068f88740 1 -- 172.21.15.146:0/2076261454 wait complete. 2024-04-24T08:29:38.200 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f9068f88740 20 client.16956 trim_cache size 0 max 16384 2024-04-24T08:29:38.200 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:29:38.200 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:29:38.201 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "7a399780-6e43-4152-be0f-10b7f09d34dd", 2024-04-24T08:29:38.201 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:29:38.201 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:29:38.201 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f90400a3d20 msgr2=0x7f90400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f9066200640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f90400a3d20 0x7f90400a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x558973625650 tx=0x7f90500f6530 comp rx=0 tx=0).stop 2024-04-24T08:29:38.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90601044b0 msgr2=0x7f90601a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.192+0000 7f9066200640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90601044b0 0x7f90601a07b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9050035040 tx=0x7f9050033b60 comp rx=0 tx=0).stop 2024-04-24T08:29:38.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 shutdown_connections 2024-04-24T08:29:38.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f90400a3d20 0x7f90400a61e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060109580 0x7f90601a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.202 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f906019a9b0 0x7f906019ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 --2- 172.21.15.146:0/548879702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90601044b0 0x7f90601a07b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 >> 172.21.15.146:0/548879702 conn(0x7f90600ffd90 msgr2=0x7f90601010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 shutdown_connections 2024-04-24T08:29:38.203 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.193+0000 7f9066200640 1 -- 172.21.15.146:0/548879702 wait complete. 2024-04-24T08:29:38.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:37 smithi153 ceph-mon[32746]: pgmap v260: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:38.212 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:37 smithi146 ceph-mon[27805]: pgmap v260: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:38.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:29:38.216 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:29:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:37 smithi177 ceph-mon[33046]: pgmap v260: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:38.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.466+0000 7ff018a00640 1 -- 172.21.15.146:0/186887131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102ac0 msgr2=0x7ff014106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.466+0000 7ff018a00640 1 --2- 172.21.15.146:0/186887131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102ac0 0x7ff014106b00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7feffc00fb10 tx=0x7feffc02e570 comp rx=0 tx=0).stop 2024-04-24T08:29:38.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.466+0000 7ff018a00640 1 -- 172.21.15.146:0/186887131 shutdown_connections 2024-04-24T08:29:38.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.466+0000 7ff018a00640 1 --2- 172.21.15.146:0/186887131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102ac0 0x7ff014106b00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.466+0000 7ff018a00640 1 --2- 172.21.15.146:0/186887131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0141071e0 0x7ff01410edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.466+0000 7ff018a00640 1 --2- 172.21.15.146:0/186887131 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff014102110 0x7ff0141024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.466+0000 7ff018a00640 1 -- 172.21.15.146:0/186887131 >> 172.21.15.146:0/186887131 conn(0x7ff0140fd9d0 msgr2=0x7ff0140ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.467+0000 7ff018a00640 1 -- 172.21.15.146:0/186887131 shutdown_connections 2024-04-24T08:29:38.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.467+0000 7ff018a00640 1 -- 172.21.15.146:0/186887131 wait complete. 2024-04-24T08:29:38.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.468+0000 7ff018a00640 1 Processor -- start 2024-04-24T08:29:38.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.468+0000 7ff018a00640 1 -- start start 2024-04-24T08:29:38.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.468+0000 7ff018a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102110 0x7ff014117610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.468+0000 7ff018a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0141071e0 0x7ff014117b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.468+0000 7ff018a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff014111880 0x7ff014111d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.468+0000 7ff018a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0141100d0 con 0x7ff014111880 2024-04-24T08:29:38.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff018a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff01410ff50 con 0x7ff0141071e0 2024-04-24T08:29:38.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff018a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff014110250 con 0x7ff014102110 2024-04-24T08:29:38.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff012800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102110 0x7ff014117610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff012800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102110 0x7ff014117610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40848/0 (socket says 172.21.15.146:40848) 2024-04-24T08:29:38.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff011e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0141071e0 0x7ff014117b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff012800640 1 -- 172.21.15.146:0/3360343939 learned_addr learned my addr 172.21.15.146:0/3360343939 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff012800640 1 -- 172.21.15.146:0/3360343939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0141071e0 msgr2=0x7ff014117b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff012800640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0141071e0 0x7ff014117b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff012800640 1 -- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff014111880 msgr2=0x7ff014111d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.469+0000 7ff013200640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff014111880 0x7ff014111d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff012800640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff014111880 0x7ff014111d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff012800640 1 -- 172.21.15.146:0/3360343939 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0141b3e60 con 0x7ff014102110 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff013200640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff014111880 0x7ff014111d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff012800640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102110 0x7ff014117610 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff00400e9f0 tx=0x7ff00400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff00b400640 1 -- 172.21.15.146:0/3360343939 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00400cd80 con 0x7ff014102110 2024-04-24T08:29:38.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff00b400640 1 -- 172.21.15.146:0/3360343939 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff004010910 con 0x7ff014102110 2024-04-24T08:29:38.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff00b400640 1 -- 172.21.15.146:0/3360343939 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff004018990 con 0x7ff014102110 2024-04-24T08:29:38.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.470+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0141b4030 con 0x7ff014102110 2024-04-24T08:29:38.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.471+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0141b4440 con 0x7ff014102110 2024-04-24T08:29:38.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.475+0000 7ff01c277740 1 Processor -- start 2024-04-24T08:29:38.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.475+0000 7ff01c277740 1 -- start start 2024-04-24T08:29:38.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.475+0000 7ff00b400640 1 -- 172.21.15.146:0/3360343939 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7ff004018b50 con 0x7ff014102110 2024-04-24T08:29:38.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.475+0000 7ff01c277740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff01c277740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76db4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff01c277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76dde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff01c277740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55eba76361f0 con 0x55eba76d0a30 2024-04-24T08:29:38.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff01c277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eba75dc350 con 0x55eba76dba20 2024-04-24T08:29:38.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff01c277740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eba76321b0 con 0x55eba76d1350 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff012800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76db4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff012800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76db4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40862/0 (socket says 172.21.15.146:40862) 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.476+0000 7ff012800640 1 -- 172.21.15.146:0/702980200 learned_addr learned my addr 172.21.15.146:0/702980200 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.477+0000 7ff011e00640 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76dde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.477+0000 7ff013200640 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.477+0000 7ff012800640 1 -- 172.21.15.146:0/702980200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 msgr2=0x55eba76dde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.477+0000 7ff012800640 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76dde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.477+0000 7ff012800640 1 -- 172.21.15.146:0/702980200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 msgr2=0x55eba76d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.477+0000 7ff012800640 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.477+0000 7ff012800640 1 -- 172.21.15.146:0/702980200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eba7630f50 con 0x55eba76d1350 2024-04-24T08:29:38.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.478+0000 7ff012800640 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76db4e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff0040ce1d0 tx=0x7ff0040d9670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eca051857012a5f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.478+0000 7ff003400640 1 -- 172.21.15.146:0/702980200 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0040e0020 con 0x55eba76d1350 2024-04-24T08:29:38.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.478+0000 7ff00b400640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7feff80a3de0 0x7feff80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.479+0000 7ff00b400640 1 -- 172.21.15.146:0/3360343939 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7ff0040d0bb0 con 0x7ff014102110 2024-04-24T08:29:38.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.479+0000 7ff003400640 1 -- 172.21.15.146:0/702980200 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff00409d1e0 con 0x55eba76d1350 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.479+0000 7ff003400640 1 -- 172.21.15.146:0/702980200 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00409c440 con 0x55eba76d1350 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff013200640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7feff80a3de0 0x7feff80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 -- 172.21.15.146:0/702980200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 msgr2=0x55eba76db4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76db4e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff0040ce1d0 tx=0x7ff0040d9670 comp rx=0 tx=0).stop 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 -- 172.21.15.146:0/702980200 shutdown_connections 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76db4e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76dde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 --2- 172.21.15.146:0/702980200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 -- 172.21.15.146:0/702980200 >> 172.21.15.146:0/702980200 conn(0x55eba76ccd50 msgr2=0x55eba76cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 -- 172.21.15.146:0/702980200 shutdown_connections 2024-04-24T08:29:38.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.480+0000 7ff01c277740 1 -- 172.21.15.146:0/702980200 wait complete. 2024-04-24T08:29:38.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.481+0000 7ff013200640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7feff80a3de0 0x7feff80a62a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff00c005fd0 tx=0x7ff00c0093c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.481+0000 7ff01c277740 1 -- start start 2024-04-24T08:29:38.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.481+0000 7ff01c277740 1 Processor -- start 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff01c277740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d0a30 0x55eba76d89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff01c277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff01c277740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76dba20 0x55eba76d9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff01c277740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55eba76de7b0 con 0x55eba76dba20 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff01c277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55eba75ef690 con 0x55eba76d1350 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff01c277740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eba76de930 con 0x55eba76d0a30 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff012800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d0a30 0x55eba76d89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff012800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d0a30 0x55eba76d89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40868/0 (socket says 172.21.15.146:40868) 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff012800640 1 -- 172.21.15.146:0/2590016362 learned_addr learned my addr 172.21.15.146:0/2590016362 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff01c277740 10 client.0 _ll_register_callbacks cb 0x55eba762b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.482+0000 7ff011e00640 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7ff011e00640 1 -- 172.21.15.146:0/2590016362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d0a30 msgr2=0x55eba76d89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7ff011e00640 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d0a30 0x55eba76d89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7ff011e00640 1 -- 172.21.15.146:0/2590016362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76dba20 msgr2=0x55eba76d9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7ff011e00640 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76dba20 0x55eba76d9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7ff011e00640 1 -- 172.21.15.146:0/2590016362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eba75ee920 con 0x55eba76d1350 2024-04-24T08:29:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7ff012800640 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d0a30 0x55eba76d89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:29:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7ff011e00640 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d8ee0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7feffc00fa60 tx=0x7feffc00e440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d075ee869b292cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.483+0000 7feff7e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:29:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.484+0000 7feff7e00640 1 -- 172.21.15.146:0/2590016362 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc0048c0 con 0x55eba76d1350 2024-04-24T08:29:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.484+0000 7feff7e00640 1 -- 172.21.15.146:0/2590016362 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7feffc004a60 con 0x55eba76d1350 2024-04-24T08:29:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.484+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff00409d1e0 con 0x55eba76d1350 2024-04-24T08:29:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.484+0000 7feff7e00640 1 -- 172.21.15.146:0/2590016362 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feffc00ea20 con 0x55eba76d1350 2024-04-24T08:29:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.484+0000 7ff01c277740 10 client.27110 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.484+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eba74127d0 con 0x55eba76d1350 2024-04-24T08:29:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.485+0000 7feff7e00640 1 -- 172.21.15.146:0/2590016362 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff00409d1e0 con 0x55eba76d1350 2024-04-24T08:29:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.485+0000 7feff7e00640 1 -- 172.21.15.146:0/2590016362 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7feffc003890 con 0x55eba76d1350 2024-04-24T08:29:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.486+0000 7ff01c277740 10 client.27110 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.486+0000 7ff01c277740 10 client.27110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:38.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.486+0000 7ff01c277740 20 client.27110 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:38.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.486+0000 7ff01c277740 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba76ebd00 0x55eba76ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.486+0000 7ff01c277740 4 client.27110 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:29:38.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.486+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eba7643c10 con 0x55eba76ebd00 2024-04-24T08:29:38.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.486+0000 7ff012800640 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba76ebd00 0x55eba76ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.487+0000 7ff012800640 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba76ebd00 0x55eba76ee1a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33c1d06423aa214b server_cookie=5ee16180420c5891 in_seq=0 out_seq=0 2024-04-24T08:29:38.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.487+0000 7feff7e00640 10 client.27110 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.488+0000 7feff7e00640 1 -- 172.21.15.146:0/2590016362 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eba7643c10 con 0x55eba76ebd00 2024-04-24T08:29:38.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.488+0000 7feff7e00640 10 client.27110 handle_command_reply: tid=0 2024-04-24T08:29:38.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.488+0000 7ff01c277740 1 client.27110 shutdown 2024-04-24T08:29:38.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.488+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 msgr2=0x55eba76d8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.488+0000 7ff01c277740 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d8ee0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7feffc00fa60 tx=0x7feffc00e440 comp rx=0 tx=0).stop 2024-04-24T08:29:38.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 shutdown_connections 2024-04-24T08:29:38.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba76ebd00 0x55eba76ee1a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76dba20 0x55eba76d9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d8ee0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 --2- 172.21.15.146:0/2590016362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d0a30 0x55eba76d89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 >> 172.21.15.146:0/2590016362 conn(0x55eba76ccd50 msgr2=0x55eba76d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7feff7e00640 0 client.27110 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 shutdown_connections 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.489+0000 7ff01c277740 1 -- 172.21.15.146:0/2590016362 wait complete. 2024-04-24T08:29:38.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.490+0000 7ff01c277740 20 client.27110 trim_cache size 0 max 16384 2024-04-24T08:29:38.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.517+0000 7ff01c277740 1 Processor -- start 2024-04-24T08:29:38.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.517+0000 7ff01c277740 1 -- start start 2024-04-24T08:29:38.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.517+0000 7ff01c277740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff01c277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff01c277740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76dba20 0x55eba76f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff01c277740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0141bf3c0 con 0x55eba76d0a30 2024-04-24T08:29:38.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff01c277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0141bcc90 con 0x55eba76d1350 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff01c277740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff0141bab30 con 0x55eba76dba20 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76dba20 0x55eba76f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff012800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76dba20 0x55eba76f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:40882/0 (socket says 172.21.15.146:40882) 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 -- 172.21.15.146:0/2082033394 learned_addr learned my addr 172.21.15.146:0/2082033394 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff011e00640 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 -- 172.21.15.146:0/2082033394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 msgr2=0x55eba76d3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 -- 172.21.15.146:0/2082033394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 msgr2=0x55eba76e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.518+0000 7ff013200640 1 -- 172.21.15.146:0/2082033394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eba73483f0 con 0x55eba76dba20 2024-04-24T08:29:38.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.519+0000 7ff013200640 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76dba20 0x55eba76f3a70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff00c011600 tx=0x7ff00c013070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3ff485829d2b742 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.519+0000 7ff002a00640 1 -- 172.21.15.146:0/2082033394 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c01e6e0 con 0x55eba76dba20 2024-04-24T08:29:38.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.519+0000 7ff002a00640 1 -- 172.21.15.146:0/2082033394 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff00c01ed60 con 0x55eba76dba20 2024-04-24T08:29:38.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.519+0000 7ff002a00640 1 -- 172.21.15.146:0/2082033394 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c01c3b0 con 0x55eba76dba20 2024-04-24T08:29:38.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 -- 172.21.15.146:0/2082033394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76dba20 msgr2=0x55eba76f3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76dba20 0x55eba76f3a70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff00c011600 tx=0x7ff00c013070 comp rx=0 tx=0).stop 2024-04-24T08:29:38.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 -- 172.21.15.146:0/2082033394 shutdown_connections 2024-04-24T08:29:38.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76dba20 0x55eba76f3a70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76d1350 0x55eba76d3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 --2- 172.21.15.146:0/2082033394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 -- 172.21.15.146:0/2082033394 >> 172.21.15.146:0/2082033394 conn(0x55eba76ccd50 msgr2=0x55eba76d3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 -- 172.21.15.146:0/2082033394 shutdown_connections 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 -- 172.21.15.146:0/2082033394 wait complete. 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.520+0000 7ff01c277740 1 -- start start 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 1 Processor -- start 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76e40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff0141bb180 con 0x55eba76d0a30 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff0141bc5c0 con 0x55eba76dba20 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff0141bb850 con 0x55eba76d1350 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff01c277740 10 client.0 _ll_register_callbacks cb 0x7feffc00f7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff012800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff012800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:40364/0 (socket says 172.21.15.146:40364) 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff012800640 1 -- 172.21.15.146:0/4113964608 learned_addr learned my addr 172.21.15.146:0/4113964608 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff011e00640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76e40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.521+0000 7ff013200640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff012800640 1 -- 172.21.15.146:0/4113964608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 msgr2=0x55eba76e40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff012800640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76e40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff012800640 1 -- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 msgr2=0x55eba76d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff012800640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff012800640 1 -- 172.21.15.146:0/4113964608 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eba73080f0 con 0x55eba76d0a30 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff013200640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff012800640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e3b80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff004098f40 tx=0x7ff004018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8203a27cf134fe64 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:29:38.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7feff7e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7feff7e00640 1 -- 172.21.15.146:0/4113964608 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0040e0040 con 0x55eba76d0a30 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7feff7e00640 1 -- 172.21.15.146:0/4113964608 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8542+0+0 (secure 0 0 0) 0x7ff00409ade0 con 0x55eba76d0a30 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff00c01ed60 con 0x55eba76d0a30 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7feff7e00640 1 -- 172.21.15.146:0/4113964608 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff00409d1e0 con 0x55eba76d0a30 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff01c277740 10 client.16965 fetch_fsmap learned FSMap version 26 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.522+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eba75fb810 con 0x55eba76d0a30 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7feff7e00640 1 -- 172.21.15.146:0/4113964608 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff00c01ed60 con 0x55eba76d0a30 2024-04-24T08:29:38.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7feff7e00640 1 -- 172.21.15.146:0/4113964608 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff004099150 con 0x55eba76d0a30 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7ff01c277740 10 client.16965 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7ff01c277740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7ff01c277740 20 client.16965 populate_metadata read hostname 'smithi146' 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7ff01c277740 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba7751ac0 0x55eba7753eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7ff01c277740 4 client.16965 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eba763da80 con 0x55eba7751ac0 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.523+0000 7ff011e00640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba7751ac0 0x55eba7753eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:29:38.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff011e00640 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba7751ac0 0x55eba7753eb0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaed42435180f1e server_cookie=868a05158481ce43 in_seq=0 out_seq=0 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7feff7e00640 10 client.16965 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7feff7e00640 1 -- 172.21.15.146:0/4113964608 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55eba763da80 con 0x55eba7751ac0 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7feff7e00640 10 client.16965 handle_command_reply: tid=0 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 client.16965 shutdown 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 msgr2=0x55eba76e3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e3b80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff004098f40 tx=0x7ff004018420 comp rx=0 tx=0).stop 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 shutdown_connections 2024-04-24T08:29:38.533 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55eba7751ac0 0x55eba7753eb0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eba76d1350 0x55eba76e40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55eba76dba20 0x55eba76d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7feff7e00640 0 client.16965 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:29:38.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 --2- 172.21.15.146:0/4113964608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55eba76d0a30 0x55eba76e3b80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.524+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 >> 172.21.15.146:0/4113964608 conn(0x55eba76ccd50 msgr2=0x55eba76cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.525+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 shutdown_connections 2024-04-24T08:29:38.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.525+0000 7ff01c277740 1 -- 172.21.15.146:0/4113964608 wait complete. 2024-04-24T08:29:38.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.525+0000 7ff01c277740 20 client.16965 trim_cache size 0 max 16384 2024-04-24T08:29:38.536 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:29:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:29:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:29:38.536 INFO:teuthology.orchestra.run.smithi146.stdout: "7a399780-6e43-4152-be0f-10b7f09d34dd": { 2024-04-24T08:29:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:29:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "7a399780-6e43-4152-be0f-10b7f09d34dd", 2024-04-24T08:29:38.537 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:29:38.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:29:38.538 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:29:38.538 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:29:38.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.525+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7feff80a3de0 msgr2=0x7feff80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.525+0000 7ff018a00640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7feff80a3de0 0x7feff80a62a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff00c005fd0 tx=0x7ff00c0093c0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.525+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102110 msgr2=0x7ff014117610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:29:38.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.525+0000 7ff018a00640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102110 0x7ff014117610 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff00400e9f0 tx=0x7ff00400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 shutdown_connections 2024-04-24T08:29:38.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7feff80a3de0 0x7feff80a62a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff014111880 0x7ff014111d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0141071e0 0x7ff014117b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 --2- 172.21.15.146:0/3360343939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff014102110 0x7ff014117610 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:29:38.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 >> 172.21.15.146:0/3360343939 conn(0x7ff0140fd9d0 msgr2=0x7ff0140ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:29:38.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 shutdown_connections 2024-04-24T08:29:38.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:29:38.526+0000 7ff018a00640 1 -- 172.21.15.146:0/3360343939 wait complete. 2024-04-24T08:29:38.545 INFO:tasks.cephfs.filesystem:scrub status for tag:7a399780-6e43-4152-be0f-10b7f09d34dd - {'path': '/', 'tag': '7a399780-6e43-4152-be0f-10b7f09d34dd', 'options': 'recursive,force'} 2024-04-24T08:29:39.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:38 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:29:39.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:38 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:39.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:38 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:29:39.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:38 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:29:39.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:38 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:39.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:38 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:29:39.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:38 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:29:39.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:38 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:39.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:38 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:29:39.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:38 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:29:39.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:38 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:39.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:38 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:29:39.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:38 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:29:39.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:38 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:39.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:38 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:29:39.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:38 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:29:39.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:38 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:39.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:38 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:29:40.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:39 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:40.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:39 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:40.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:39 smithi153 ceph-mon[32746]: pgmap v261: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:40.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:39 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:40.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:39 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:40.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:39 smithi146 ceph-mon[27805]: pgmap v261: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:40.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:39 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:40.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:39 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:29:40.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:39 smithi177 ceph-mon[33046]: pgmap v261: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:43.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:42 smithi153 ceph-mon[32746]: pgmap v262: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:43.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:42 smithi146 ceph-mon[27805]: pgmap v262: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:43.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:42 smithi177 ceph-mon[33046]: pgmap v262: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:45.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:44 smithi153 ceph-mon[32746]: pgmap v263: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:45.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:44 smithi146 ceph-mon[27805]: pgmap v263: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:45.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:44 smithi177 ceph-mon[33046]: pgmap v263: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:47.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:46 smithi153 ceph-mon[32746]: pgmap v264: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:47.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:46 smithi146 ceph-mon[27805]: pgmap v264: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:47.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:46 smithi177 ceph-mon[33046]: pgmap v264: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:48.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:47 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:29:48.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:47 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:29:48.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:47 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:29:48.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:47 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:29:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:47 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:29:48.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:47 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:29:49.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:48 smithi153 ceph-mon[32746]: pgmap v265: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:49.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:48 smithi146 ceph-mon[27805]: pgmap v265: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:49.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:48 smithi177 ceph-mon[33046]: pgmap v265: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:51.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:50 smithi153 ceph-mon[32746]: pgmap v266: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:51.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:50 smithi146 ceph-mon[27805]: pgmap v266: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:51.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:50 smithi177 ceph-mon[33046]: pgmap v266: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:53.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:52 smithi153 ceph-mon[32746]: pgmap v267: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:53.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:52 smithi146 ceph-mon[27805]: pgmap v267: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:53.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:52 smithi177 ceph-mon[33046]: pgmap v267: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:29:54.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:53 smithi153 ceph-mon[32746]: pgmap v268: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:54.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:53 smithi146 ceph-mon[27805]: pgmap v268: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:54.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:53 smithi177 ceph-mon[33046]: pgmap v268: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:29:56.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:56 smithi153 ceph-mon[32746]: pgmap v269: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:56 smithi146 ceph-mon[27805]: pgmap v269: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:56 smithi177 ceph-mon[33046]: pgmap v269: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:29:58.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:29:58 smithi153 ceph-mon[32746]: pgmap v270: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:29:58 smithi146 ceph-mon[27805]: pgmap v270: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:29:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:29:58 smithi177 ceph-mon[33046]: pgmap v270: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:00.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:00 smithi153 ceph-mon[32746]: pgmap v271: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:00.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:00 smithi153 ceph-mon[32746]: overall HEALTH_OK 2024-04-24T08:30:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:00 smithi146 ceph-mon[27805]: pgmap v271: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:00.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:00 smithi146 ceph-mon[27805]: overall HEALTH_OK 2024-04-24T08:30:00.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:00 smithi177 ceph-mon[33046]: pgmap v271: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:00.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:00 smithi177 ceph-mon[33046]: overall HEALTH_OK 2024-04-24T08:30:02.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:02 smithi153 ceph-mon[32746]: pgmap v272: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:02.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:02 smithi146 ceph-mon[27805]: pgmap v272: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:02 smithi177 ceph-mon[33046]: pgmap v272: 97 pgs: 97 active+clean; 74 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:04.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:04 smithi153 ceph-mon[32746]: pgmap v273: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:30:04.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:04 smithi146 ceph-mon[27805]: pgmap v273: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:30:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:04 smithi177 ceph-mon[33046]: pgmap v273: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:30:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:06 smithi153 ceph-mon[32746]: pgmap v274: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:30:06.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:06 smithi146 ceph-mon[27805]: pgmap v274: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:30:06.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:06 smithi177 ceph-mon[33046]: pgmap v274: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:30:08.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:30:08.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:30:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:08 smithi153 ceph-mon[32746]: pgmap v275: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:30:08.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:08 smithi153 ceph-mon[32746]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T08:30:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:08 smithi146 ceph-mon[27805]: pgmap v275: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:30:08.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:08 smithi146 ceph-mon[27805]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T08:30:08.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.762+0000 7fe469600640 1 -- 172.21.15.146:0/961155784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4641044b0 msgr2=0x7fe464104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.762+0000 7fe469600640 1 --2- 172.21.15.146:0/961155784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4641044b0 0x7fe464104890 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe454009ca0 tx=0x7fe45402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.762+0000 7fe469600640 1 -- 172.21.15.146:0/961155784 shutdown_connections 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.762+0000 7fe469600640 1 --2- 172.21.15.146:0/961155784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe464104e60 0x7fe464108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.762+0000 7fe469600640 1 --2- 172.21.15.146:0/961155784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe464109580 0x7fe464111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.762+0000 7fe469600640 1 --2- 172.21.15.146:0/961155784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4641044b0 0x7fe464104890 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.762+0000 7fe469600640 1 -- 172.21.15.146:0/961155784 >> 172.21.15.146:0/961155784 conn(0x7fe4640ffd90 msgr2=0x7fe4641021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.763+0000 7fe469600640 1 -- 172.21.15.146:0/961155784 shutdown_connections 2024-04-24T08:30:08.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.763+0000 7fe469600640 1 -- 172.21.15.146:0/961155784 wait complete. 2024-04-24T08:30:08.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.764+0000 7fe469600640 1 Processor -- start 2024-04-24T08:30:08.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.764+0000 7fe469600640 1 -- start start 2024-04-24T08:30:08.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.764+0000 7fe469600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4641044b0 0x7fe4641a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.764+0000 7fe469600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe464109580 0x7fe4641a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe469600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe46419a980 0x7fe46419ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe469600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe464114660 con 0x7fe4641044b0 2024-04-24T08:30:08.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe469600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641144e0 con 0x7fe46419a980 2024-04-24T08:30:08.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe469600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641147e0 con 0x7fe464109580 2024-04-24T08:30:08.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4641044b0 0x7fe4641a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe464109580 0x7fe4641a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4641044b0 0x7fe4641a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56202/0 (socket says 172.21.15.146:56202) 2024-04-24T08:30:08.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462a00640 1 -- 172.21.15.146:0/152284839 learned_addr learned my addr 172.21.15.146:0/152284839 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:08.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe463400640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe46419a980 0x7fe46419ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462000640 1 -- 172.21.15.146:0/152284839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe46419a980 msgr2=0x7fe46419ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462000640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe46419a980 0x7fe46419ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462000640 1 -- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4641044b0 msgr2=0x7fe4641a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462000640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4641044b0 0x7fe4641a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462000640 1 -- 172.21.15.146:0/152284839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46419b4a0 con 0x7fe464109580 2024-04-24T08:30:08.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.765+0000 7fe462a00640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4641044b0 0x7fe4641a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:08.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.766+0000 7fe462000640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe464109580 0x7fe4641a0d10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e9c0 tx=0x7fe44c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:08.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.766+0000 7fe45b400640 1 -- 172.21.15.146:0/152284839 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c00ccc0 con 0x7fe464109580 2024-04-24T08:30:08.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.766+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46419b790 con 0x7fe464109580 2024-04-24T08:30:08.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.766+0000 7fe45b400640 1 -- 172.21.15.146:0/152284839 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe44c010910 con 0x7fe464109580 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.766+0000 7fe45b400640 1 -- 172.21.15.146:0/152284839 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c018880 con 0x7fe464109580 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.766+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4641a7a30 con 0x7fe464109580 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.769+0000 7fe46c47b740 1 Processor -- start 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe46c47b740 1 -- start start 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe45b400640 1 -- 172.21.15.146:0/152284839 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fe44c010ac0 con 0x7fe464109580 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991befc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bfa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bfcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe46c47b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560991b551f0 con 0x560991bfa850 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe46c47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560991afb350 con 0x560991bf0180 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe46c47b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560991b511b0 con 0x560991bef860 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe462a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bfa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe462000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bfcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe463400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991befc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe462a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bfa310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53436/0 (socket says 172.21.15.146:53436) 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.770+0000 7fe462a00640 1 -- 172.21.15.146:0/785348327 learned_addr learned my addr 172.21.15.146:0/785348327 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:08.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe463400640 1 -- 172.21.15.146:0/785348327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 msgr2=0x560991bfa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe463400640 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bfa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe463400640 1 -- 172.21.15.146:0/785348327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 msgr2=0x560991bfcc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe463400640 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bfcc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe463400640 1 -- 172.21.15.146:0/785348327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560991b4ff50 con 0x560991bef860 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe462a00640 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bfa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe463400640 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991befc40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009b80 tx=0x7fe45c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16be538f77c6eef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe45b400640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe4480a3d20 0x7fe4480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe453400640 1 -- 172.21.15.146:0/785348327 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c004050 con 0x560991bef860 2024-04-24T08:30:08.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe453400640 1 -- 172.21.15.146:0/785348327 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe45c005500 con 0x560991bef860 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.771+0000 7fe453400640 1 -- 172.21.15.146:0/785348327 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c004d00 con 0x560991bef860 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe45b400640 1 -- 172.21.15.146:0/152284839 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7fe44c0d0bd0 con 0x7fe464109580 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe462a00640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe4480a3d20 0x7fe4480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe46c47b740 1 -- 172.21.15.146:0/785348327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 msgr2=0x560991befc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe46c47b740 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991befc40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009b80 tx=0x7fe45c011810 comp rx=0 tx=0).stop 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe46c47b740 1 -- 172.21.15.146:0/785348327 shutdown_connections 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe46c47b740 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bfcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe46c47b740 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bfa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe46c47b740 1 --2- 172.21.15.146:0/785348327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991befc40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.772+0000 7fe46c47b740 1 -- 172.21.15.146:0/785348327 >> 172.21.15.146:0/785348327 conn(0x560991bebb80 msgr2=0x560991bebf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.773+0000 7fe462a00640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe4480a3d20 0x7fe4480a61e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x560991bf0e20 tx=0x7fe454005d80 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.773+0000 7fe46c47b740 1 -- 172.21.15.146:0/785348327 shutdown_connections 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.773+0000 7fe46c47b740 1 -- 172.21.15.146:0/785348327 wait complete. 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.773+0000 7fe46c47b740 1 -- start start 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.773+0000 7fe46c47b740 1 Processor -- start 2024-04-24T08:30:08.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bef860 0x560991bf7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bf0180 0x560991bf7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bfa850 0x560991bf8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe46c47b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560991bfd640 con 0x560991bef860 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe46c47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560991b0e690 con 0x560991bfa850 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe46c47b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560991bfd7c0 con 0x560991bf0180 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe46c47b740 10 client.0 _ll_register_callbacks cb 0x560991b4a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bef860 0x560991bf7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bef860 0x560991bf7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56234/0 (socket says 172.21.15.146:56234) 2024-04-24T08:30:08.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 -- 172.21.15.146:0/4119351934 learned_addr learned my addr 172.21.15.146:0/4119351934 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462000640 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bf0180 0x560991bf7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe463400640 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bfa850 0x560991bf8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 -- 172.21.15.146:0/4119351934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bf0180 msgr2=0x560991bf7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bf0180 0x560991bf7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 -- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bfa850 msgr2=0x560991bf8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bfa850 0x560991bf8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.774+0000 7fe462a00640 1 -- 172.21.15.146:0/4119351934 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560991b0d920 con 0x560991bef860 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.775+0000 7fe462a00640 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bef860 0x560991bf7810 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe454000f80 tx=0x7fe45402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588f8c288e389ea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:08.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.775+0000 7fe447e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.775+0000 7fe447e00640 1 -- 172.21.15.146:0/4119351934 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe454004250 con 0x560991bef860 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.775+0000 7fe447e00640 1 -- 172.21.15.146:0/4119351934 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7fe4540043f0 con 0x560991bef860 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.775+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe45c005500 con 0x560991bef860 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.775+0000 7fe447e00640 1 -- 172.21.15.146:0/4119351934 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe4540045f0 con 0x560991bef860 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.775+0000 7fe46c47b740 10 client.16977 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5609919317d0 con 0x560991bef860 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe447e00640 1 -- 172.21.15.146:0/4119351934 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe45c005500 con 0x560991bef860 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe447e00640 1 -- 172.21.15.146:0/4119351934 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe45405a030 con 0x560991bef860 2024-04-24T08:30:08.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe46c47b740 10 client.16977 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:08.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe46c47b740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:08.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe46c47b740 20 client.16977 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:08.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe46c47b740 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c0aae0 0x560991c0cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe46c47b740 4 client.16977 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:08.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560991b62c10 con 0x560991c0aae0 2024-04-24T08:30:08.786 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.776+0000 7fe462000640 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c0aae0 0x560991c0cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.787 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.777+0000 7fe462000640 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c0aae0 0x560991c0cf30 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88c7d05d496429ba server_cookie=70f2b101d6bcecae in_seq=0 out_seq=0 2024-04-24T08:30:08.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.777+0000 7fe447e00640 10 client.16977 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:08.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe447e00640 1 -- 172.21.15.146:0/4119351934 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560991b62c10 con 0x560991c0aae0 2024-04-24T08:30:08.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe447e00640 10 client.16977 handle_command_reply: tid=0 2024-04-24T08:30:08.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 client.16977 shutdown 2024-04-24T08:30:08.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bef860 msgr2=0x560991bf7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.788 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bef860 0x560991bf7810 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe454000f80 tx=0x7fe45402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 shutdown_connections 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c0aae0 0x560991c0cf30 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bf0180 0x560991bf7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bfa850 0x560991bf8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 --2- 172.21.15.146:0/4119351934 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bef860 0x560991bf7810 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 >> 172.21.15.146:0/4119351934 conn(0x560991bebb80 msgr2=0x560991bf1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe447e00640 0 client.16977 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:08.789 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 shutdown_connections 2024-04-24T08:30:08.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 1 -- 172.21.15.146:0/4119351934 wait complete. 2024-04-24T08:30:08.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.778+0000 7fe46c47b740 20 client.16977 trim_cache size 0 max 16384 2024-04-24T08:30:08.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 Processor -- start 2024-04-24T08:30:08.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 -- start start 2024-04-24T08:30:08.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991c06500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bf6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bf1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641b2870 con 0x560991bfa850 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641b0140 con 0x560991bf0180 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.797+0000 7fe46c47b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641adfb0 con 0x560991bef860 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bf1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bf1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56250/0 (socket says 172.21.15.146:56250) 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 -- 172.21.15.146:0/553061616 learned_addr learned my addr 172.21.15.146:0/553061616 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe462000640 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bf6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe462a00640 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991c06500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 -- 172.21.15.146:0/553061616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 msgr2=0x560991c06500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991c06500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 -- 172.21.15.146:0/553061616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 msgr2=0x560991bf6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bf6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe463400640 1 -- 172.21.15.146:0/553061616 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609918673f0 con 0x560991bfa850 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe462000640 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bf6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.798+0000 7fe462a00640 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991c06500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.799+0000 7fe463400640 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bf1fc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe45c004550 tx=0x7fe45c009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64a648622f55f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:08.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.799+0000 7fe452a00640 1 -- 172.21.15.146:0/553061616 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c004050 con 0x560991bfa850 2024-04-24T08:30:08.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.799+0000 7fe452a00640 1 -- 172.21.15.146:0/553061616 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe45c028070 con 0x560991bfa850 2024-04-24T08:30:08.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.799+0000 7fe452a00640 1 -- 172.21.15.146:0/553061616 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c02d6d0 con 0x560991bfa850 2024-04-24T08:30:08.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.799+0000 7fe46c47b740 1 -- 172.21.15.146:0/553061616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 msgr2=0x560991bf1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.799+0000 7fe46c47b740 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bf1fc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe45c004550 tx=0x7fe45c009910 comp rx=0 tx=0).stop 2024-04-24T08:30:08.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 -- 172.21.15.146:0/553061616 shutdown_connections 2024-04-24T08:30:08.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bfa850 0x560991bf1fc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bf0180 0x560991bf6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 --2- 172.21.15.146:0/553061616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bef860 0x560991c06500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 -- 172.21.15.146:0/553061616 >> 172.21.15.146:0/553061616 conn(0x560991bebb80 msgr2=0x560991bea390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 -- 172.21.15.146:0/553061616 shutdown_connections 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 -- 172.21.15.146:0/553061616 wait complete. 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 -- start start 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.800+0000 7fe46c47b740 1 Processor -- start 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bef860 0x560991c02670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bf0180 0x560991c02bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe46c47b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bfa850 0x560991bfdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe46c47b740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641ae600 con 0x560991bf0180 2024-04-24T08:30:08.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe46c47b740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641afa70 con 0x560991bef860 2024-04-24T08:30:08.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe46c47b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4641aecd0 con 0x560991bfa850 2024-04-24T08:30:08.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe46c47b740 10 client.0 _ll_register_callbacks cb 0x7fe454009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:08.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bf0180 0x560991c02bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bef860 0x560991c02670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bf0180 0x560991c02bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56262/0 (socket says 172.21.15.146:56262) 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 -- 172.21.15.146:0/86275376 learned_addr learned my addr 172.21.15.146:0/86275376 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe463400640 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bfa850 0x560991bfdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 -- 172.21.15.146:0/86275376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bfa850 msgr2=0x560991bfdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bfa850 0x560991bfdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 -- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bef860 msgr2=0x560991c02670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bef860 0x560991c02670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462000640 1 -- 172.21.15.146:0/86275376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609918270f0 con 0x560991bf0180 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.801+0000 7fe462a00640 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bef860 0x560991c02670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe462000640 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bf0180 0x560991c02bb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe44c09cb10 tx=0x7fe44c0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf10de92fcc8f461 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe447e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe447e00640 1 -- 172.21.15.146:0/86275376 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0a0e80 con 0x560991bf0180 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe447e00640 1 -- 172.21.15.146:0/86275376 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7fe44c09ae80 con 0x560991bf0180 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe45c028070 con 0x560991bf0180 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe447e00640 1 -- 172.21.15.146:0/86275376 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe44c0a5740 con 0x560991bf0180 2024-04-24T08:30:08.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe46c47b740 10 client.16983 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560991b1a810 con 0x560991bf0180 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe447e00640 1 -- 172.21.15.146:0/86275376 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe45c028070 con 0x560991bf0180 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.802+0000 7fe447e00640 1 -- 172.21.15.146:0/86275376 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe44c0b8050 con 0x560991bf0180 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe46c47b740 10 client.16983 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe46c47b740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe46c47b740 20 client.16983 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe46c47b740 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c70e60 0x560991c73280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe46c47b740 4 client.16983 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560991b5ca80 con 0x560991c70e60 2024-04-24T08:30:08.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe462a00640 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c70e60 0x560991c73280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe447e00640 10 client.16983 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.803+0000 7fe462a00640 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c70e60 0x560991c73280 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afe4d6f1722677ec server_cookie=9c36ab0e21584979 in_seq=0 out_seq=0 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe447e00640 1 -- 172.21.15.146:0/86275376 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560991b5ca80 con 0x560991c70e60 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe447e00640 10 client.16983 handle_command_reply: tid=0 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 client.16983 shutdown 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bf0180 msgr2=0x560991c02bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bf0180 0x560991c02bb0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe44c09cb10 tx=0x7fe44c0bd040 comp rx=0 tx=0).stop 2024-04-24T08:30:08.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 shutdown_connections 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560991c70e60 0x560991c73280 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560991bfa850 0x560991bfdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560991bf0180 0x560991c02bb0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 --2- 172.21.15.146:0/86275376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560991bef860 0x560991c02670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe447e00640 0 client.16983 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 >> 172.21.15.146:0/86275376 conn(0x560991bebb80 msgr2=0x560991bea390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 shutdown_connections 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 1 -- 172.21.15.146:0/86275376 wait complete. 2024-04-24T08:30:08.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.804+0000 7fe46c47b740 20 client.16983 trim_cache size 0 max 16384 2024-04-24T08:30:08.815 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:30:08.815 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:30:08.815 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:30:08.815 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:30:08.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe4480a3d20 msgr2=0x7fe4480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe4480a3d20 0x7fe4480a61e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x560991bf0e20 tx=0x7fe454005d80 comp rx=0 tx=0).stop 2024-04-24T08:30:08.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe464109580 msgr2=0x7fe4641a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:08.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe464109580 0x7fe4641a0d10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e9c0 tx=0x7fe44c00ee80 comp rx=0 tx=0).stop 2024-04-24T08:30:08.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 shutdown_connections 2024-04-24T08:30:08.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fe4480a3d20 0x7fe4480a61e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe464109580 0x7fe4641a0d10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe46419a980 0x7fe46419ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 --2- 172.21.15.146:0/152284839 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4641044b0 0x7fe4641a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:08.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 >> 172.21.15.146:0/152284839 conn(0x7fe4640ffd90 msgr2=0x7fe4641020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:08.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 shutdown_connections 2024-04-24T08:30:08.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:08.805+0000 7fe469600640 1 -- 172.21.15.146:0/152284839 wait complete. 2024-04-24T08:30:08.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:08 smithi177 ceph-mon[33046]: pgmap v275: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:30:08.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:08 smithi177 ceph-mon[33046]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T08:30:08.822 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:30:08.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:30:09.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 -- 172.21.15.146:0/1084095124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c1084a0 msgr2=0x7f9d5c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1084095124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c1084a0 0x7f9d5c108900 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009860 tx=0x7f9d4802a3d0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 -- 172.21.15.146:0/1084095124 shutdown_connections 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1084095124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c108e40 0x7f9d5c1152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1084095124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c1084a0 0x7f9d5c108900 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1084095124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d5c10d3c0 0x7f9d5c10d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 -- 172.21.15.146:0/1084095124 >> 172.21.15.146:0/1084095124 conn(0x7f9d5c077b80 msgr2=0x7f9d5c079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 -- 172.21.15.146:0/1084095124 shutdown_connections 2024-04-24T08:30:09.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.012+0000 7f9d5be00640 1 -- 172.21.15.146:0/1084095124 wait complete. 2024-04-24T08:30:09.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 Processor -- start 2024-04-24T08:30:09.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 -- start start 2024-04-24T08:30:09.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c1084a0 0x7f9d5c1a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d5c108e40 0x7f9d5c1a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c10d3c0 0x7f9d5c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c1187e0 con 0x7f9d5c1084a0 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c118660 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.013+0000 7f9d5be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5c118960 con 0x7f9d5c108e40 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c10d3c0 0x7f9d5c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c10d3c0 0x7f9d5c19eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53484/0 (socket says 172.21.15.146:53484) 2024-04-24T08:30:09.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 -- 172.21.15.146:0/1464708174 learned_addr learned my addr 172.21.15.146:0/1464708174 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5aa00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c1084a0 0x7f9d5c1a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5a000640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d5c108e40 0x7f9d5c1a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 -- 172.21.15.146:0/1464708174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d5c108e40 msgr2=0x7f9d5c1a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d5c108e40 0x7f9d5c1a4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 -- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c1084a0 msgr2=0x7f9d5c1a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c1084a0 0x7f9d5c1a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 -- 172.21.15.146:0/1464708174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5c19f260 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.014+0000 7f9d5b400640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c10d3c0 0x7f9d5c19eae0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009fd0 tx=0x7f9d5400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.015+0000 7f9d53400640 1 -- 172.21.15.146:0/1464708174 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54004b20 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.015+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5c19f550 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.015+0000 7f9d53400640 1 -- 172.21.15.146:0/1464708174 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d54003e50 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.015+0000 7f9d53400640 1 -- 172.21.15.146:0/1464708174 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5400b630 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.015+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d5c1ab7f0 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.016+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d5c069190 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.018+0000 7f9d53400640 1 -- 172.21.15.146:0/1464708174 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f9d5401d020 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.019+0000 7f9d53400640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9d400a3d20 0x7f9d400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.020+0000 7f9d53400640 1 -- 172.21.15.146:0/1464708174 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f9d540d1230 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.020+0000 7f9d5aa00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9d400a3d20 0x7f9d400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.020+0000 7f9d5aa00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9d400a3d20 0x7f9d400a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0095f0 tx=0x7f9d4c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.022+0000 7f9d53400640 1 -- 172.21.15.146:0/1464708174 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d54024030 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.265+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d5c02d620 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.266+0000 7f9d53400640 1 -- 172.21.15.146:0/1464708174 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f9d5409d830 con 0x7f9d5c10d3c0 2024-04-24T08:30:09.268 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:30:09.268 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:30:09.268 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:30:09.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9d400a3d20 msgr2=0x7f9d400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9d400a3d20 0x7f9d400a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c0095f0 tx=0x7f9d4c005f90 comp rx=0 tx=0).stop 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c10d3c0 msgr2=0x7f9d5c19eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c10d3c0 0x7f9d5c19eae0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009fd0 tx=0x7f9d5400c800 comp rx=0 tx=0).stop 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 shutdown_connections 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f9d400a3d20 0x7f9d400a61e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d5c108e40 0x7f9d5c1a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d5c10d3c0 0x7f9d5c19eae0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 --2- 172.21.15.146:0/1464708174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d5c1084a0 0x7f9d5c1a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 >> 172.21.15.146:0/1464708174 conn(0x7f9d5c077b80 msgr2=0x7f9d5c078200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.269+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 shutdown_connections 2024-04-24T08:30:09.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.270+0000 7f9d5be00640 1 -- 172.21.15.146:0/1464708174 wait complete. 2024-04-24T08:30:09.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:30:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.497+0000 7f3a22200640 1 -- 172.21.15.146:0/3179204458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c104450 msgr2=0x7f3a1c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.497+0000 7f3a22200640 1 --2- 172.21.15.146:0/3179204458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c104450 0x7f3a1c104830 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009ca0 tx=0x7f3a0c02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.497+0000 7f3a22200640 1 -- 172.21.15.146:0/3179204458 shutdown_connections 2024-04-24T08:30:09.499 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.497+0000 7f3a22200640 1 --2- 172.21.15.146:0/3179204458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a1c104e00 0x7f3a1c108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.497+0000 7f3a22200640 1 --2- 172.21.15.146:0/3179204458 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1c109520 0x7f3a1c1110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.497+0000 7f3a22200640 1 --2- 172.21.15.146:0/3179204458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c104450 0x7f3a1c104830 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.497+0000 7f3a22200640 1 -- 172.21.15.146:0/3179204458 >> 172.21.15.146:0/3179204458 conn(0x7f3a1c0ffc70 msgr2=0x7f3a1c102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.498+0000 7f3a22200640 1 -- 172.21.15.146:0/3179204458 shutdown_connections 2024-04-24T08:30:09.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.498+0000 7f3a22200640 1 -- 172.21.15.146:0/3179204458 wait complete. 2024-04-24T08:30:09.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.498+0000 7f3a22200640 1 Processor -- start 2024-04-24T08:30:09.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a22200640 1 -- start start 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a22200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a1c104450 0x7f3a1c1a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a22200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1c109520 0x7f3a1c1a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a22200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c19a9a0 0x7f3a1c19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a22200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c114620 con 0x7f3a1c19a9a0 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a22200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1144a0 con 0x7f3a1c109520 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a22200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1147a0 con 0x7f3a1c104450 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a1c104450 0x7f3a1c1a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.503 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1c109520 0x7f3a1c1a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a1c104450 0x7f3a1c1a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44804/0 (socket says 172.21.15.146:44804) 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a1be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c19a9a0 0x7f3a1c19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.499+0000 7f3a1b400640 1 -- 172.21.15.146:0/2694926111 learned_addr learned my addr 172.21.15.146:0/2694926111 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a12a00640 1 -- 172.21.15.146:0/2694926111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a1c104450 msgr2=0x7f3a1c1a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a12a00640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a1c104450 0x7f3a1c1a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a12a00640 1 -- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c19a9a0 msgr2=0x7f3a1c19ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a12a00640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c19a9a0 0x7f3a1c19ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a12a00640 1 -- 172.21.15.146:0/2694926111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c19b4c0 con 0x7f3a1c109520 2024-04-24T08:30:09.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a12a00640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1c109520 0x7f3a1c1a0d30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400e9c0 tx=0x7f3a0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a18c00640 1 -- 172.21.15.146:0/2694926111 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0400ccc0 con 0x7f3a1c109520 2024-04-24T08:30:09.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c19b7b0 con 0x7f3a1c109520 2024-04-24T08:30:09.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a18c00640 1 -- 172.21.15.146:0/2694926111 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a04005020 con 0x7f3a1c109520 2024-04-24T08:30:09.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.500+0000 7f3a18c00640 1 -- 172.21.15.146:0/2694926111 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a04004a00 con 0x7f3a1c109520 2024-04-24T08:30:09.505 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.501+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c1a7a00 con 0x7f3a1c109520 2024-04-24T08:30:09.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 Processor -- start 2024-04-24T08:30:09.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 -- start start 2024-04-24T08:30:09.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a18c00640 1 -- 172.21.15.146:0/2694926111 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3a040051d0 con 0x7f3a1c109520 2024-04-24T08:30:09.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5613969271d0 con 0x5613969d5150 2024-04-24T08:30:09.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5613968cd450 con 0x5613969c2030 2024-04-24T08:30:09.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.504+0000 7f3a251d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561396923190 con 0x5613969c1680 2024-04-24T08:30:09.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969c1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44808/0 (socket says 172.21.15.146:44808) 2024-04-24T08:30:09.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 -- 172.21.15.146:0/4066810280 learned_addr learned my addr 172.21.15.146:0/4066810280 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 -- 172.21.15.146:0/4066810280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 msgr2=0x5613969d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 -- 172.21.15.146:0/4066810280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 msgr2=0x5613969d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 -- 172.21.15.146:0/4066810280 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56139698a310 con 0x5613969c1680 2024-04-24T08:30:09.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:09.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:09.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a1be00640 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969c1a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400b6e0 tx=0x7f3a14004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2d1ed41dd1ac3d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.505+0000 7f3a0be00640 1 -- 172.21.15.146:0/4066810280 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14003bd0 con 0x5613969c1680 2024-04-24T08:30:09.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a0be00640 1 -- 172.21.15.146:0/4066810280 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a14015030 con 0x5613969c1680 2024-04-24T08:30:09.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a0be00640 1 -- 172.21.15.146:0/4066810280 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a140106c0 con 0x5613969c1680 2024-04-24T08:30:09.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a18c00640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a000a3da0 0x7f3a000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a18c00640 1 -- 172.21.15.146:0/2694926111 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f3a040d0750 con 0x7f3a1c109520 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a1b400640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a000a3da0 0x7f3a000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a251d7740 1 -- 172.21.15.146:0/4066810280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 msgr2=0x5613969c1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969c1a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400b6e0 tx=0x7f3a14004610 comp rx=0 tx=0).stop 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a251d7740 1 -- 172.21.15.146:0/4066810280 shutdown_connections 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4066810280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969c1a60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.506+0000 7f3a251d7740 1 -- 172.21.15.146:0/4066810280 >> 172.21.15.146:0/4066810280 conn(0x5613969bdc70 msgr2=0x5613969be050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.507+0000 7f3a251d7740 1 -- 172.21.15.146:0/4066810280 shutdown_connections 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.507+0000 7f3a251d7740 1 -- 172.21.15.146:0/4066810280 wait complete. 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.507+0000 7f3a1b400640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a000a3da0 0x7f3a000a6260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c000f80 tx=0x7f3a0c005d80 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.507+0000 7f3a251d7740 1 -- start start 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.507+0000 7f3a251d7740 1 Processor -- start 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969d4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a251d7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5613969d7fb0 con 0x5613969d5150 2024-04-24T08:30:09.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a251d7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5613968e0810 con 0x5613969c2030 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a251d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5613969d8130 con 0x5613969c1680 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a251d7740 10 client.0 _ll_register_callbacks cb 0x5613968ae140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969d4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53528/0 (socket says 172.21.15.146:53528) 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a1be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 -- 172.21.15.146:0/4038623720 learned_addr learned my addr 172.21.15.146:0/4038623720 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 -- 172.21.15.146:0/4038623720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 msgr2=0x5613969d4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969d4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 -- 172.21.15.146:0/4038623720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 msgr2=0x5613969d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.508+0000 7f3a12a00640 1 -- 172.21.15.146:0/4038623720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613968f13f0 con 0x5613969c2030 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d1ed0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3a04004610 tx=0x7f3a0409bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e06f7f81ee8ec52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a08c00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a08c00640 1 -- 172.21.15.146:0/4038623720 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0401c070 con 0x5613969c2030 2024-04-24T08:30:09.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a08c00640 1 -- 172.21.15.146:0/4038623720 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f3a040a4e30 con 0x5613969c2030 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a14015030 con 0x5613969c2030 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a08c00640 1 -- 172.21.15.146:0/4038623720 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3a040ce280 con 0x5613969c2030 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a251d7740 10 client.27152 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5613966393f0 con 0x5613969c2030 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.509+0000 7f3a08c00640 1 -- 172.21.15.146:0/4038623720 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a040b3030 con 0x5613969c2030 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a08c00640 1 -- 172.21.15.146:0/4038623720 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3a04004ba0 con 0x5613969c2030 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a251d7740 10 client.27152 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:09.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a251d7740 10 client.27152 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:30:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a251d7740 20 client.27152 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5613969e52c0 0x5613969e7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a251d7740 4 client.27152 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561396934bf0 con 0x5613969e52c0 2024-04-24T08:30:09.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.510+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5613969e52c0 0x5613969e7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.511+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5613969e52c0 0x5613969e7730 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ba70bf761693c830 server_cookie=9744a9115c5f3a44 in_seq=0 out_seq=0 2024-04-24T08:30:09.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.511+0000 7f3a08c00640 10 client.27152 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.511+0000 7f3a08c00640 1 -- 172.21.15.146:0/4038623720 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561396934bf0 con 0x5613969e52c0 2024-04-24T08:30:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.511+0000 7f3a08c00640 10 client.27152 handle_command_reply: tid=0 2024-04-24T08:30:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.511+0000 7f3a251d7740 1 client.27152 shutdown 2024-04-24T08:30:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 msgr2=0x5613969d1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d1ed0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3a04004610 tx=0x7f3a0409bfb0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 shutdown_connections 2024-04-24T08:30:09.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x5613969e52c0 0x5613969e7730 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969d5150 0x5613969d25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969c2030 0x5613969d1ed0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4038623720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969d4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 >> 172.21.15.146:0/4038623720 conn(0x5613969bdc70 msgr2=0x5613969c3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a08c00640 0 client.27152 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 shutdown_connections 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 1 -- 172.21.15.146:0/4038623720 wait complete. 2024-04-24T08:30:09.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.512+0000 7f3a251d7740 20 client.27152 trim_cache size 0 max 16384 2024-04-24T08:30:09.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.526+0000 7f3a251d7740 1 Processor -- start 2024-04-24T08:30:09.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a251d7740 1 -- start start 2024-04-24T08:30:09.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969e0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c2030 0x5613969d4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ed000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a251d7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1ae630 con 0x5613969c2030 2024-04-24T08:30:09.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a251d7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1abf00 con 0x5613969d5150 2024-04-24T08:30:09.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a251d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1a9d70 con 0x5613969c1680 2024-04-24T08:30:09.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c2030 0x5613969d4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c2030 0x5613969d4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56308/0 (socket says 172.21.15.146:56308) 2024-04-24T08:30:09.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a12a00640 1 -- 172.21.15.146:0/2379067841 learned_addr learned my addr 172.21.15.146:0/2379067841 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a1b400640 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969e0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.527+0000 7f3a1be00640 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a12a00640 1 -- 172.21.15.146:0/2379067841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 msgr2=0x5613969e0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a12a00640 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969e0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a12a00640 1 -- 172.21.15.146:0/2379067841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 msgr2=0x5613969ed000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a12a00640 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ed000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a12a00640 1 -- 172.21.15.146:0/2379067841 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613967037d0 con 0x5613969c2030 2024-04-24T08:30:09.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a1be00640 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:09.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a12a00640 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c2030 0x5613969d4710 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3a04098fd0 tx=0x7f3a04004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0ef448c1546a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a0b400640 1 -- 172.21.15.146:0/2379067841 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a040a0a00 con 0x5613969c2030 2024-04-24T08:30:09.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a0b400640 1 -- 172.21.15.146:0/2379067841 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a040b3070 con 0x5613969c2030 2024-04-24T08:30:09.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.528+0000 7f3a0b400640 1 -- 172.21.15.146:0/2379067841 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a040a4e30 con 0x5613969c2030 2024-04-24T08:30:09.536 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 -- 172.21.15.146:0/2379067841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c2030 msgr2=0x5613969d4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c2030 0x5613969d4710 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3a04098fd0 tx=0x7f3a04004f90 comp rx=0 tx=0).stop 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 -- 172.21.15.146:0/2379067841 shutdown_connections 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c2030 0x5613969d4710 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 --2- 172.21.15.146:0/2379067841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c1680 0x5613969e0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 -- 172.21.15.146:0/2379067841 >> 172.21.15.146:0/2379067841 conn(0x5613969bdc70 msgr2=0x5613969c3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 -- 172.21.15.146:0/2379067841 shutdown_connections 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 -- 172.21.15.146:0/2379067841 wait complete. 2024-04-24T08:30:09.537 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.529+0000 7f3a251d7740 1 -- start start 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 1 Processor -- start 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c1680 0x5613969dd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c2030 0x5613969dd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1aa3c0 con 0x5613969c1680 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1ab830 con 0x5613969d5150 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c1aaa90 con 0x5613969c2030 2024-04-24T08:30:09.538 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a251d7740 10 client.0 _ll_register_callbacks cb 0x7f3a040046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c2030 0x5613969dd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a12a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c2030 0x5613969dd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44848/0 (socket says 172.21.15.146:44848) 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a12a00640 1 -- 172.21.15.146:0/4271377511 learned_addr learned my addr 172.21.15.146:0/4271377511 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c1680 0x5613969dd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.530+0000 7f3a1be00640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a12a00640 1 -- 172.21.15.146:0/4271377511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 msgr2=0x5613969ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.539 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a12a00640 1 -- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c1680 msgr2=0x5613969dd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c1680 0x5613969dd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a12a00640 1 -- 172.21.15.146:0/4271377511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613969ce930 con 0x5613969c2030 2024-04-24T08:30:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c1680 0x5613969dd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a12a00640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c2030 0x5613969dd900 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3a04099020 tx=0x7f3a0409ab60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1eb6fb4c83e756 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a08c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:30:09.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a08c00640 1 -- 172.21.15.146:0/4271377511 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a040a52c0 con 0x5613969c2030 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a08c00640 1 -- 172.21.15.146:0/4271377511 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f3a040ce080 con 0x5613969c2030 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a040b3070 con 0x5613969c2030 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a08c00640 1 -- 172.21.15.146:0/4271377511 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3a040a0a00 con 0x5613969c2030 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a251d7740 10 client.27157 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.531+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5613969cfc30 con 0x5613969c2030 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a08c00640 1 -- 172.21.15.146:0/4271377511 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a040b3070 con 0x5613969c2030 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a08c00640 1 -- 172.21.15.146:0/4271377511 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3a04081030 con 0x5613969c2030 2024-04-24T08:30:09.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a251d7740 10 client.27157 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a251d7740 10 client.27157 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:30:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a251d7740 20 client.27157 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561396a4c880 0x561396a4ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a251d7740 4 client.27157 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:30:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56139692ea60 con 0x561396a4c880 2024-04-24T08:30:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.532+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561396a4c880 0x561396a4ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a1b400640 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561396a4c880 0x561396a4ec70 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6d58e8621ac8d77 server_cookie=cb1cfe058c619eec in_seq=0 out_seq=0 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a08c00640 10 client.27157 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a08c00640 1 -- 172.21.15.146:0/4271377511 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56139692ea60 con 0x561396a4c880 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a08c00640 10 client.27157 handle_command_reply: tid=0 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 client.27157 shutdown 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c2030 msgr2=0x5613969dd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c2030 0x5613969dd900 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3a04099020 tx=0x7f3a0409ab60 comp rx=0 tx=0).stop 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 shutdown_connections 2024-04-24T08:30:09.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561396a4c880 0x561396a4ec70 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613969c2030 0x5613969dd900 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5613969d5150 0x5613969ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a08c00640 0 client.27157 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 --2- 172.21.15.146:0/4271377511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5613969c1680 0x5613969dd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 >> 172.21.15.146:0/4271377511 conn(0x5613969bdc70 msgr2=0x5613969c3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.533+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 shutdown_connections 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a251d7740 1 -- 172.21.15.146:0/4271377511 wait complete. 2024-04-24T08:30:09.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a251d7740 20 client.27157 trim_cache size 0 max 16384 2024-04-24T08:30:09.545 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:30:09.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a000a3da0 msgr2=0x7f3a000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a000a3da0 0x7f3a000a6260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c000f80 tx=0x7f3a0c005d80 comp rx=0 tx=0).stop 2024-04-24T08:30:09.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1c109520 msgr2=0x7f3a1c1a0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1c109520 0x7f3a1c1a0d30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400e9c0 tx=0x7f3a0400ee80 comp rx=0 tx=0).stop 2024-04-24T08:30:09.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 shutdown_connections 2024-04-24T08:30:09.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3a000a3da0 0x7f3a000a6260 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a1c19a9a0 0x7f3a1c19ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a1c109520 0x7f3a1c1a0d30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 --2- 172.21.15.146:0/2694926111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a1c104450 0x7f3a1c1a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 >> 172.21.15.146:0/2694926111 conn(0x7f3a1c0ffc70 msgr2=0x7f3a1c100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.534+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 shutdown_connections 2024-04-24T08:30:09.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.535+0000 7f3a22200640 1 -- 172.21.15.146:0/2694926111 wait complete. 2024-04-24T08:30:09.550 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:09 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1464708174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:30:09.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:30:09.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:09 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1464708174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:30:09.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 -- 172.21.15.146:0/2697049752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b400743a0 msgr2=0x7f0b40074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 --2- 172.21.15.146:0/2697049752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b400743a0 0x7f0b40074780 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009900 tx=0x7f0b3002a480 comp rx=0 tx=0).stop 2024-04-24T08:30:09.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 -- 172.21.15.146:0/2697049752 shutdown_connections 2024-04-24T08:30:09.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 --2- 172.21.15.146:0/2697049752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b40074cc0 0x7f0b40112970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 --2- 172.21.15.146:0/2697049752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b40112eb0 0x7f0b401152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 --2- 172.21.15.146:0/2697049752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b400743a0 0x7f0b40074780 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 -- 172.21.15.146:0/2697049752 >> 172.21.15.146:0/2697049752 conn(0x7f0b400fd9d0 msgr2=0x7f0b400ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 -- 172.21.15.146:0/2697049752 shutdown_connections 2024-04-24T08:30:09.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.759+0000 7f0b3f200640 1 -- 172.21.15.146:0/2697049752 wait complete. 2024-04-24T08:30:09.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.760+0000 7f0b3f200640 1 Processor -- start 2024-04-24T08:30:09.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.760+0000 7f0b3f200640 1 -- start start 2024-04-24T08:30:09.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b400743a0 0x7f0b4019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b40074cc0 0x7f0b4019f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b40112eb0 0x7f0b401a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401188f0 con 0x7f0b40112eb0 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40118770 con 0x7f0b400743a0 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40118a70 con 0x7f0b40074cc0 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b40112eb0 0x7f0b401a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b40074cc0 0x7f0b4019f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b400743a0 0x7f0b4019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b40074cc0 0x7f0b4019f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44866/0 (socket says 172.21.15.146:44866) 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.761+0000 7f0b3d400640 1 -- 172.21.15.146:0/3273486465 learned_addr learned my addr 172.21.15.146:0/3273486465 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3de00640 1 -- 172.21.15.146:0/3273486465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b40074cc0 msgr2=0x7f0b4019f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3de00640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b40074cc0 0x7f0b4019f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3de00640 1 -- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b40112eb0 msgr2=0x7f0b401a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3de00640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b40112eb0 0x7f0b401a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3de00640 1 -- 172.21.15.146:0/3273486465 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b401a3dc0 con 0x7f0b400743a0 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3e800640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b40112eb0 0x7f0b401a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3de00640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b400743a0 0x7f0b4019edd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0b3002a9a0 tx=0x7f0b30003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b36a00640 1 -- 172.21.15.146:0/3273486465 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3003f070 con 0x7f0b400743a0 2024-04-24T08:30:09.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b401a4050 con 0x7f0b400743a0 2024-04-24T08:30:09.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b36a00640 1 -- 172.21.15.146:0/3273486465 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b30031c90 con 0x7f0b400743a0 2024-04-24T08:30:09.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b36a00640 1 -- 172.21.15.146:0/3273486465 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30035030 con 0x7f0b400743a0 2024-04-24T08:30:09.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.762+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b401ab940 con 0x7f0b400743a0 2024-04-24T08:30:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.765+0000 7f0b47771740 1 Processor -- start 2024-04-24T08:30:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b47771740 1 -- start start 2024-04-24T08:30:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b36a00640 1 -- 172.21.15.146:0/3273486465 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f0b30042070 con 0x7f0b400743a0 2024-04-24T08:30:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b47771740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27db40 0x5620bb27df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b47771740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb2885f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.770 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b47771740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb288b30 0x5620bb28af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b47771740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5620bb1e31d0 con 0x5620bb27db40 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b47771740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5620bb189450 con 0x5620bb27e460 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b47771740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5620bb1df190 con 0x5620bb288b30 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27db40 0x5620bb27df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27db40 0x5620bb27df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56332/0 (socket says 172.21.15.146:56332) 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b3e800640 1 -- 172.21.15.146:0/3736411239 learned_addr learned my addr 172.21.15.146:0/3736411239 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b3de00640 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb2885f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.766+0000 7f0b3d400640 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb288b30 0x5620bb28af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b3e800640 1 -- 172.21.15.146:0/3736411239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb288b30 msgr2=0x5620bb28af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b3e800640 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb288b30 0x5620bb28af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b3e800640 1 -- 172.21.15.146:0/3736411239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 msgr2=0x5620bb2885f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b3e800640 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb2885f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b3e800640 1 -- 172.21.15.146:0/3736411239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620bb246310 con 0x5620bb27db40 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b3d400640 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb288b30 0x5620bb28af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b3e800640 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27db40 0x5620bb27df20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009f90 tx=0x7f0b380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23fc855a9f773d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b2f000640 1 -- 172.21.15.146:0/3736411239 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3801e070 con 0x5620bb27db40 2024-04-24T08:30:09.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b2f000640 1 -- 172.21.15.146:0/3736411239 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b38004310 con 0x5620bb27db40 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b2f000640 1 -- 172.21.15.146:0/3736411239 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3800f9a0 con 0x5620bb27db40 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.767+0000 7f0b36a00640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0b200a3da0 0x7f0b200a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b36a00640 1 -- 172.21.15.146:0/3273486465 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f0b300f10e0 con 0x7f0b400743a0 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b3de00640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0b200a3da0 0x7f0b200a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 -- 172.21.15.146:0/3736411239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27db40 msgr2=0x5620bb27df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27db40 0x5620bb27df20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009f90 tx=0x7f0b380118d0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 -- 172.21.15.146:0/3736411239 shutdown_connections 2024-04-24T08:30:09.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb288b30 0x5620bb28af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb2885f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 --2- 172.21.15.146:0/3736411239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27db40 0x5620bb27df20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 -- 172.21.15.146:0/3736411239 >> 172.21.15.146:0/3736411239 conn(0x5620bb279e60 msgr2=0x5620bb27a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b3de00640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0b200a3da0 0x7f0b200a6260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5620bb27f100 tx=0x7f0b300f4b90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 -- 172.21.15.146:0/3736411239 shutdown_connections 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.768+0000 7f0b47771740 1 -- 172.21.15.146:0/3736411239 wait complete. 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.769+0000 7f0b47771740 1 -- start start 2024-04-24T08:30:09.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.769+0000 7f0b47771740 1 Processor -- start 2024-04-24T08:30:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.769+0000 7f0b47771740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb2885f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.769+0000 7f0b47771740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b47771740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb286020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b47771740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5620bb28b9c0 con 0x5620bb27e460 2024-04-24T08:30:09.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b47771740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5620bb19c810 con 0x5620bb288b30 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b47771740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5620bb28bb40 con 0x5620bb27db40 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b47771740 10 client.0 _ll_register_callbacks cb 0x5620bb16a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb2885f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2858e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56344/0 (socket says 172.21.15.146:56344) 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 -- 172.21.15.146:0/2164777655 learned_addr learned my addr 172.21.15.146:0/2164777655 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3e800640 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb286020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 -- 172.21.15.146:0/2164777655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 msgr2=0x5620bb2885f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb2885f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 -- 172.21.15.146:0/2164777655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 msgr2=0x5620bb286020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb286020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3d400640 1 -- 172.21.15.146:0/2164777655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620bb1ad3f0 con 0x5620bb27e460 2024-04-24T08:30:09.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.770+0000 7f0b3e800640 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb2885f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:09.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b3d400640 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2858e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0b28011cf0 tx=0x7f0b2800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7713231736aa02a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b27400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:09.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b27400640 1 -- 172.21.15.146:0/2164777655 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2801f060 con 0x5620bb27e460 2024-04-24T08:30:09.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b27400640 1 -- 172.21.15.146:0/2164777655 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f0b2801a3f0 con 0x5620bb27e460 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b38004310 con 0x5620bb27e460 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b27400640 1 -- 172.21.15.146:0/2164777655 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0b2801a5f0 con 0x5620bb27e460 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b47771740 10 client.17019 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5620baef53f0 con 0x5620bb27e460 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.771+0000 7f0b27400640 1 -- 172.21.15.146:0/2164777655 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b38004310 con 0x5620bb27e460 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b27400640 1 -- 172.21.15.146:0/2164777655 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0b28019030 con 0x5620bb27e460 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b47771740 10 client.17019 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:09.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b47771740 10 client.17019 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:30:09.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b47771740 20 client.17019 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:09.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b47771740 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb298e70 0x5620bb29b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b47771740 4 client.17019 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:09.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620bb1f0bf0 con 0x5620bb298e70 2024-04-24T08:30:09.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.772+0000 7f0b3e800640 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb298e70 0x5620bb29b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.773+0000 7f0b3e800640 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb298e70 0x5620bb29b290 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=16a3df7c66bd325 server_cookie=9e1a31881aed98ad in_seq=0 out_seq=0 2024-04-24T08:30:09.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.773+0000 7f0b27400640 10 client.17019 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.773+0000 7f0b27400640 1 -- 172.21.15.146:0/2164777655 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620bb1f0bf0 con 0x5620bb298e70 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.773+0000 7f0b27400640 10 client.17019 handle_command_reply: tid=0 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.773+0000 7f0b47771740 1 client.17019 shutdown 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 msgr2=0x5620bb2858e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2858e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0b28011cf0 tx=0x7f0b2800f790 comp rx=0 tx=0).stop 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 shutdown_connections 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb298e70 0x5620bb29b290 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb286020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2858e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 --2- 172.21.15.146:0/2164777655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb2885f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b27400640 0 client.17019 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:09.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 >> 172.21.15.146:0/2164777655 conn(0x5620bb279e60 msgr2=0x5620bb27fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 shutdown_connections 2024-04-24T08:30:09.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 1 -- 172.21.15.146:0/2164777655 wait complete. 2024-04-24T08:30:09.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.774+0000 7f0b47771740 20 client.17019 trim_cache size 0 max 16384 2024-04-24T08:30:09.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.788+0000 7f0b47771740 1 Processor -- start 2024-04-24T08:30:09.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.788+0000 7f0b47771740 1 -- start start 2024-04-24T08:30:09.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b47771740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb286000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b47771740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b47771740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb288120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b47771740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b6b50 con 0x5620bb27e460 2024-04-24T08:30:09.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b47771740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b4420 con 0x5620bb288b30 2024-04-24T08:30:09.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b47771740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b2290 con 0x5620bb27db40 2024-04-24T08:30:09.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2948e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56354/0 (socket says 172.21.15.146:56354) 2024-04-24T08:30:09.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 -- 172.21.15.146:0/2523776343 learned_addr learned my addr 172.21.15.146:0/2523776343 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3e800640 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb286000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3e800640 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb288120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 -- 172.21.15.146:0/2523776343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 msgr2=0x5620bb286000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb286000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 -- 172.21.15.146:0/2523776343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 msgr2=0x5620bb288120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb288120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3d400640 1 -- 172.21.15.146:0/2523776343 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620bafbf7d0 con 0x5620bb27e460 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.789+0000 7f0b3e800640 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb288120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.790+0000 7f0b3d400640 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2948e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0b28002230 tx=0x7f0b280099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c777f24e09c1524 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.790+0000 7f0b2e600640 1 -- 172.21.15.146:0/2523776343 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800fc90 con 0x5620bb27e460 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.790+0000 7f0b2e600640 1 -- 172.21.15.146:0/2523776343 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b28030070 con 0x5620bb27e460 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.790+0000 7f0b2e600640 1 -- 172.21.15.146:0/2523776343 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2801ab10 con 0x5620bb27e460 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 -- 172.21.15.146:0/2523776343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 msgr2=0x5620bb2948e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2948e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0b28002230 tx=0x7f0b280099b0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 -- 172.21.15.146:0/2523776343 shutdown_connections 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb288b30 0x5620bb288120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb27e460 0x5620bb2948e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 --2- 172.21.15.146:0/2523776343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb286000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 -- 172.21.15.146:0/2523776343 >> 172.21.15.146:0/2523776343 conn(0x5620bb279e60 msgr2=0x5620bb2841b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 -- 172.21.15.146:0/2523776343 shutdown_connections 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 -- 172.21.15.146:0/2523776343 wait complete. 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 -- start start 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.791+0000 7f0b47771740 1 Processor -- start 2024-04-24T08:30:09.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b47771740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb289320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b47771740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb289860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b47771740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb288b30 0x5620bb28fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b47771740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b28e0 con 0x5620bb288b30 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b47771740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b3d50 con 0x5620bb27e460 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b47771740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401b2fb0 con 0x5620bb27db40 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b47771740 10 client.0 _ll_register_callbacks cb 0x7f0b28009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb289860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb289320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb289860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53620/0 (socket says 172.21.15.146:53620) 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 -- 172.21.15.146:0/3245879528 learned_addr learned my addr 172.21.15.146:0/3245879528 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3e800640 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb288b30 0x5620bb28fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 -- 172.21.15.146:0/3245879528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 msgr2=0x5620bb289320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb289320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 -- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb288b30 msgr2=0x5620bb28fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb288b30 0x5620bb28fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.792+0000 7f0b3d400640 1 -- 172.21.15.146:0/3245879528 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620bb290330 con 0x5620bb27e460 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b3d400640 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb289860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0b28011c90 tx=0x7f0b28009320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12cba3ee23c081ba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:09.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b27400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:09.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b27400640 1 -- 172.21.15.146:0/3245879528 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28038020 con 0x5620bb27e460 2024-04-24T08:30:09.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b28030070 con 0x5620bb27e460 2024-04-24T08:30:09.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b27400640 1 -- 172.21.15.146:0/3245879528 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f0b2801a3f0 con 0x5620bb27e460 2024-04-24T08:30:09.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b27400640 1 -- 172.21.15.146:0/3245879528 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0b2801f020 con 0x5620bb27e460 2024-04-24T08:30:09.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b47771740 10 client.27173 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:09.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.793+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5620bb291510 con 0x5620bb27e460 2024-04-24T08:30:09.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b27400640 1 -- 172.21.15.146:0/3245879528 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b28030070 con 0x5620bb27e460 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b27400640 1 -- 172.21.15.146:0/3245879528 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0b28019040 con 0x5620bb27e460 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b47771740 10 client.27173 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b47771740 10 client.27173 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b47771740 20 client.27173 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b47771740 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb2ff6b0 0x5620bb301aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b47771740 4 client.27173 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5620bb1eaa60 con 0x5620bb2ff6b0 2024-04-24T08:30:09.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.794+0000 7f0b3e800640 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb2ff6b0 0x5620bb301aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b3e800640 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb2ff6b0 0x5620bb301aa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72ba5653840b6c90 server_cookie=893c7e61a0fda in_seq=0 out_seq=0 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b27400640 10 client.27173 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b27400640 1 -- 172.21.15.146:0/3245879528 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5620bb1eaa60 con 0x5620bb2ff6b0 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b27400640 10 client.27173 handle_command_reply: tid=0 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 client.27173 shutdown 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 msgr2=0x5620bb289860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb289860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0b28011c90 tx=0x7f0b28009320 comp rx=0 tx=0).stop 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 shutdown_connections 2024-04-24T08:30:09.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5620bb2ff6b0 0x5620bb301aa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5620bb288b30 0x5620bb28fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5620bb27e460 0x5620bb289860 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 --2- 172.21.15.146:0/3245879528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620bb27db40 0x5620bb289320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b27400640 0 client.27173 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 >> 172.21.15.146:0/3245879528 conn(0x5620bb279e60 msgr2=0x5620bb27fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 shutdown_connections 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 1 -- 172.21.15.146:0/3245879528 wait complete. 2024-04-24T08:30:09.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.795+0000 7f0b47771740 20 client.27173 trim_cache size 0 max 16384 2024-04-24T08:30:09.805 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:30:09.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0b200a3da0 msgr2=0x7f0b200a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0b200a3da0 0x7f0b200a6260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5620bb27f100 tx=0x7f0b300f4b90 comp rx=0 tx=0).stop 2024-04-24T08:30:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b400743a0 msgr2=0x7f0b4019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b400743a0 0x7f0b4019edd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0b3002a9a0 tx=0x7f0b30003b20 comp rx=0 tx=0).stop 2024-04-24T08:30:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 shutdown_connections 2024-04-24T08:30:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0b200a3da0 0x7f0b200a6260 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b40074cc0 0x7f0b4019f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b40112eb0 0x7f0b401a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 --2- 172.21.15.146:0/3273486465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0b400743a0 0x7f0b4019edd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 >> 172.21.15.146:0/3273486465 conn(0x7f0b400fd9d0 msgr2=0x7f0b400fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 shutdown_connections 2024-04-24T08:30:09.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:09.796+0000 7f0b3f200640 1 -- 172.21.15.146:0/3273486465 wait complete. 2024-04-24T08:30:09.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:30:09.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:09 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1464708174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:30:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.017+0000 7f4245400640 1 -- 172.21.15.146:0/1013325395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4240108610 msgr2=0x7f4240108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.017+0000 7f4245400640 1 --2- 172.21.15.146:0/1013325395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4240108610 0x7f4240108a90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4228009930 tx=0x7f422802a440 comp rx=0 tx=0).stop 2024-04-24T08:30:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.017+0000 7f4245400640 1 -- 172.21.15.146:0/1013325395 shutdown_connections 2024-04-24T08:30:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.017+0000 7f4245400640 1 --2- 172.21.15.146:0/1013325395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4240108fd0 0x7f4240115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.017+0000 7f4245400640 1 --2- 172.21.15.146:0/1013325395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4240108610 0x7f4240108a90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.017+0000 7f4245400640 1 --2- 172.21.15.146:0/1013325395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f424010e660 0x7f424010ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.018+0000 7f4245400640 1 -- 172.21.15.146:0/1013325395 >> 172.21.15.146:0/1013325395 conn(0x7f4240076540 msgr2=0x7f4240078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.018+0000 7f4245400640 1 -- 172.21.15.146:0/1013325395 shutdown_connections 2024-04-24T08:30:10.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.018+0000 7f4245400640 1 -- 172.21.15.146:0/1013325395 wait complete. 2024-04-24T08:30:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 Processor -- start 2024-04-24T08:30:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 -- start start 2024-04-24T08:30:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240108610 0x7f424019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4240108fd0 0x7f424019f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424010e660 0x7f42401a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42401189a0 con 0x7f424010e660 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4240118820 con 0x7f4240108610 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.019+0000 7f4245400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4240118b20 con 0x7f4240108fd0 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424010e660 0x7f42401a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f4235e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4240108fd0 0x7f424019f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240108610 0x7f424019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f4235e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4240108fd0 0x7f424019f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44898/0 (socket says 172.21.15.146:44898) 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240108610 0x7f424019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53638/0 (socket says 172.21.15.146:53638) 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424010e660 0x7f42401a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56390/0 (socket says 172.21.15.146:56390) 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f4235e00640 1 -- 172.21.15.146:0/666940454 learned_addr learned my addr 172.21.15.146:0/666940454 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 -- 172.21.15.146:0/666940454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4240108fd0 msgr2=0x7f424019f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4240108fd0 0x7f424019f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 -- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424010e660 msgr2=0x7f42401a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424010e660 0x7f42401a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 -- 172.21.15.146:0/666940454 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42401a3e90 con 0x7f4240108610 2024-04-24T08:30:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.020+0000 7f423e800640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240108610 0x7f424019eea0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f423000e9c0 tx=0x7f423000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.021+0000 7f4237e00640 1 -- 172.21.15.146:0/666940454 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423000ccc0 con 0x7f4240108610 2024-04-24T08:30:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.021+0000 7f4237e00640 1 -- 172.21.15.146:0/666940454 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4230010910 con 0x7f4240108610 2024-04-24T08:30:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.021+0000 7f4237e00640 1 -- 172.21.15.146:0/666940454 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230018920 con 0x7f4240108610 2024-04-24T08:30:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.021+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42401a4180 con 0x7f4240108610 2024-04-24T08:30:10.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.021+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42401aba10 con 0x7f4240108610 2024-04-24T08:30:10.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 Processor -- start 2024-04-24T08:30:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 -- start start 2024-04-24T08:30:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4237e00640 1 -- 172.21.15.146:0/666940454 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f423001f020 con 0x7f4240108610 2024-04-24T08:30:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c9807d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9808270 0x55f0c9812400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c9814d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f0c976d1d0 con 0x55f0c9812940 2024-04-24T08:30:10.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f0c9713450 con 0x55f0c9808270 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.024+0000 7f4248174740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0c9769190 con 0x55f0c9807950 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9808270 0x55f0c9812400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f4235e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c9814d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c9807d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9808270 0x55f0c9812400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53650/0 (socket says 172.21.15.146:53650) 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f4235e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c9814d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56404/0 (socket says 172.21.15.146:56404) 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 -- 172.21.15.146:0/50174025 learned_addr learned my addr 172.21.15.146:0/50174025 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:10.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c9807d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:44910/0 (socket says 172.21.15.146:44910) 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 -- 172.21.15.146:0/50174025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 msgr2=0x55f0c9807d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c9807d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 -- 172.21.15.146:0/50174025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 msgr2=0x55f0c9814d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c9814d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 -- 172.21.15.146:0/50174025 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0c97d0310 con 0x55f0c9808270 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f4235e00640 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c9814d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.025+0000 7f423e800640 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9808270 0x55f0c9812400 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f42300cddb0 tx=0x7f42300d6cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b8c5edc6b1ef135 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:10.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.026+0000 7f4237e00640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42240a3da0 0x7f42240a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.026+0000 7f422f400640 1 -- 172.21.15.146:0/50174025 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42300df070 con 0x55f0c9808270 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.026+0000 7f422f400640 1 -- 172.21.15.146:0/50174025 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42300be560 con 0x55f0c9808270 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.026+0000 7f422f400640 1 -- 172.21.15.146:0/50174025 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230065600 con 0x55f0c9808270 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.026+0000 7f4237e00640 1 -- 172.21.15.146:0/666940454 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f42300d07c0 con 0x7f4240108610 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.026+0000 7f423f200640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42240a3da0 0x7f42240a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.026+0000 7f4248174740 1 -- 172.21.15.146:0/50174025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9808270 msgr2=0x55f0c9812400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9808270 0x55f0c9812400 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f42300cddb0 tx=0x7f42300d6cb0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 -- 172.21.15.146:0/50174025 shutdown_connections 2024-04-24T08:30:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c9814d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9808270 0x55f0c9812400 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 --2- 172.21.15.146:0/50174025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c9807d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 -- 172.21.15.146:0/50174025 >> 172.21.15.146:0/50174025 conn(0x55f0c9803c70 msgr2=0x55f0c9804050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f423f200640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42240a3da0 0x7f42240a6260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4238005fd0 tx=0x7f4238009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 -- 172.21.15.146:0/50174025 shutdown_connections 2024-04-24T08:30:10.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 -- 172.21.15.146:0/50174025 wait complete. 2024-04-24T08:30:10.035 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 -- start start 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.027+0000 7f4248174740 1 Processor -- start 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4248174740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c98123f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4248174740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c980f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4248174740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c980fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4248174740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f0c98157d0 con 0x55f0c9808270 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4248174740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f0c9726810 con 0x55f0c9812940 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4248174740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0c9815950 con 0x55f0c9807950 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4248174740 10 client.0 _ll_register_callbacks cb 0x55f0c96f4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:10.036 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f4235e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c980f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f423f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c980fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f423f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c980fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53660/0 (socket says 172.21.15.146:53660) 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f423e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c98123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.028+0000 7f423f200640 1 -- 172.21.15.146:0/1749632254 learned_addr learned my addr 172.21.15.146:0/1749632254 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f423f200640 1 -- 172.21.15.146:0/1749632254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 msgr2=0x55f0c98123f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f423f200640 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c98123f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f423f200640 1 -- 172.21.15.146:0/1749632254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 msgr2=0x55f0c980f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f423f200640 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c980f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f423f200640 1 -- 172.21.15.146:0/1749632254 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0c97373f0 con 0x55f0c9812940 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f423f200640 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c980fe20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4238003f90 tx=0x7f4238004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1d0800b97cda968 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f4223e00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f4223e00640 1 -- 172.21.15.146:0/1749632254 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423800f950 con 0x55f0c9812940 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f4223e00640 1 -- 172.21.15.146:0/1749632254 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f423800faf0 con 0x55f0c9812940 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.029+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42300be560 con 0x55f0c9812940 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4223e00640 1 -- 172.21.15.146:0/1749632254 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f423800fcf0 con 0x55f0c9812940 2024-04-24T08:30:10.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 10 client.27185 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f0c947f3f0 con 0x55f0c9812940 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4223e00640 1 -- 172.21.15.146:0/1749632254 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42300be560 con 0x55f0c9812940 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4223e00640 1 -- 172.21.15.146:0/1749632254 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f423802f030 con 0x55f0c9812940 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 10 client.27185 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 10 client.27185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 20 client.27185 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9822d80 0x55f0c9825200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 4 client.27185 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:10.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.030+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0c977abf0 con 0x55f0c9822d80 2024-04-24T08:30:10.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.031+0000 7f423e800640 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9822d80 0x55f0c9825200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.031+0000 7f423e800640 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9822d80 0x55f0c9825200 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c74fa583debb7f71 server_cookie=e52e43a3ec0036a0 in_seq=0 out_seq=0 2024-04-24T08:30:10.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.031+0000 7f4223e00640 10 client.27185 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:10.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4223e00640 1 -- 172.21.15.146:0/1749632254 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0c977abf0 con 0x55f0c9822d80 2024-04-24T08:30:10.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4223e00640 10 client.27185 handle_command_reply: tid=0 2024-04-24T08:30:10.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 client.27185 shutdown 2024-04-24T08:30:10.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 msgr2=0x55f0c980fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c980fe20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4238003f90 tx=0x7f4238004ab0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 shutdown_connections 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9822d80 0x55f0c9825200 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c980fe20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c980f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4223e00640 0 client.27185 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 --2- 172.21.15.146:0/1749632254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c98123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 >> 172.21.15.146:0/1749632254 conn(0x55f0c9803c70 msgr2=0x55f0c9809a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:10.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 shutdown_connections 2024-04-24T08:30:10.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.032+0000 7f4248174740 1 -- 172.21.15.146:0/1749632254 wait complete. 2024-04-24T08:30:10.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.033+0000 7f4248174740 20 client.27185 trim_cache size 0 max 16384 2024-04-24T08:30:10.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.047+0000 7f4248174740 1 Processor -- start 2024-04-24T08:30:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.047+0000 7f4248174740 1 -- start start 2024-04-24T08:30:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f4248174740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c981e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f4248174740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c9811f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f4248174740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c982aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f4248174740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b6e00 con 0x55f0c9808270 2024-04-24T08:30:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f4248174740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b46d0 con 0x55f0c9812940 2024-04-24T08:30:10.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f4248174740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b2540 con 0x55f0c9807950 2024-04-24T08:30:10.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f423f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c982aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f4235e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c9811f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f423f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c982aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:53676/0 (socket says 172.21.15.146:53676) 2024-04-24T08:30:10.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f423f200640 1 -- 172.21.15.146:0/2686259148 learned_addr learned my addr 172.21.15.146:0/2686259148 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:10.052 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.048+0000 7f423e800640 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c981e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f423f200640 1 -- 172.21.15.146:0/2686259148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 msgr2=0x55f0c981e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f423f200640 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c981e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f423f200640 1 -- 172.21.15.146:0/2686259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 msgr2=0x55f0c9811f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f423f200640 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c9811f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f423f200640 1 -- 172.21.15.146:0/2686259148 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0c95497d0 con 0x55f0c9812940 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f4235e00640 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c9811f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f423f200640 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c982aad0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4238029770 tx=0x7f4238029440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cf35dcf275c14e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:10.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f422ea00640 1 -- 172.21.15.146:0/2686259148 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238004890 con 0x55f0c9812940 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f422ea00640 1 -- 172.21.15.146:0/2686259148 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423800f950 con 0x55f0c9812940 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.049+0000 7f422ea00640 1 -- 172.21.15.146:0/2686259148 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423801e470 con 0x55f0c9812940 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 -- 172.21.15.146:0/2686259148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 msgr2=0x55f0c982aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c982aad0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4238029770 tx=0x7f4238029440 comp rx=0 tx=0).stop 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 -- 172.21.15.146:0/2686259148 shutdown_connections 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9812940 0x55f0c982aad0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9808270 0x55f0c9811f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 --2- 172.21.15.146:0/2686259148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9807950 0x55f0c981e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 -- 172.21.15.146:0/2686259148 >> 172.21.15.146:0/2686259148 conn(0x55f0c9803c70 msgr2=0x55f0c980e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 -- 172.21.15.146:0/2686259148 shutdown_connections 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 -- 172.21.15.146:0/2686259148 wait complete. 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.050+0000 7f4248174740 1 -- start start 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 1 Processor -- start 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9807950 0x55f0c981b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9808270 0x55f0c980cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c980d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b2b90 con 0x55f0c9812940 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b4000 con 0x55f0c9807950 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42401b3260 con 0x55f0c9808270 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4248174740 10 client.0 _ll_register_callbacks cb 0x7f4238009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f423f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c980d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f4235e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9808270 0x55f0c980cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f423e800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9807950 0x55f0c981b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f423f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c980d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:56418/0 (socket says 172.21.15.146:56418) 2024-04-24T08:30:10.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.051+0000 7f423f200640 1 -- 172.21.15.146:0/3200316540 learned_addr learned my addr 172.21.15.146:0/3200316540 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4235e00640 1 -- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9807950 msgr2=0x55f0c981b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4235e00640 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9807950 0x55f0c981b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4235e00640 1 -- 172.21.15.146:0/3200316540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 msgr2=0x55f0c980d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4235e00640 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c980d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4235e00640 1 -- 172.21.15.146:0/3200316540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0c9813700 con 0x55f0c9808270 2024-04-24T08:30:10.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f423e800640 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9807950 0x55f0c981b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f423f200640 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c980d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4235e00640 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9808270 0x55f0c980cb80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4228009900 tx=0x7f4228031a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fadc89d8de4db72f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4223e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4223e00640 1 -- 172.21.15.146:0/3200316540 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422804b020 con 0x55f0c9808270 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f423800f950 con 0x55f0c9808270 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.052+0000 7f4223e00640 1 -- 172.21.15.146:0/3200316540 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f4228005020 con 0x55f0c9808270 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4223e00640 1 -- 172.21.15.146:0/3200316540 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4228053050 con 0x55f0c9808270 2024-04-24T08:30:10.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 10 client.27193 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f0c9889640 con 0x55f0c9808270 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4223e00640 1 -- 172.21.15.146:0/3200316540 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423800f950 con 0x55f0c9808270 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4223e00640 1 -- 172.21.15.146:0/3200316540 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f42280035f0 con 0x55f0c9808270 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 10 client.27193 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 10 client.27193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 20 client.27193 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9889d60 0x55f0c988c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:10.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 4 client.27193 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.053+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f0c9774a60 con 0x55f0c9889d60 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.054+0000 7f423e800640 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9889d60 0x55f0c988c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.054+0000 7f423e800640 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9889d60 0x55f0c988c180 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c32103da4c87870 server_cookie=35034d0c03622325 in_seq=0 out_seq=0 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.054+0000 7f4223e00640 10 client.27193 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.054+0000 7f4223e00640 1 -- 172.21.15.146:0/3200316540 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f0c9774a60 con 0x55f0c9889d60 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.054+0000 7f4223e00640 10 client.27193 handle_command_reply: tid=0 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.054+0000 7f4248174740 1 client.27193 shutdown 2024-04-24T08:30:10.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.054+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9808270 msgr2=0x55f0c980cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9808270 0x55f0c980cb80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4228009900 tx=0x7f4228031a10 comp rx=0 tx=0).stop 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 shutdown_connections 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f0c9889d60 0x55f0c988c180 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0c9808270 0x55f0c980cb80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f0c9812940 0x55f0c980d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 --2- 172.21.15.146:0/3200316540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f0c9807950 0x55f0c981b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 >> 172.21.15.146:0/3200316540 conn(0x55f0c9803c70 msgr2=0x55f0c9802560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:10.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4223e00640 0 client.27193 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:10.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 shutdown_connections 2024-04-24T08:30:10.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 1 -- 172.21.15.146:0/3200316540 wait complete. 2024-04-24T08:30:10.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4248174740 20 client.27193 trim_cache size 0 max 16384 2024-04-24T08:30:10.064 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:30:10.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42240a3da0 msgr2=0x7f42240a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4245400640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42240a3da0 0x7f42240a6260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4238005fd0 tx=0x7f4238009500 comp rx=0 tx=0).stop 2024-04-24T08:30:10.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240108610 msgr2=0x7f424019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:10.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.055+0000 7f4245400640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240108610 0x7f424019eea0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f423000e9c0 tx=0x7f423000edc0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 shutdown_connections 2024-04-24T08:30:10.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42240a3da0 0x7f42240a6260 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4240108fd0 0x7f424019f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424010e660 0x7f42401a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 --2- 172.21.15.146:0/666940454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4240108610 0x7f424019eea0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:10.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 >> 172.21.15.146:0/666940454 conn(0x7f4240076540 msgr2=0x7f4240077d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:10.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 shutdown_connections 2024-04-24T08:30:10.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:10.056+0000 7f4245400640 1 -- 172.21.15.146:0/666940454 wait complete. 2024-04-24T08:30:10.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:10 smithi153 ceph-mon[32746]: pgmap v277: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:30:10.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:10 smithi146 ceph-mon[27805]: pgmap v277: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:30:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:10 smithi177 ceph-mon[33046]: pgmap v277: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:30:11.074 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:30:11.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:30:11.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:30:11.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 -- 172.21.15.146:0/2656866791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104e00 msgr2=0x7f0fac108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2656866791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104e00 0x7f0fac108e40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009900 tx=0x7f0f9402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 -- 172.21.15.146:0/2656866791 shutdown_connections 2024-04-24T08:30:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2656866791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fac109520 0x7f0fac1110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2656866791 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104e00 0x7f0fac108e40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2656866791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac104450 0x7f0fac104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 -- 172.21.15.146:0/2656866791 >> 172.21.15.146:0/2656866791 conn(0x7f0fac0ffc70 msgr2=0x7f0fac102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 -- 172.21.15.146:0/2656866791 shutdown_connections 2024-04-24T08:30:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.296+0000 7f0fb2800640 1 -- 172.21.15.146:0/2656866791 wait complete. 2024-04-24T08:30:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.297+0000 7f0fb2800640 1 Processor -- start 2024-04-24T08:30:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.297+0000 7f0fb2800640 1 -- start start 2024-04-24T08:30:11.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fb2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104450 0x7f0fac1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fb2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fac109520 0x7f0fac1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fb2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac19a970 0x7f0fac19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fb2800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac114590 con 0x7f0fac19a970 2024-04-24T08:30:11.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fb2800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac114410 con 0x7f0fac104450 2024-04-24T08:30:11.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fb2800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac114710 con 0x7f0fac109520 2024-04-24T08:30:11.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac19a970 0x7f0fac19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fab400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104450 0x7f0fac1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.305 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac19a970 0x7f0fac19ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35022/0 (socket says 172.21.15.146:35022) 2024-04-24T08:30:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fab400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104450 0x7f0fac1a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52294/0 (socket says 172.21.15.146:52294) 2024-04-24T08:30:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fa2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fac109520 0x7f0fac1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fabe00640 1 -- 172.21.15.146:0/2430153404 learned_addr learned my addr 172.21.15.146:0/2430153404 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fab400640 1 -- 172.21.15.146:0/2430153404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fac109520 msgr2=0x7f0fac1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fab400640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fac109520 0x7f0fac1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fab400640 1 -- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac19a970 msgr2=0x7f0fac19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fab400640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac19a970 0x7f0fac19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fab400640 1 -- 172.21.15.146:0/2430153404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fac19b490 con 0x7f0fac104450 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.298+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac19a970 0x7f0fac19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.299+0000 7f0fab400640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104450 0x7f0fac1a07c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00ec80 tx=0x7f0f9c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.299+0000 7f0fa8c00640 1 -- 172.21.15.146:0/2430153404 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c003b70 con 0x7f0fac104450 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.299+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fac19b780 con 0x7f0fac104450 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.299+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fac1a7a20 con 0x7f0fac104450 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.299+0000 7f0fa8c00640 1 -- 172.21.15.146:0/2430153404 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f9c003d10 con 0x7f0fac104450 2024-04-24T08:30:11.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.299+0000 7f0fa8c00640 1 -- 172.21.15.146:0/2430153404 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c010a10 con 0x7f0fac104450 2024-04-24T08:30:11.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.302+0000 7f0fb57b0740 1 Processor -- start 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.302+0000 7f0fb57b0740 1 -- start start 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01070470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a0107ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a0107d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564a00fd5e70 con 0x564a01070090 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a00f7c380 con 0x564a0107b080 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a00fd1e30 con 0x564a010709b0 2024-04-24T08:30:11.315 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fa8c00640 1 -- 172.21.15.146:0/2430153404 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f0f9c004600 con 0x7f0fac104450 2024-04-24T08:30:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fa2a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a0107d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fa2a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a0107d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52300/0 (socket says 172.21.15.146:52300) 2024-04-24T08:30:11.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fa2a00640 1 -- 172.21.15.146:0/3346822779 learned_addr learned my addr 172.21.15.146:0/3346822779 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.303+0000 7f0fab400640 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a0107ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa2a00640 1 -- 172.21.15.146:0/3346822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 msgr2=0x564a0107ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa2a00640 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a0107ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa2a00640 1 -- 172.21.15.146:0/3346822779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 msgr2=0x564a01070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa2a00640 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa2a00640 1 -- 172.21.15.146:0/3346822779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a00db07e0 con 0x564a0107b080 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa2a00640 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a0107d470 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009900 tx=0x7f0f94005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbff76fd31bf3ddf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa0c00640 1 -- 172.21.15.146:0/3346822779 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94040070 con 0x564a0107b080 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa0c00640 1 -- 172.21.15.146:0/3346822779 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f94031870 con 0x564a0107b080 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.304+0000 7f0fa0c00640 1 -- 172.21.15.146:0/3346822779 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94035040 con 0x564a0107b080 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fa8c00640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0f900a3da0 0x7f0f900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 -- 172.21.15.146:0/3346822779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 msgr2=0x564a0107d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a0107d470 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009900 tx=0x7f0f94005bd0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fa8c00640 1 -- 172.21.15.146:0/2430153404 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f0f9c0d0b90 con 0x7f0fac104450 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 -- 172.21.15.146:0/3346822779 shutdown_connections 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a0107ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a0107d470 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/3346822779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 -- 172.21.15.146:0/3346822779 >> 172.21.15.146:0/3346822779 conn(0x564a0106c3b0 msgr2=0x564a0106c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0f900a3da0 0x7f0f900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 -- 172.21.15.146:0/3346822779 shutdown_connections 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.305+0000 7f0fb57b0740 1 -- 172.21.15.146:0/3346822779 wait complete. 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.306+0000 7f0fb57b0740 1 -- start start 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.306+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0f900a3da0 0x7f0f900a6260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009fd0 tx=0x7f0fa4005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.306+0000 7f0fb57b0740 1 Processor -- start 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.306+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01077fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.306+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a01078500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.306+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01078a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.306+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564a0107de10 con 0x564a01070090 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564a00f8f660 con 0x564a0107b080 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a0107df90 con 0x564a010709b0 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fb57b0740 10 client.0 _ll_register_callbacks cb 0x564a00f7daa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01078a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01077fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01078a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52306/0 (socket says 172.21.15.146:52306) 2024-04-24T08:30:11.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 -- 172.21.15.146:0/2076045961 learned_addr learned my addr 172.21.15.146:0/2076045961 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fa2a00640 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a01078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 -- 172.21.15.146:0/2076045961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 msgr2=0x564a01078500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a01078500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 -- 172.21.15.146:0/2076045961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 msgr2=0x564a01077fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01077fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.307+0000 7f0fabe00640 1 -- 172.21.15.146:0/2076045961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a00e10f70 con 0x564a0107b080 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01078a40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400fbb0 tx=0x7f0fa4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76904dd21db37cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0f99600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:11.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0f99600640 1 -- 172.21.15.146:0/2076045961 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa402e060 con 0x564a0107b080 2024-04-24T08:30:11.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f94031870 con 0x564a0107b080 2024-04-24T08:30:11.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0f99600640 1 -- 172.21.15.146:0/2076045961 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f0fa4018030 con 0x564a0107b080 2024-04-24T08:30:11.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0f99600640 1 -- 172.21.15.146:0/2076045961 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0fa401e5e0 con 0x564a0107b080 2024-04-24T08:30:11.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0fb57b0740 10 client.27203 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:11.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.308+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a00ca8170 con 0x564a0107b080 2024-04-24T08:30:11.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0f99600640 1 -- 172.21.15.146:0/2076045961 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f94031870 con 0x564a0107b080 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0f99600640 1 -- 172.21.15.146:0/2076045961 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0fa4029aa0 con 0x564a0107b080 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0fb57b0740 10 client.27203 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0fb57b0740 10 client.27203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0fb57b0740 20 client.27203 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a0108b350 0x564a0108d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0fb57b0740 4 client.27203 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a00fe3890 con 0x564a0108b350 2024-04-24T08:30:11.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.309+0000 7f0fab400640 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a0108b350 0x564a0108d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.310+0000 7f0fab400640 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a0108b350 0x564a0108d7a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ba13a6bfd808536 server_cookie=c3ea98172da71519 in_seq=0 out_seq=0 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.310+0000 7f0f99600640 10 client.27203 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.310+0000 7f0f99600640 1 -- 172.21.15.146:0/2076045961 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a00fe3890 con 0x564a0108b350 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.310+0000 7f0f99600640 10 client.27203 handle_command_reply: tid=0 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 client.27203 shutdown 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 msgr2=0x564a01078a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01078a40 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400fbb0 tx=0x7f0fa4004730 comp rx=0 tx=0).stop 2024-04-24T08:30:11.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 shutdown_connections 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a0108b350 0x564a0108d7a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a010709b0 0x564a01078500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01078a40 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2076045961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a01070090 0x564a01077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0f99600640 0 client.27203 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 >> 172.21.15.146:0/2076045961 conn(0x564a0106c3b0 msgr2=0x564a010721b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 shutdown_connections 2024-04-24T08:30:11.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2076045961 wait complete. 2024-04-24T08:30:11.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.311+0000 7f0fb57b0740 20 client.27203 trim_cache size 0 max 16384 2024-04-24T08:30:11.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.330+0000 7f0fb57b0740 1 Processor -- start 2024-04-24T08:30:11.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.330+0000 7f0fb57b0740 1 -- start start 2024-04-24T08:30:11.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01086cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a010727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01093070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac1b2970 con 0x564a010709b0 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac1b0240 con 0x564a0107b080 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac1ae0b0 con 0x564a01070090 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a010727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a010727f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35044/0 (socket says 172.21.15.146:35044) 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 -- 172.21.15.146:0/4169218905 learned_addr learned my addr 172.21.15.146:0/4169218905 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fabe00640 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01093070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fab400640 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01086cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 -- 172.21.15.146:0/4169218905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 msgr2=0x564a01086cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01086cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 -- 172.21.15.146:0/4169218905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 msgr2=0x564a01093070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01093070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.331+0000 7f0fa2a00640 1 -- 172.21.15.146:0/4169218905 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a01076510 con 0x564a010709b0 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.332+0000 7f0fabe00640 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01093070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.332+0000 7f0fa2a00640 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a010727f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0f94005c40 tx=0x7f0f94009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782eb9c726948db1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.332+0000 7f0f9be00640 1 -- 172.21.15.146:0/4169218905 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94040050 con 0x564a010709b0 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.332+0000 7f0f9be00640 1 -- 172.21.15.146:0/4169218905 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f94031870 con 0x564a010709b0 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.332+0000 7f0f9be00640 1 -- 172.21.15.146:0/4169218905 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f94035040 con 0x564a010709b0 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 -- 172.21.15.146:0/4169218905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 msgr2=0x564a010727f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a010727f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0f94005c40 tx=0x7f0f94009260 comp rx=0 tx=0).stop 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 -- 172.21.15.146:0/4169218905 shutdown_connections 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a01093070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a010727f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/4169218905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01086cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 -- 172.21.15.146:0/4169218905 >> 172.21.15.146:0/4169218905 conn(0x564a0106c3b0 msgr2=0x564a0106b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 -- 172.21.15.146:0/4169218905 shutdown_connections 2024-04-24T08:30:11.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.333+0000 7f0fb57b0740 1 -- 172.21.15.146:0/4169218905 wait complete. 2024-04-24T08:30:11.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 -- start start 2024-04-24T08:30:11.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 Processor -- start 2024-04-24T08:30:11.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01074d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a01075270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a010757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac1ae700 con 0x564a010709b0 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac1afb70 con 0x564a0107b080 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0fac1aedd0 con 0x564a01070090 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.334+0000 7f0fb57b0740 10 client.0 _ll_register_callbacks cb 0x7f0fa4009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fa2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a01075270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fabe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a010757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.341 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fa2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a01075270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35048/0 (socket says 172.21.15.146:35048) 2024-04-24T08:30:11.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fa2a00640 1 -- 172.21.15.146:0/2129718250 learned_addr learned my addr 172.21.15.146:0/2129718250 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fab400640 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01074d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fabe00640 1 -- 172.21.15.146:0/2129718250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 msgr2=0x564a01074d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01074d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fabe00640 1 -- 172.21.15.146:0/2129718250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 msgr2=0x564a01075270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a01075270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.342 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fabe00640 1 -- 172.21.15.146:0/2129718250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a010f91a0 con 0x564a0107b080 2024-04-24T08:30:11.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0fabe00640 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a010757b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4019040 tx=0x7f0fa4012270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ad34541ddb736f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.335+0000 7f0f99600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:11.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0f99600640 1 -- 172.21.15.146:0/2129718250 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa403f060 con 0x564a0107b080 2024-04-24T08:30:11.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f94031870 con 0x564a0107b080 2024-04-24T08:30:11.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0f99600640 1 -- 172.21.15.146:0/2129718250 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f0fa4031b40 con 0x564a0107b080 2024-04-24T08:30:11.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0f99600640 1 -- 172.21.15.146:0/2129718250 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0fa400f9a0 con 0x564a0107b080 2024-04-24T08:30:11.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0fb57b0740 10 client.27209 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:11.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a010fa520 con 0x564a0107b080 2024-04-24T08:30:11.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0f99600640 1 -- 172.21.15.146:0/2129718250 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f94031870 con 0x564a0107b080 2024-04-24T08:30:11.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0f99600640 1 -- 172.21.15.146:0/2129718250 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0fa4029420 con 0x564a0107b080 2024-04-24T08:30:11.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0fb57b0740 10 client.27209 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:11.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0fb57b0740 10 client.27209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:11.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.336+0000 7f0fb57b0740 20 client.27209 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:11.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.337+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a010fac50 0x564a010fd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.337+0000 7f0fb57b0740 4 client.27209 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:30:11.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.337+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564a00fdd700 con 0x564a010fac50 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.337+0000 7f0fab400640 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a010fac50 0x564a010fd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.337+0000 7f0fab400640 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a010fac50 0x564a010fd070 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fae876a57c10336 server_cookie=401ad3c4c2200ca7 in_seq=0 out_seq=0 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.337+0000 7f0f99600640 10 client.27209 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0f99600640 1 -- 172.21.15.146:0/2129718250 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564a00fdd700 con 0x564a010fac50 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0f99600640 10 client.27209 handle_command_reply: tid=0 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 client.27209 shutdown 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 msgr2=0x564a010757b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a010757b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4019040 tx=0x7f0fa4012270 comp rx=0 tx=0).stop 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 shutdown_connections 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564a010fac50 0x564a010fd070 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564a0107b080 0x564a010757b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564a010709b0 0x564a01075270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 --2- 172.21.15.146:0/2129718250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a01070090 0x564a01074d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 >> 172.21.15.146:0/2129718250 conn(0x564a0106c3b0 msgr2=0x564a01077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0f99600640 0 client.27209 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 shutdown_connections 2024-04-24T08:30:11.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.339+0000 7f0fb57b0740 1 -- 172.21.15.146:0/2129718250 wait complete. 2024-04-24T08:30:11.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb57b0740 20 client.27209 trim_cache size 0 max 16384 2024-04-24T08:30:11.349 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:30:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:30:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "2fd56a5e-cd2a-4df5-aa75-2815da928b0e", 2024-04-24T08:30:11.349 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:30:11.349 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:30:11.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0f900a3da0 msgr2=0x7f0f900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0f900a3da0 0x7f0f900a6260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009fd0 tx=0x7f0fa4005f90 comp rx=0 tx=0).stop 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104450 msgr2=0x7f0fac1a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104450 0x7f0fac1a07c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00ec80 tx=0x7f0f9c00c6a0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 shutdown_connections 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0f900a3da0 0x7f0f900a6260 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0fac109520 0x7f0fac1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fac19a970 0x7f0fac19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 --2- 172.21.15.146:0/2430153404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0fac104450 0x7f0fac1a07c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.340+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 >> 172.21.15.146:0/2430153404 conn(0x7f0fac0ffc70 msgr2=0x7f0fac101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.341+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 shutdown_connections 2024-04-24T08:30:11.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.341+0000 7f0fb2800640 1 -- 172.21.15.146:0/2430153404 wait complete. 2024-04-24T08:30:11.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:30:11.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:30:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.625+0000 7f8b2e600640 1 -- 172.21.15.146:0/2934679749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b28074bf0 msgr2=0x7f8b2810e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.625+0000 7f8b2e600640 1 --2- 172.21.15.146:0/2934679749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b28074bf0 0x7f8b2810e3a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009900 tx=0x7f8b1402a480 comp rx=0 tx=0).stop 2024-04-24T08:30:11.630 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.626+0000 7f8b2e600640 1 -- 172.21.15.146:0/2934679749 shutdown_connections 2024-04-24T08:30:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.626+0000 7f8b2e600640 1 --2- 172.21.15.146:0/2934679749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b2810e8e0 0x7f8b28110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.626+0000 7f8b2e600640 1 --2- 172.21.15.146:0/2934679749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b28074bf0 0x7f8b2810e3a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.626+0000 7f8b2e600640 1 --2- 172.21.15.146:0/2934679749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b280742d0 0x7f8b280746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.626+0000 7f8b2e600640 1 -- 172.21.15.146:0/2934679749 >> 172.21.15.146:0/2934679749 conn(0x7f8b28079450 msgr2=0x7f8b28079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.626+0000 7f8b2e600640 1 -- 172.21.15.146:0/2934679749 shutdown_connections 2024-04-24T08:30:11.631 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.626+0000 7f8b2e600640 1 -- 172.21.15.146:0/2934679749 wait complete. 2024-04-24T08:30:11.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.628+0000 7f8b2e600640 1 Processor -- start 2024-04-24T08:30:11.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.628+0000 7f8b2e600640 1 -- start start 2024-04-24T08:30:11.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b2e600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b280742d0 0x7f8b281a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b2e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b28074bf0 0x7f8b281a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b2e600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b2810e8e0 0x7f8b2819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b2e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b28114620 con 0x7f8b28074bf0 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b2e600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b281144a0 con 0x7f8b2810e8e0 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b2e600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b281147a0 con 0x7f8b280742d0 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b26a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b28074bf0 0x7f8b281a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b280742d0 0x7f8b281a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.634 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b26a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b28074bf0 0x7f8b281a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:35066/0 (socket says 172.21.15.146:35066) 2024-04-24T08:30:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b26a00640 1 -- 172.21.15.146:0/558266197 learned_addr learned my addr 172.21.15.146:0/558266197 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.629+0000 7f8b27e00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b2810e8e0 0x7f8b2819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.630+0000 7f8b26a00640 1 -- 172.21.15.146:0/558266197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b280742d0 msgr2=0x7f8b281a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.630+0000 7f8b26a00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b280742d0 0x7f8b281a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.630+0000 7f8b26a00640 1 -- 172.21.15.146:0/558266197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b2810e8e0 msgr2=0x7f8b2819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.630+0000 7f8b26a00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b2810e8e0 0x7f8b2819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.635 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.630+0000 7f8b26a00640 1 -- 172.21.15.146:0/558266197 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2819b130 con 0x7f8b28074bf0 2024-04-24T08:30:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.630+0000 7f8b27e00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b2810e8e0 0x7f8b2819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:30:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.631+0000 7f8b26a00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b28074bf0 0x7f8b281a0d20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8b1402a9f0 tx=0x7f8b14003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.632+0000 7f8b1fe00640 1 -- 172.21.15.146:0/558266197 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1403f070 con 0x7f8b28074bf0 2024-04-24T08:30:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.632+0000 7f8b1fe00640 1 -- 172.21.15.146:0/558266197 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b14031c90 con 0x7f8b28074bf0 2024-04-24T08:30:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.632+0000 7f8b1fe00640 1 -- 172.21.15.146:0/558266197 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14035030 con 0x7f8b28074bf0 2024-04-24T08:30:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.632+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2819b3c0 con 0x7f8b28074bf0 2024-04-24T08:30:11.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.632+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b2806ba20 con 0x7f8b28074bf0 2024-04-24T08:30:11.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.637+0000 7f8b1fe00640 1 -- 172.21.15.146:0/558266197 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f8b14042070 con 0x7f8b28074bf0 2024-04-24T08:30:11.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.637+0000 7f8b31397740 1 Processor -- start 2024-04-24T08:30:11.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.637+0000 7f8b31397740 1 -- start start 2024-04-24T08:30:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b31397740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8e860 0x55ea20e8ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b31397740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8f180 0x55ea20e99310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b31397740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e9bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b31397740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55ea20df41f0 con 0x55ea20e8e860 2024-04-24T08:30:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b31397740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ea20d9a350 con 0x55ea20e99850 2024-04-24T08:30:11.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b31397740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ea20df01b0 con 0x55ea20e8f180 2024-04-24T08:30:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b26a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e9bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8f180 0x55ea20e99310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8f180 0x55ea20e99310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33508/0 (socket says 172.21.15.146:33508) 2024-04-24T08:30:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b27400640 1 -- 172.21.15.146:0/1384919784 learned_addr learned my addr 172.21.15.146:0/1384919784 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.641 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.638+0000 7f8b27e00640 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8e860 0x55ea20e8ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b26a00640 1 -- 172.21.15.146:0/1384919784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8f180 msgr2=0x55ea20e99310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b26a00640 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8f180 0x55ea20e99310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b26a00640 1 -- 172.21.15.146:0/1384919784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8e860 msgr2=0x55ea20e8ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b26a00640 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8e860 0x55ea20e8ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.642 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b26a00640 1 -- 172.21.15.146:0/1384919784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea20deef50 con 0x55ea20e99850 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b27400640 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8f180 0x55ea20e99310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b27e00640 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8e860 0x55ea20e8ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b1fe00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b0c0a3ea0 0x7f8b0c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.639+0000 7f8b1fe00640 1 -- 172.21.15.146:0/558266197 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f8b140f0e90 con 0x7f8b28074bf0 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.640+0000 7f8b26a00640 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e9bc40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8b140ee5a0 tx=0x7f8b140f4f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6701e4c7bec9f87 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.640+0000 7f8b1cc00640 1 -- 172.21.15.146:0/1384919784 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b140b9090 con 0x55ea20e99850 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.640+0000 7f8b1cc00640 1 -- 172.21.15.146:0/1384919784 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b1403b460 con 0x55ea20e99850 2024-04-24T08:30:11.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.640+0000 7f8b1cc00640 1 -- 172.21.15.146:0/1384919784 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b140bc750 con 0x55ea20e99850 2024-04-24T08:30:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.640+0000 7f8b27e00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b0c0a3ea0 0x7f8b0c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.641+0000 7f8b27e00640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b0c0a3ea0 0x7f8b0c0a6360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8b2806b8d0 tx=0x7f8b18005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.641+0000 7f8b31397740 1 -- 172.21.15.146:0/1384919784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 msgr2=0x55ea20e9bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.641+0000 7f8b31397740 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e9bc40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8b140ee5a0 tx=0x7f8b140f4f90 comp rx=0 tx=0).stop 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.642+0000 7f8b31397740 1 -- 172.21.15.146:0/1384919784 shutdown_connections 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.642+0000 7f8b31397740 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8f180 0x55ea20e99310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.642+0000 7f8b31397740 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e9bc40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.642+0000 7f8b31397740 1 --2- 172.21.15.146:0/1384919784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8e860 0x55ea20e8ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.642+0000 7f8b31397740 1 -- 172.21.15.146:0/1384919784 >> 172.21.15.146:0/1384919784 conn(0x55ea20e8ab80 msgr2=0x55ea20e8af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.642+0000 7f8b31397740 1 -- 172.21.15.146:0/1384919784 shutdown_connections 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.642+0000 7f8b31397740 1 -- 172.21.15.146:0/1384919784 wait complete. 2024-04-24T08:30:11.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.643+0000 7f8b31397740 1 -- start start 2024-04-24T08:30:11.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.644+0000 7f8b31397740 1 Processor -- start 2024-04-24T08:30:11.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.644+0000 7f8b31397740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20e96840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b31397740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e96d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b31397740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b31397740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55ea20e9c400 con 0x55ea20e8f180 2024-04-24T08:30:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b31397740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ea20dad690 con 0x55ea20e99850 2024-04-24T08:30:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b31397740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ea20e9c580 con 0x55ea20e8e860 2024-04-24T08:30:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20e96840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20e96840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33520/0 (socket says 172.21.15.146:33520) 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 -- 172.21.15.146:0/3872230685 learned_addr learned my addr 172.21.15.146:0/3872230685 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b31397740 10 client.0 _ll_register_callbacks cb 0x55ea20de9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b26a00640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e96d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27e00640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 -- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 msgr2=0x55ea20e972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 -- 172.21.15.146:0/3872230685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 msgr2=0x55ea20e96d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e96d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27400640 1 -- 172.21.15.146:0/3872230685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea20dac920 con 0x55ea20e8e860 2024-04-24T08:30:11.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.645+0000 7f8b27e00640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:11.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.646+0000 7f8b27400640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20e96840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8b20003bf0 tx=0x7f8b20004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed248e694174aabc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.646+0000 7f8b11600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:30:11.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.646+0000 7f8b11600640 1 -- 172.21.15.146:0/3872230685 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b20011910 con 0x55ea20e8e860 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.646+0000 7f8b11600640 1 -- 172.21.15.146:0/3872230685 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f8b20011ab0 con 0x55ea20e8e860 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.646+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ea20e01c10 con 0x55ea20e8e860 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.647+0000 7f8b11600640 1 -- 172.21.15.146:0/3872230685 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8b20011cb0 con 0x55ea20e8e860 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.647+0000 7f8b31397740 10 client.27223 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.647+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ea20bd07d0 con 0x55ea20e8e860 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.647+0000 7f8b11600640 1 -- 172.21.15.146:0/3872230685 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ea20e01c10 con 0x55ea20e8e860 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.647+0000 7f8b11600640 1 -- 172.21.15.146:0/3872230685 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8b20019020 con 0x55ea20e8e860 2024-04-24T08:30:11.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.647+0000 7f8b31397740 10 client.27223 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.647+0000 7f8b31397740 10 client.27223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.648+0000 7f8b31397740 20 client.27223 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.648+0000 7f8b31397740 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20eb2120 0x55ea20eb4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.648+0000 7f8b31397740 4 client.27223 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.648+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea20dfba80 con 0x55ea20eb2120 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.648+0000 7f8b26a00640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20eb2120 0x55ea20eb4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.649+0000 7f8b26a00640 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20eb2120 0x55ea20eb4590 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9bcac3b1cafbbf2 server_cookie=a05c7cfe2eb4cd1a in_seq=0 out_seq=0 2024-04-24T08:30:11.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.649+0000 7f8b11600640 10 client.27223 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.650+0000 7f8b11600640 1 -- 172.21.15.146:0/3872230685 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea20dfba80 con 0x55ea20eb2120 2024-04-24T08:30:11.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.650+0000 7f8b11600640 10 client.27223 handle_command_reply: tid=0 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.650+0000 7f8b31397740 1 client.27223 shutdown 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 msgr2=0x55ea20e96840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b31397740 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20e96840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8b20003bf0 tx=0x7f8b20004410 comp rx=0 tx=0).stop 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 shutdown_connections 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b31397740 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20eb2120 0x55ea20eb4590 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b31397740 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b31397740 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e96d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b11600640 0 client.27223 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.651+0000 7f8b31397740 1 --2- 172.21.15.146:0/3872230685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20e96840 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.652+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 >> 172.21.15.146:0/3872230685 conn(0x55ea20e8ab80 msgr2=0x55ea20e90980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.652+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 shutdown_connections 2024-04-24T08:30:11.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.653+0000 7f8b31397740 1 -- 172.21.15.146:0/3872230685 wait complete. 2024-04-24T08:30:11.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.653+0000 7f8b31397740 20 client.27223 trim_cache size 0 max 16384 2024-04-24T08:30:11.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 Processor -- start 2024-04-24T08:30:11.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 -- start start 2024-04-24T08:30:11.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eadac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e8f180 0x55ea20e90fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e99850 0x55ea20e98780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b28072930 con 0x55ea20e99850 2024-04-24T08:30:11.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b28070200 con 0x55ea20e8f180 2024-04-24T08:30:11.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.674+0000 7f8b31397740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2806e070 con 0x55ea20e8e860 2024-04-24T08:30:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eadac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b26a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e8f180 0x55ea20e90fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eadac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33524/0 (socket says 172.21.15.146:33524) 2024-04-24T08:30:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b27400640 1 -- 172.21.15.146:0/4167645123 learned_addr learned my addr 172.21.15.146:0/4167645123 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b27e00640 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e99850 0x55ea20e98780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b26a00640 1 -- 172.21.15.146:0/4167645123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 msgr2=0x55ea20eadac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b26a00640 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eadac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b26a00640 1 -- 172.21.15.146:0/4167645123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e99850 msgr2=0x55ea20e98780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b26a00640 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e99850 0x55ea20e98780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b26a00640 1 -- 172.21.15.146:0/4167645123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea20b063f0 con 0x55ea20e8f180 2024-04-24T08:30:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b27400640 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eadac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b26a00640 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e8f180 0x55ea20e90fc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8b140bb0a0 tx=0x7f8b1403b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0c6496650e411c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b13e00640 1 -- 172.21.15.146:0/4167645123 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b140bc930 con 0x55ea20e8f180 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.675+0000 7f8b13e00640 1 -- 172.21.15.146:0/4167645123 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b140bd4f0 con 0x55ea20e8f180 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b13e00640 1 -- 172.21.15.146:0/4167645123 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b140bcbc0 con 0x55ea20e8f180 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 -- 172.21.15.146:0/4167645123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e8f180 msgr2=0x55ea20e90fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e8f180 0x55ea20e90fc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8b140bb0a0 tx=0x7f8b1403b650 comp rx=0 tx=0).stop 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 -- 172.21.15.146:0/4167645123 shutdown_connections 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e99850 0x55ea20e98780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e8f180 0x55ea20e90fc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 --2- 172.21.15.146:0/4167645123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eadac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 -- 172.21.15.146:0/4167645123 >> 172.21.15.146:0/4167645123 conn(0x55ea20e8ab80 msgr2=0x55ea20e98070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 -- 172.21.15.146:0/4167645123 shutdown_connections 2024-04-24T08:30:11.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.676+0000 7f8b31397740 1 -- 172.21.15.146:0/4167645123 wait complete. 2024-04-24T08:30:11.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 -- start start 2024-04-24T08:30:11.681 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 Processor -- start 2024-04-24T08:30:11.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eaa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e930b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e935f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2806e6c0 con 0x55ea20e8f180 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2806fb30 con 0x55ea20e99850 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2806ed90 con 0x55ea20e8e860 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.677+0000 7f8b31397740 10 client.0 _ll_register_callbacks cb 0x7f8b20009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b26a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e930b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b27e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e935f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eaa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b27e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e935f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52408/0 (socket says 172.21.15.146:52408) 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b27e00640 1 -- 172.21.15.146:0/1866783345 learned_addr learned my addr 172.21.15.146:0/1866783345 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b26a00640 1 -- 172.21.15.146:0/1866783345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 msgr2=0x55ea20eaa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b26a00640 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eaa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b26a00640 1 -- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 msgr2=0x55ea20e935f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b26a00640 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e935f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b26a00640 1 -- 172.21.15.146:0/1866783345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea20ac60f0 con 0x55ea20e8f180 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b27e00640 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e935f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b27400640 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eaa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:11.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b26a00640 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e930b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55ea20e978c0 tx=0x7f8b14009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5daa752c4d3d0b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.678+0000 7f8b11600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b11600640 1 -- 172.21.15.146:0/1866783345 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b140bcbc0 con 0x55ea20e8f180 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b11600640 1 -- 172.21.15.146:0/1866783345 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8586+0+0 (secure 0 0 0) 0x7f8b140ee820 con 0x55ea20e8f180 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b140bd4f0 con 0x55ea20e8f180 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b11600640 1 -- 172.21.15.146:0/1866783345 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8b140bc930 con 0x55ea20e8f180 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 10 client.17079 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ea20db9810 con 0x55ea20e8f180 2024-04-24T08:30:11.687 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b11600640 1 -- 172.21.15.146:0/1866783345 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b140bd4f0 con 0x55ea20e8f180 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b11600640 1 -- 172.21.15.146:0/1866783345 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8b140bd050 con 0x55ea20e8f180 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 10 client.17079 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 20 client.17079 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20f18390 0x55ea20f1a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 4 client.17079 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.679+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ea20d77360 con 0x55ea20f18390 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.680+0000 7f8b27400640 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20f18390 0x55ea20f1a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:11.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.680+0000 7f8b27400640 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20f18390 0x55ea20f1a780 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8137091c0394462e server_cookie=a7fcc935ea0b851d in_seq=0 out_seq=0 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.680+0000 7f8b11600640 10 client.17079 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.680+0000 7f8b11600640 1 -- 172.21.15.146:0/1866783345 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ea20d77360 con 0x55ea20f18390 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.680+0000 7f8b11600640 10 client.17079 handle_command_reply: tid=0 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 client.17079 shutdown 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 msgr2=0x55ea20e930b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e930b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55ea20e978c0 tx=0x7f8b14009560 comp rx=0 tx=0).stop 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 shutdown_connections 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ea20f18390 0x55ea20f1a780 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ea20e99850 0x55ea20e935f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ea20e8f180 0x55ea20e930b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 --2- 172.21.15.146:0/1866783345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ea20e8e860 0x55ea20eaa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b11600640 0 client.17079 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 >> 172.21.15.146:0/1866783345 conn(0x55ea20e8ab80 msgr2=0x55ea20e98900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 shutdown_connections 2024-04-24T08:30:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 1 -- 172.21.15.146:0/1866783345 wait complete. 2024-04-24T08:30:11.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.681+0000 7f8b31397740 20 client.17079 trim_cache size 0 max 16384 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout: "2fd56a5e-cd2a-4df5-aa75-2815da928b0e": { 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "2fd56a5e-cd2a-4df5-aa75-2815da928b0e", 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:30:11.691 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:30:11.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:30:11.692 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:30:11.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b0c0a3ea0 msgr2=0x7f8b0c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b0c0a3ea0 0x7f8b0c0a6360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8b2806b8d0 tx=0x7f8b18005f90 comp rx=0 tx=0).stop 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b28074bf0 msgr2=0x7f8b281a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b28074bf0 0x7f8b281a0d20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8b1402a9f0 tx=0x7f8b14003b90 comp rx=0 tx=0).stop 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 shutdown_connections 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8b0c0a3ea0 0x7f8b0c0a6360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b2810e8e0 0x7f8b2819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b28074bf0 0x7f8b281a0d20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 --2- 172.21.15.146:0/558266197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b280742d0 0x7f8b281a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:11.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 >> 172.21.15.146:0/558266197 conn(0x7f8b28079450 msgr2=0x7f8b280fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:11.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 shutdown_connections 2024-04-24T08:30:11.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:11.682+0000 7f8b2e600640 1 -- 172.21.15.146:0/558266197 wait complete. 2024-04-24T08:30:11.698 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:11 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:30:11.701 INFO:tasks.cephfs.filesystem:scrub status for tag:2fd56a5e-cd2a-4df5-aa75-2815da928b0e - {'path': '/', 'tag': '2fd56a5e-cd2a-4df5-aa75-2815da928b0e', 'options': 'recursive,force'} 2024-04-24T08:30:11.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:11 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:30:11.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:11 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:30:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: pgmap v278: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:30:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:30:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:30:12.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:30:12.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:12 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: pgmap v278: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:30:12.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:30:12.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:30:12.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:30:12.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:12 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: pgmap v278: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:30:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:30:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:30:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:30:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:12.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:12 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:14.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:14 smithi153 ceph-mon[32746]: pgmap v279: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:14.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:14 smithi146 ceph-mon[27805]: pgmap v279: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:14 smithi177 ceph-mon[33046]: pgmap v279: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:16 smithi153 ceph-mon[32746]: pgmap v280: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:16 smithi153 ceph-mon[32746]: osdmap e97: 12 total, 12 up, 12 in 2024-04-24T08:30:17.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:16 smithi146 ceph-mon[27805]: pgmap v280: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:17.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:16 smithi146 ceph-mon[27805]: osdmap e97: 12 total, 12 up, 12 in 2024-04-24T08:30:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:16 smithi177 ceph-mon[33046]: pgmap v280: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:17.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:16 smithi177 ceph-mon[33046]: osdmap e97: 12 total, 12 up, 12 in 2024-04-24T08:30:18.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:17 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:30:18.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:17 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:30:18.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:17 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:30:18.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:17 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:30:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:17 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:30:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:17 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:30:19.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:18 smithi153 ceph-mon[32746]: pgmap v282: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-24T08:30:19.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:18 smithi146 ceph-mon[27805]: pgmap v282: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-24T08:30:19.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:18 smithi177 ceph-mon[33046]: pgmap v282: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-24T08:30:21.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:20 smithi153 ceph-mon[32746]: pgmap v283: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:21.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:20 smithi146 ceph-mon[27805]: pgmap v283: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:21.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:20 smithi177 ceph-mon[33046]: pgmap v283: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:23.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:22 smithi153 ceph-mon[32746]: pgmap v284: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:23.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:22 smithi146 ceph-mon[27805]: pgmap v284: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:23.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:22 smithi177 ceph-mon[33046]: pgmap v284: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:30:24.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:23 smithi153 ceph-mon[32746]: pgmap v285: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:30:24.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:23 smithi146 ceph-mon[27805]: pgmap v285: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:30:24.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:23 smithi177 ceph-mon[33046]: pgmap v285: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:30:26.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:26 smithi153 ceph-mon[32746]: pgmap v286: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:30:26.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:26 smithi146 ceph-mon[27805]: pgmap v286: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:30:26.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:26 smithi177 ceph-mon[33046]: pgmap v286: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:30:28.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:28 smithi153 ceph-mon[32746]: pgmap v287: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:30:28.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:28 smithi146 ceph-mon[27805]: pgmap v287: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:30:28.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:28 smithi177 ceph-mon[33046]: pgmap v287: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:30:30.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:30 smithi153 ceph-mon[32746]: pgmap v288: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:30.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:30 smithi146 ceph-mon[27805]: pgmap v288: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:30.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:30 smithi177 ceph-mon[33046]: pgmap v288: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:32.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:32 smithi153 ceph-mon[32746]: pgmap v289: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:32.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:32 smithi146 ceph-mon[27805]: pgmap v289: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:32 smithi177 ceph-mon[33046]: pgmap v289: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:33.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:33 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:30:33.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:33 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:30:33.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:33 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:30:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:34 smithi153 ceph-mon[32746]: pgmap v290: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:34 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:30:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:34 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:30:34.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:34 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:30:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:34 smithi146 ceph-mon[27805]: pgmap v290: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:34 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:30:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:34 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:30:34.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:34 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:30:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:34 smithi177 ceph-mon[33046]: pgmap v290: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:34 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:30:34.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:34 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:30:34.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:34 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:30:36.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:36 smithi153 ceph-mon[32746]: pgmap v291: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:36.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:36 smithi146 ceph-mon[27805]: pgmap v291: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:36 smithi177 ceph-mon[33046]: pgmap v291: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:38.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:38 smithi153 ceph-mon[32746]: pgmap v292: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:38.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:38 smithi146 ceph-mon[27805]: pgmap v292: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:38.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:38 smithi177 ceph-mon[33046]: pgmap v292: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:40.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:40 smithi153 ceph-mon[32746]: pgmap v293: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:40.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:40 smithi146 ceph-mon[27805]: pgmap v293: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:40.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:40 smithi177 ceph-mon[33046]: pgmap v293: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:41.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:30:41.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:30:41.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.923+0000 7f007d000640 1 -- 172.21.15.146:0/2176677945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 msgr2=0x7f00781153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.923+0000 7f007d000640 1 --2- 172.21.15.146:0/2176677945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 0x7f00781153b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0070009cd0 tx=0x7f007002a8f0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.924+0000 7f007d000640 1 -- 172.21.15.146:0/2176677945 shutdown_connections 2024-04-24T08:30:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.924+0000 7f007d000640 1 --2- 172.21.15.146:0/2176677945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 0x7f0078108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.924+0000 7f007d000640 1 --2- 172.21.15.146:0/2176677945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 0x7f00781153b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.924+0000 7f007d000640 1 --2- 172.21.15.146:0/2176677945 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f007810d500 0x7f007810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.924+0000 7f007d000640 1 -- 172.21.15.146:0/2176677945 >> 172.21.15.146:0/2176677945 conn(0x7f0078077e20 msgr2=0x7f0078106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.924+0000 7f007d000640 1 -- 172.21.15.146:0/2176677945 shutdown_connections 2024-04-24T08:30:41.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.924+0000 7f007d000640 1 -- 172.21.15.146:0/2176677945 wait complete. 2024-04-24T08:30:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 Processor -- start 2024-04-24T08:30:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 -- start start 2024-04-24T08:30:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 0x7f00781a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 0x7f00781a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f007810d500 0x7f007819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f00781188d0 con 0x7f0078108f40 2024-04-24T08:30:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0078118750 con 0x7f007810d500 2024-04-24T08:30:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f007d000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0078118a50 con 0x7f0078108580 2024-04-24T08:30:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f0077400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 0x7f00781a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f006fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 0x7f00781a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f0077400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 0x7f00781a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47576/0 (socket says 172.21.15.146:47576) 2024-04-24T08:30:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.925+0000 7f0077400640 1 -- 172.21.15.146:0/3462748632 learned_addr learned my addr 172.21.15.146:0/3462748632 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:41.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077e00640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f007810d500 0x7f007819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077400640 1 -- 172.21.15.146:0/3462748632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f007810d500 msgr2=0x7f007819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077400640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f007810d500 0x7f007819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077400640 1 -- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 msgr2=0x7f00781a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077400640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 0x7f00781a4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077400640 1 -- 172.21.15.146:0/3462748632 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007819f290 con 0x7f0078108580 2024-04-24T08:30:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f006fe00640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 0x7f00781a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:41.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077e00640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f007810d500 0x7f007819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0077400640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 0x7f00781a4910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f006800e9c0 tx=0x7f006800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0074c00640 1 -- 172.21.15.146:0/3462748632 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006800cde0 con 0x7f0078108580 2024-04-24T08:30:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007819f580 con 0x7f0078108580 2024-04-24T08:30:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0074c00640 1 -- 172.21.15.146:0/3462748632 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0068010910 con 0x7f0078108580 2024-04-24T08:30:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.926+0000 7f0074c00640 1 -- 172.21.15.146:0/3462748632 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068018a10 con 0x7f0078108580 2024-04-24T08:30:41.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.927+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00781ab7d0 con 0x7f0078108580 2024-04-24T08:30:41.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 Processor -- start 2024-04-24T08:30:41.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 -- start start 2024-04-24T08:30:41.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0074c00640 1 -- 172.21.15.146:0/3462748632 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f006801f020 con 0x7f0078108580 2024-04-24T08:30:41.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c4fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c5a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c5ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55afc2bb51f0 con 0x55afc2c50350 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55afc2b5b350 con 0x55afc2c5aa20 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0081420740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55afc2bb11b0 con 0x55afc2c4fa30 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c4fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0077400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c5a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c4fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47588/0 (socket says 172.21.15.146:47588) 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.930+0000 7f0077e00640 1 -- 172.21.15.146:0/534152996 learned_addr learned my addr 172.21.15.146:0/534152996 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f006fe00640 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c5ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f0077e00640 1 -- 172.21.15.146:0/534152996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 msgr2=0x55afc2c5ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f0077e00640 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c5ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f0077e00640 1 -- 172.21.15.146:0/534152996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 msgr2=0x55afc2c5a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f0077e00640 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c5a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f0077e00640 1 -- 172.21.15.146:0/534152996 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afc2baff50 con 0x55afc2c4fa30 2024-04-24T08:30:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f0077e00640 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c4fe10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0070005f70 tx=0x7f0070009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32d6f2cbb06108c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.931+0000 7f006cc00640 1 -- 172.21.15.146:0/534152996 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070030720 con 0x55afc2c4fa30 2024-04-24T08:30:41.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f006cc00640 1 -- 172.21.15.146:0/534152996 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0070036040 con 0x55afc2c4fa30 2024-04-24T08:30:41.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f006cc00640 1 -- 172.21.15.146:0/534152996 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070004500 con 0x55afc2c4fa30 2024-04-24T08:30:41.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0074c00640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f005c0a3d20 0x7f005c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0074c00640 1 -- 172.21.15.146:0/3462748632 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f00680d0ac0 con 0x7f0078108580 2024-04-24T08:30:41.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0077400640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f005c0a3d20 0x7f005c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0081420740 1 -- 172.21.15.146:0/534152996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 msgr2=0x55afc2c4fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0081420740 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c4fe10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0070005f70 tx=0x7f0070009a20 comp rx=0 tx=0).stop 2024-04-24T08:30:41.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0081420740 1 -- 172.21.15.146:0/534152996 shutdown_connections 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0081420740 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c5ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0081420740 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c5a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0081420740 1 --2- 172.21.15.146:0/534152996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c4fe10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.932+0000 7f0081420740 1 -- 172.21.15.146:0/534152996 >> 172.21.15.146:0/534152996 conn(0x55afc2c4bd50 msgr2=0x55afc2c4c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.933+0000 7f0081420740 1 -- 172.21.15.146:0/534152996 shutdown_connections 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.933+0000 7f0081420740 1 -- 172.21.15.146:0/534152996 wait complete. 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.933+0000 7f0077400640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f005c0a3d20 0x7f005c0a61e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55afc2c50ff0 tx=0x7f00680d4570 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:41.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.933+0000 7f0081420740 1 -- start start 2024-04-24T08:30:41.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.933+0000 7f0081420740 1 Processor -- start 2024-04-24T08:30:41.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0081420740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c4fa30 0x55afc2c579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0081420740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c57f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0081420740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c5aa20 0x55afc2c58450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0081420740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55afc2c5d7b0 con 0x55afc2c4fa30 2024-04-24T08:30:41.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0081420740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55afc2b6e690 con 0x55afc2c50350 2024-04-24T08:30:41.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0081420740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55afc2c5d930 con 0x55afc2c5aa20 2024-04-24T08:30:41.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0081420740 10 client.0 _ll_register_callbacks cb 0x55afc2baa650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:41.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0077e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c4fa30 0x55afc2c579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f006fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c57f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0077e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c4fa30 0x55afc2c579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43648/0 (socket says 172.21.15.146:43648) 2024-04-24T08:30:41.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0077e00640 1 -- 172.21.15.146:0/2288246444 learned_addr learned my addr 172.21.15.146:0/2288246444 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:41.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0077e00640 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c5aa20 0x55afc2c58450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f006fe00640 1 -- 172.21.15.146:0/2288246444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c5aa20 msgr2=0x55afc2c58450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f006fe00640 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c5aa20 0x55afc2c58450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f006fe00640 1 -- 172.21.15.146:0/2288246444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c4fa30 msgr2=0x55afc2c579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f006fe00640 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c4fa30 0x55afc2c579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f006fe00640 1 -- 172.21.15.146:0/2288246444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afc2b6d920 con 0x55afc2c50350 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.934+0000 7f0077e00640 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c4fa30 0x55afc2c579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f006fe00640 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c57f10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0078107f40 tx=0x7f006000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b5e5fe5d46c08 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f0065600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:41.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f0065600640 1 -- 172.21.15.146:0/2288246444 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060009e30 con 0x55afc2c50350 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f0065600640 1 -- 172.21.15.146:0/2288246444 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f0060012110 con 0x55afc2c50350 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0070036040 con 0x55afc2c50350 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f0065600640 1 -- 172.21.15.146:0/2288246444 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0060004050 con 0x55afc2c50350 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f0081420740 10 client.27236 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.935+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55afc29917d0 con 0x55afc2c50350 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0065600640 1 -- 172.21.15.146:0/2288246444 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0070036040 con 0x55afc2c50350 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0065600640 1 -- 172.21.15.146:0/2288246444 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0060010430 con 0x55afc2c50350 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0081420740 10 client.27236 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:41.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0081420740 10 client.27236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:41.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0081420740 20 client.27236 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:41.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0081420740 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2c6adf0 0x55afc2c6d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0081420740 4 client.27236 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:41.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afc2bc2c10 con 0x55afc2c6adf0 2024-04-24T08:30:41.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.936+0000 7f0077e00640 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2c6adf0 0x55afc2c6d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.937+0000 7f0077e00640 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2c6adf0 0x55afc2c6d220 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=127b98fd170404be server_cookie=53de2f53f07306e8 in_seq=0 out_seq=0 2024-04-24T08:30:41.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.937+0000 7f0065600640 10 client.27236 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0065600640 1 -- 172.21.15.146:0/2288246444 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55afc2bc2c10 con 0x55afc2c6adf0 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0065600640 10 client.27236 handle_command_reply: tid=0 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 client.27236 shutdown 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 msgr2=0x55afc2c57f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c57f10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0078107f40 tx=0x7f006000f790 comp rx=0 tx=0).stop 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 shutdown_connections 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2c6adf0 0x55afc2c6d220 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.951 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c5aa20 0x55afc2c58450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0065600640 0 client.27236 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:41.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c57f10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 --2- 172.21.15.146:0/2288246444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c4fa30 0x55afc2c579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 >> 172.21.15.146:0/2288246444 conn(0x55afc2c4bd50 msgr2=0x55afc2c4afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:41.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.938+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 shutdown_connections 2024-04-24T08:30:41.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.939+0000 7f0081420740 1 -- 172.21.15.146:0/2288246444 wait complete. 2024-04-24T08:30:41.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.939+0000 7f0081420740 20 client.27236 trim_cache size 0 max 16384 2024-04-24T08:30:41.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.957+0000 7f0081420740 1 Processor -- start 2024-04-24T08:30:41.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.957+0000 7f0081420740 1 -- start start 2024-04-24T08:30:41.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0081420740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c58220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0081420740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c52190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0081420740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c5aa20 0x55afc2c72b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0081420740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f00781b6aa0 con 0x55afc2c5aa20 2024-04-24T08:30:41.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0081420740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00781b4370 con 0x55afc2c50350 2024-04-24T08:30:41.960 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0081420740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f00781b21e0 con 0x55afc2c4fa30 2024-04-24T08:30:41.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c58220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c58220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47612/0 (socket says 172.21.15.146:47612) 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 -- 172.21.15.146:0/673113912 learned_addr learned my addr 172.21.15.146:0/673113912 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c5aa20 0x55afc2c72b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f006fe00640 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c52190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 -- 172.21.15.146:0/673113912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 msgr2=0x55afc2c52190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c52190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 -- 172.21.15.146:0/673113912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c5aa20 msgr2=0x55afc2c72b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c5aa20 0x55afc2c72b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.958+0000 7f0077e00640 1 -- 172.21.15.146:0/673113912 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afc28c73f0 con 0x55afc2c4fa30 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.959+0000 7f0077e00640 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c58220 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f00700040b0 tx=0x7f0070047040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce98ee557a2e099f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.959+0000 7f0067e00640 1 -- 172.21.15.146:0/673113912 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070003b70 con 0x55afc2c4fa30 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.959+0000 7f0067e00640 1 -- 172.21.15.146:0/673113912 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0070036070 con 0x55afc2c4fa30 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.959+0000 7f0067e00640 1 -- 172.21.15.146:0/673113912 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070031440 con 0x55afc2c4fa30 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 -- 172.21.15.146:0/673113912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 msgr2=0x55afc2c58220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c58220 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f00700040b0 tx=0x7f0070047040 comp rx=0 tx=0).stop 2024-04-24T08:30:41.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 -- 172.21.15.146:0/673113912 shutdown_connections 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c5aa20 0x55afc2c72b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c50350 0x55afc2c52190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 --2- 172.21.15.146:0/673113912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c58220 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 -- 172.21.15.146:0/673113912 >> 172.21.15.146:0/673113912 conn(0x55afc2c4bd50 msgr2=0x55afc2c4a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 -- 172.21.15.146:0/673113912 shutdown_connections 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 -- 172.21.15.146:0/673113912 wait complete. 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.960+0000 7f0081420740 1 -- start start 2024-04-24T08:30:41.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 1 Processor -- start 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c546f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c54c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c55170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f00781b2830 con 0x55afc2c50350 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f00781b3ca0 con 0x55afc2c5aa20 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f00781b2f00 con 0x55afc2c4fa30 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0081420740 10 client.0 _ll_register_callbacks cb 0x7f0060009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:41.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0077e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47618/0 (socket says 172.21.15.146:47618) 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f006fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c54c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0077e00640 1 -- 172.21.15.146:0/3950314636 learned_addr learned my addr 172.21.15.146:0/3950314636 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.961+0000 7f0077e00640 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c55170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f006fe00640 1 -- 172.21.15.146:0/3950314636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 msgr2=0x55afc2c546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f006fe00640 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f006fe00640 1 -- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 msgr2=0x55afc2c55170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:41.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f006fe00640 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c55170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f006fe00640 1 -- 172.21.15.146:0/3950314636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afc28870f0 con 0x55afc2c50350 2024-04-24T08:30:41.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f0077e00640 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:41.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f006fe00640 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c54c30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0060010a40 tx=0x7f0060010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a26264794b8f0c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:41.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f0065600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f0065600640 1 -- 172.21.15.146:0/3950314636 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060032060 con 0x55afc2c50350 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0070036070 con 0x55afc2c50350 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.962+0000 7f0065600640 1 -- 172.21.15.146:0/3950314636 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f0060010ac0 con 0x55afc2c50350 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0065600640 1 -- 172.21.15.146:0/3950314636 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0060029040 con 0x55afc2c50350 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 10 client.17097 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55afc2b7a810 con 0x55afc2c50350 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0065600640 1 -- 172.21.15.146:0/3950314636 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0070036070 con 0x55afc2c50350 2024-04-24T08:30:41.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0065600640 1 -- 172.21.15.146:0/3950314636 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f006002c070 con 0x55afc2c50350 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 10 client.17097 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 20 client.17097 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2cd1b30 0x55afc2cd3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 4 client.17097 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.963+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55afc2bbca80 con 0x55afc2cd1b30 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.964+0000 7f0077e00640 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2cd1b30 0x55afc2cd3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:41.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.964+0000 7f0077e00640 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2cd1b30 0x55afc2cd3f50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1907c1f1347cffa3 server_cookie=34fce44db9eea233 in_seq=0 out_seq=0 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.964+0000 7f0065600640 10 client.17097 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.964+0000 7f0065600640 1 -- 172.21.15.146:0/3950314636 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55afc2bbca80 con 0x55afc2cd1b30 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.964+0000 7f0065600640 10 client.17097 handle_command_reply: tid=0 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.964+0000 7f0081420740 1 client.17097 shutdown 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 msgr2=0x55afc2c54c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c54c30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0060010a40 tx=0x7f0060010430 comp rx=0 tx=0).stop 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 shutdown_connections 2024-04-24T08:30:41.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55afc2cd1b30 0x55afc2cd3f50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55afc2c5aa20 0x55afc2c55170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55afc2c50350 0x55afc2c54c30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 --2- 172.21.15.146:0/3950314636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55afc2c4fa30 0x55afc2c546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0065600640 0 client.17097 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 >> 172.21.15.146:0/3950314636 conn(0x55afc2c4bd50 msgr2=0x55afc2c4afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 shutdown_connections 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 1 -- 172.21.15.146:0/3950314636 wait complete. 2024-04-24T08:30:41.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.965+0000 7f0081420740 20 client.17097 trim_cache size 0 max 16384 2024-04-24T08:30:41.975 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:30:41.975 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:30:41.975 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:30:41.975 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:30:41.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f005c0a3d20 msgr2=0x7f005c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f005c0a3d20 0x7f005c0a61e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55afc2c50ff0 tx=0x7f00680d4570 comp rx=0 tx=0).stop 2024-04-24T08:30:41.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 msgr2=0x7f00781a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:41.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 0x7f00781a4910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f006800e9c0 tx=0x7f006800ee80 comp rx=0 tx=0).stop 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 shutdown_connections 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f005c0a3d20 0x7f005c0a61e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f007810d500 0x7f007819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0078108f40 0x7f00781a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 --2- 172.21.15.146:0/3462748632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0078108580 0x7f00781a4910 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 >> 172.21.15.146:0/3462748632 conn(0x7f0078077e20 msgr2=0x7f0078078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 shutdown_connections 2024-04-24T08:30:41.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:41.966+0000 7f007d000640 1 -- 172.21.15.146:0/3462748632 wait complete. 2024-04-24T08:30:41.986 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:30:41.986 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:30:42.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 -- 172.21.15.146:0/1546434966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103b80 msgr2=0x7f188810c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 --2- 172.21.15.146:0/1546434966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103b80 0x7f188810c060 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1874009ca0 tx=0x7f187402a950 comp rx=0 tx=0).stop 2024-04-24T08:30:42.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 -- 172.21.15.146:0/1546434966 shutdown_connections 2024-04-24T08:30:42.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 --2- 172.21.15.146:0/1546434966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f188810c630 0x7f188810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 --2- 172.21.15.146:0/1546434966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103b80 0x7f188810c060 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 --2- 172.21.15.146:0/1546434966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1888103260 0x7f1888103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 -- 172.21.15.146:0/1546434966 >> 172.21.15.146:0/1546434966 conn(0x7f1888077280 msgr2=0x7f1888077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 -- 172.21.15.146:0/1546434966 shutdown_connections 2024-04-24T08:30:42.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.185+0000 7f1890400640 1 -- 172.21.15.146:0/1546434966 wait complete. 2024-04-24T08:30:42.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.186+0000 7f1890400640 1 Processor -- start 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.186+0000 7f1890400640 1 -- start start 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f1890400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103260 0x7f18881a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f1890400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1888103b80 0x7f18881a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f1890400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f188810c630 0x7f188819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f1890400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f18881123a0 con 0x7f1888103260 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f1890400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1888112220 con 0x7f1888103b80 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f1890400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1888112520 con 0x7f188810c630 2024-04-24T08:30:42.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103260 0x7f18881a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103260 0x7f18881a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43662/0 (socket says 172.21.15.146:43662) 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 -- 172.21.15.146:0/342782950 learned_addr learned my addr 172.21.15.146:0/342782950 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188d000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1888103b80 0x7f18881a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188e400640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f188810c630 0x7f188819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 -- 172.21.15.146:0/342782950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f188810c630 msgr2=0x7f188819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f188810c630 0x7f188819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 -- 172.21.15.146:0/342782950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1888103b80 msgr2=0x7f18881a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1888103b80 0x7f18881a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.187+0000 7f188da00640 1 -- 172.21.15.146:0/342782950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188819b120 con 0x7f1888103260 2024-04-24T08:30:42.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.188+0000 7f188da00640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103260 0x7f18881a07f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f188000ea60 tx=0x7f188000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.188+0000 7f1886000640 1 -- 172.21.15.146:0/342782950 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188000cde0 con 0x7f1888103260 2024-04-24T08:30:42.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.188+0000 7f1886000640 1 -- 172.21.15.146:0/342782950 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18800035f0 con 0x7f1888103260 2024-04-24T08:30:42.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.188+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188819b410 con 0x7f1888103260 2024-04-24T08:30:42.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.188+0000 7f1886000640 1 -- 172.21.15.146:0/342782950 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1880010670 con 0x7f1888103260 2024-04-24T08:30:42.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.188+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18881a76b0 con 0x7f1888103260 2024-04-24T08:30:42.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.189+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f188806ae80 con 0x7f1888103260 2024-04-24T08:30:42.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.191+0000 7f1886000640 1 -- 172.21.15.146:0/342782950 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f18800037a0 con 0x7f1888103260 2024-04-24T08:30:42.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.193+0000 7f1886000640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f186c0a3da0 0x7f186c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.193+0000 7f1886000640 1 -- 172.21.15.146:0/342782950 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f18800d0c40 con 0x7f1888103260 2024-04-24T08:30:42.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.193+0000 7f188d000640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f186c0a3da0 0x7f186c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.194+0000 7f188d000640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f186c0a3da0 0x7f186c0a6260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f188819c190 tx=0x7f1874004a60 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.196 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.195+0000 7f1886000640 1 -- 172.21.15.146:0/342782950 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f188009d300 con 0x7f1888103260 2024-04-24T08:30:42.434 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.433+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1888069270 con 0x7f1888103260 2024-04-24T08:30:42.435 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.434+0000 7f1886000640 1 -- 172.21.15.146:0/342782950 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f188009cc00 con 0x7f1888103260 2024-04-24T08:30:42.436 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:30:42.436 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:30:42.437 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:30:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f186c0a3da0 msgr2=0x7f186c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f186c0a3da0 0x7f186c0a6260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f188819c190 tx=0x7f1874004a60 comp rx=0 tx=0).stop 2024-04-24T08:30:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103260 msgr2=0x7f18881a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103260 0x7f18881a07f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f188000ea60 tx=0x7f188000ef20 comp rx=0 tx=0).stop 2024-04-24T08:30:42.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 shutdown_connections 2024-04-24T08:30:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f186c0a3da0 0x7f186c0a6260 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f188810c630 0x7f188819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1888103b80 0x7f18881a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 --2- 172.21.15.146:0/342782950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1888103260 0x7f18881a07f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 >> 172.21.15.146:0/342782950 conn(0x7f1888077280 msgr2=0x7f1888100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 shutdown_connections 2024-04-24T08:30:42.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.438+0000 7f1890400640 1 -- 172.21.15.146:0/342782950 wait complete. 2024-04-24T08:30:42.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:30:42.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 -- 172.21.15.146:0/2888349629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4254108610 msgr2=0x7f4254108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 --2- 172.21.15.146:0/2888349629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4254108610 0x7f4254108a90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4240009930 tx=0x7f424002a440 comp rx=0 tx=0).stop 2024-04-24T08:30:42.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 -- 172.21.15.146:0/2888349629 shutdown_connections 2024-04-24T08:30:42.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 --2- 172.21.15.146:0/2888349629 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4254108fd0 0x7f4254115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 --2- 172.21.15.146:0/2888349629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4254108610 0x7f4254108a90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 --2- 172.21.15.146:0/2888349629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f425410e660 0x7f425410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 -- 172.21.15.146:0/2888349629 >> 172.21.15.146:0/2888349629 conn(0x7f4254076540 msgr2=0x7f4254078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 -- 172.21.15.146:0/2888349629 shutdown_connections 2024-04-24T08:30:42.656 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.653+0000 7f425a800640 1 -- 172.21.15.146:0/2888349629 wait complete. 2024-04-24T08:30:42.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.654+0000 7f425a800640 1 Processor -- start 2024-04-24T08:30:42.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.654+0000 7f425a800640 1 -- start start 2024-04-24T08:30:42.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f425a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4254108610 0x7f425419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f425a800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4254108fd0 0x7f425419f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f425a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425410e660 0x7f42541a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f425a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42541189f0 con 0x7f425410e660 2024-04-24T08:30:42.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f425a800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4254118870 con 0x7f4254108fd0 2024-04-24T08:30:42.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f425a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4254118b70 con 0x7f4254108610 2024-04-24T08:30:42.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425410e660 0x7f42541a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4254108610 0x7f425419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4252a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4254108fd0 0x7f425419f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425410e660 0x7f42541a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43680/0 (socket says 172.21.15.146:43680) 2024-04-24T08:30:42.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4254108610 0x7f425419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47664/0 (socket says 172.21.15.146:47664) 2024-04-24T08:30:42.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 -- 172.21.15.146:0/4172055074 learned_addr learned my addr 172.21.15.146:0/4172055074 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 -- 172.21.15.146:0/4172055074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4254108610 msgr2=0x7f425419ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4254108610 0x7f425419ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 -- 172.21.15.146:0/4172055074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4254108fd0 msgr2=0x7f425419f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4254108fd0 0x7f425419f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.655+0000 7f4253e00640 1 -- 172.21.15.146:0/4172055074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42541a3e80 con 0x7f425410e660 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.656+0000 7f4253e00640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425410e660 0x7f42541a3750 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f424c00eb10 tx=0x7f424c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.656+0000 7f424be00640 1 -- 172.21.15.146:0/4172055074 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c004b20 con 0x7f425410e660 2024-04-24T08:30:42.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.656+0000 7f424be00640 1 -- 172.21.15.146:0/4172055074 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f424c004cc0 con 0x7f425410e660 2024-04-24T08:30:42.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.656+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42541a4170 con 0x7f425410e660 2024-04-24T08:30:42.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.656+0000 7f424be00640 1 -- 172.21.15.146:0/4172055074 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c010760 con 0x7f425410e660 2024-04-24T08:30:42.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.656+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42541aba00 con 0x7f425410e660 2024-04-24T08:30:42.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.659+0000 7f424be00640 1 -- 172.21.15.146:0/4172055074 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f424c005020 con 0x7f425410e660 2024-04-24T08:30:42.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.659+0000 7f425d792740 1 Processor -- start 2024-04-24T08:30:42.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f425d792740 1 -- start start 2024-04-24T08:30:42.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f425d792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc780950 0x555bcc780d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f425d792740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc781270 0x555bcc78b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f425d792740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc78b940 0x555bcc78dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f425d792740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555bcc6e61d0 con 0x555bcc78b940 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f425d792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555bcc68c450 con 0x555bcc780950 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f425d792740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555bcc6e2190 con 0x555bcc781270 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f4253400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc781270 0x555bcc78b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f4253400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc781270 0x555bcc78b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47674/0 (socket says 172.21.15.146:47674) 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f4253400640 1 -- 172.21.15.146:0/3142368412 learned_addr learned my addr 172.21.15.146:0/3142368412 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f4252a00640 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc780950 0x555bcc780d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.660+0000 7f4252a00640 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc78b940 0x555bcc78dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4253400640 1 -- 172.21.15.146:0/3142368412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc780950 msgr2=0x555bcc780d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4253400640 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc780950 0x555bcc780d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4253400640 1 -- 172.21.15.146:0/3142368412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc78b940 msgr2=0x555bcc78dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4253400640 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc78b940 0x555bcc78dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4253400640 1 -- 172.21.15.146:0/3142368412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bcc749310 con 0x555bcc781270 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4253400640 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc781270 0x555bcc78b400 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4244009ca0 tx=0x7f4244011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70ba7b5b830936d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f424be00640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42380a3d20 0x7f42380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f424be00640 1 -- 172.21.15.146:0/4172055074 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f424c0d0df0 con 0x7f425410e660 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4248c00640 1 -- 172.21.15.146:0/3142368412 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244004220 con 0x555bcc781270 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4248c00640 1 -- 172.21.15.146:0/3142368412 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42440043c0 con 0x555bcc781270 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4248c00640 1 -- 172.21.15.146:0/3142368412 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244019910 con 0x555bcc781270 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.661+0000 7f4252a00640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42380a3d20 0x7f42380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f4252a00640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42380a3d20 0x7f42380a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4240006010 tx=0x7f424002a440 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 -- 172.21.15.146:0/3142368412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc781270 msgr2=0x555bcc78b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc781270 0x555bcc78b400 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4244009ca0 tx=0x7f4244011810 comp rx=0 tx=0).stop 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 -- 172.21.15.146:0/3142368412 shutdown_connections 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc781270 0x555bcc78b400 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc78b940 0x555bcc78dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 --2- 172.21.15.146:0/3142368412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc780950 0x555bcc780d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 -- 172.21.15.146:0/3142368412 >> 172.21.15.146:0/3142368412 conn(0x555bcc77cc70 msgr2=0x555bcc77d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 -- 172.21.15.146:0/3142368412 shutdown_connections 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.662+0000 7f425d792740 1 -- 172.21.15.146:0/3142368412 wait complete. 2024-04-24T08:30:42.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 -- start start 2024-04-24T08:30:42.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 Processor -- start 2024-04-24T08:30:42.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc78b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7885f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc788c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x555bcc78e7d0 con 0x555bcc780950 2024-04-24T08:30:42.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555bcc69f810 con 0x555bcc781270 2024-04-24T08:30:42.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.663+0000 7f425d792740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555bcc78e950 con 0x555bcc78b940 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f425d792740 10 client.0 _ll_register_callbacks cb 0x555bcc66d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc78b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc78b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43706/0 (socket says 172.21.15.146:43706) 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253400640 1 -- 172.21.15.146:0/2299257236 learned_addr learned my addr 172.21.15.146:0/2299257236 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253e00640 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc788c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4252a00640 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7885f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253e00640 1 -- 172.21.15.146:0/2299257236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 msgr2=0x555bcc7885f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253e00640 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7885f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253e00640 1 -- 172.21.15.146:0/2299257236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 msgr2=0x555bcc78b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253e00640 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc78b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253e00640 1 -- 172.21.15.146:0/2299257236 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bcc6b03f0 con 0x555bcc78b940 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f4253e00640 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc788c00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f424c0ce4d0 tx=0x7f424c09c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f15a189edecae34 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.664+0000 7f423d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f423d600640 1 -- 172.21.15.146:0/2299257236 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c0a5700 con 0x555bcc78b940 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f423d600640 1 -- 172.21.15.146:0/2299257236 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f424c005310 con 0x555bcc78b940 2024-04-24T08:30:42.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42440043c0 con 0x555bcc78b940 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f423d600640 1 -- 172.21.15.146:0/2299257236 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f424c0a10d0 con 0x555bcc78b940 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f425d792740 10 client.27265 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555bcc3f83f0 con 0x555bcc78b940 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f423d600640 1 -- 172.21.15.146:0/2299257236 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42440043c0 con 0x555bcc78b940 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f423d600640 1 -- 172.21.15.146:0/2299257236 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f424c0bd100 con 0x555bcc78b940 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f425d792740 10 client.27265 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.665+0000 7f425d792740 10 client.27265 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.666+0000 7f425d792740 20 client.27265 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:42.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.666+0000 7f425d792740 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc79ba90 0x555bcc79de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.666+0000 7f425d792740 4 client.27265 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:42.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.666+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bcc6f3bf0 con 0x555bcc79ba90 2024-04-24T08:30:42.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.666+0000 7f4253400640 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc79ba90 0x555bcc79de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.666+0000 7f4253400640 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc79ba90 0x555bcc79de80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3a141d3514eb506 server_cookie=c93d19236f89cdac in_seq=0 out_seq=0 2024-04-24T08:30:42.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.666+0000 7f423d600640 10 client.27265 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:42.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f423d600640 1 -- 172.21.15.146:0/2299257236 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555bcc6f3bf0 con 0x555bcc79ba90 2024-04-24T08:30:42.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f423d600640 10 client.27265 handle_command_reply: tid=0 2024-04-24T08:30:42.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 client.27265 shutdown 2024-04-24T08:30:42.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 msgr2=0x555bcc788c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc788c00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f424c0ce4d0 tx=0x7f424c09c740 comp rx=0 tx=0).stop 2024-04-24T08:30:42.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 shutdown_connections 2024-04-24T08:30:42.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc79ba90 0x555bcc79de80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc788c00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7885f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 --2- 172.21.15.146:0/2299257236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc78b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f423d600640 0 client.27265 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:42.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.667+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 >> 172.21.15.146:0/2299257236 conn(0x555bcc77cc70 msgr2=0x555bcc782a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.668+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 shutdown_connections 2024-04-24T08:30:42.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.668+0000 7f425d792740 1 -- 172.21.15.146:0/2299257236 wait complete. 2024-04-24T08:30:42.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.668+0000 7f425d792740 20 client.27265 trim_cache size 0 max 16384 2024-04-24T08:30:42.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 Processor -- start 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 -- start start 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc797690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7875d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc78ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42541b6c30 con 0x555bcc780950 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42541b4500 con 0x555bcc781270 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f425d792740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42541b2370 con 0x555bcc78b940 2024-04-24T08:30:42.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f4253400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc797690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f4253400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc797690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43714/0 (socket says 172.21.15.146:43714) 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f4253400640 1 -- 172.21.15.146:0/2416688516 learned_addr learned my addr 172.21.15.146:0/2416688516 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f4253e00640 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc78ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.682+0000 7f4252a00640 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7875d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f4253400640 1 -- 172.21.15.146:0/2416688516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 msgr2=0x555bcc78ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f4253400640 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc78ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f4253400640 1 -- 172.21.15.146:0/2416688516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 msgr2=0x555bcc7875d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f4253400640 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7875d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f4253400640 1 -- 172.21.15.146:0/2416688516 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bcc4c27d0 con 0x555bcc780950 2024-04-24T08:30:42.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f4253400640 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc797690 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4244009a30 tx=0x7f4244011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daea8f95cf4ab467 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f423fe00640 1 -- 172.21.15.146:0/2416688516 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244031070 con 0x555bcc780950 2024-04-24T08:30:42.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f423fe00640 1 -- 172.21.15.146:0/2416688516 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f424402b840 con 0x555bcc780950 2024-04-24T08:30:42.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.683+0000 7f423fe00640 1 -- 172.21.15.146:0/2416688516 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42440035f0 con 0x555bcc780950 2024-04-24T08:30:42.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 -- 172.21.15.146:0/2416688516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 msgr2=0x555bcc797690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc797690 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4244009a30 tx=0x7f4244011810 comp rx=0 tx=0).stop 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 -- 172.21.15.146:0/2416688516 shutdown_connections 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc78b940 0x555bcc78ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc781270 0x555bcc7875d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 --2- 172.21.15.146:0/2416688516 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc780950 0x555bcc797690 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 -- 172.21.15.146:0/2416688516 >> 172.21.15.146:0/2416688516 conn(0x555bcc77cc70 msgr2=0x555bcc786ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 -- 172.21.15.146:0/2416688516 shutdown_connections 2024-04-24T08:30:42.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 -- 172.21.15.146:0/2416688516 wait complete. 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 -- start start 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.684+0000 7f425d792740 1 Processor -- start 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f425d792740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc780950 0x555bcc78c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f425d792740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc781270 0x555bcc78c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f425d792740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc78b940 0x555bcc792ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f425d792740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f42541b29c0 con 0x555bcc781270 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f425d792740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f42541b3e30 con 0x555bcc78b940 2024-04-24T08:30:42.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f425d792740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42541b3090 con 0x555bcc780950 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f425d792740 10 client.0 _ll_register_callbacks cb 0x7f424c09cb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc781270 0x555bcc78c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc781270 0x555bcc78c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43724/0 (socket says 172.21.15.146:43724) 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 -- 172.21.15.146:0/3195411566 learned_addr learned my addr 172.21.15.146:0/3195411566 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4253e00640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc78b940 0x555bcc792ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4253400640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc780950 0x555bcc78c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 -- 172.21.15.146:0/3195411566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc780950 msgr2=0x555bcc78c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc780950 0x555bcc78c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 -- 172.21.15.146:0/3195411566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc78b940 msgr2=0x555bcc792ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc78b940 0x555bcc792ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.685+0000 7f4252a00640 1 -- 172.21.15.146:0/3195411566 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bcc7930e0 con 0x555bcc781270 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f4253400640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc780950 0x555bcc78c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f4252a00640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc781270 0x555bcc78c880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4240009930 tx=0x7f4240000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fc41cd729a44c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f423d600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f423d600640 1 -- 172.21.15.146:0/3195411566 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240057060 con 0x555bcc781270 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f424402b840 con 0x555bcc781270 2024-04-24T08:30:42.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f423d600640 1 -- 172.21.15.146:0/3195411566 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f4240004b90 con 0x555bcc781270 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f423d600640 1 -- 172.21.15.146:0/3195411566 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4240004d90 con 0x555bcc781270 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f425d792740 10 client.17121 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.686+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555bcc794460 con 0x555bcc781270 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f423d600640 1 -- 172.21.15.146:0/3195411566 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f424402b840 con 0x555bcc781270 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f423d600640 1 -- 172.21.15.146:0/3195411566 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4240051070 con 0x555bcc781270 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f425d792740 10 client.17121 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f425d792740 10 client.17121 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f425d792740 20 client.17121 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:42.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f425d792740 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc802460 0x555bcc804880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f425d792740 4 client.17121 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555bcc7841f0 con 0x555bcc802460 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.687+0000 7f4253400640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc802460 0x555bcc804880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f4253400640 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc802460 0x555bcc804880 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5686ed0d8e4fcdc1 server_cookie=e9c3cb4943a2e005 in_seq=0 out_seq=0 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f423d600640 10 client.17121 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f423d600640 1 -- 172.21.15.146:0/3195411566 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555bcc7841f0 con 0x555bcc802460 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f423d600640 10 client.17121 handle_command_reply: tid=0 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 client.17121 shutdown 2024-04-24T08:30:42.696 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc781270 msgr2=0x555bcc78c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc781270 0x555bcc78c880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4240009930 tx=0x7f4240000bc0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 shutdown_connections 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x555bcc802460 0x555bcc804880 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555bcc78b940 0x555bcc792ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x555bcc781270 0x555bcc78c880 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f423d600640 0 client.17121 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 --2- 172.21.15.146:0/3195411566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bcc780950 0x555bcc78c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 >> 172.21.15.146:0/3195411566 conn(0x555bcc77cc70 msgr2=0x555bcc787750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 shutdown_connections 2024-04-24T08:30:42.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 1 -- 172.21.15.146:0/3195411566 wait complete. 2024-04-24T08:30:42.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.688+0000 7f425d792740 20 client.17121 trim_cache size 0 max 16384 2024-04-24T08:30:42.698 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:30:42.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42380a3d20 msgr2=0x7f42380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42380a3d20 0x7f42380a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4240006010 tx=0x7f424002a440 comp rx=0 tx=0).stop 2024-04-24T08:30:42.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425410e660 msgr2=0x7f42541a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425410e660 0x7f42541a3750 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f424c00eb10 tx=0x7f424c00c2d0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 shutdown_connections 2024-04-24T08:30:42.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f42380a3d20 0x7f42380a61e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f425410e660 0x7f42541a3750 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4254108fd0 0x7f425419f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 --2- 172.21.15.146:0/4172055074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4254108610 0x7f425419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 >> 172.21.15.146:0/4172055074 conn(0x7f4254076540 msgr2=0x7f4254077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 shutdown_connections 2024-04-24T08:30:42.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.689+0000 7f425a800640 1 -- 172.21.15.146:0/4172055074 wait complete. 2024-04-24T08:30:42.704 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:42 smithi146 ceph-mon[27805]: pgmap v294: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:42.704 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:42 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/342782950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:30:42.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:30:42.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:42 smithi153 ceph-mon[32746]: pgmap v294: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:42.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:42 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/342782950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:30:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:42 smithi177 ceph-mon[33046]: pgmap v294: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:42 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/342782950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:30:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.897+0000 7f2d56000640 1 -- 172.21.15.146:0/221611851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 msgr2=0x7f2d501153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.897+0000 7f2d56000640 1 --2- 172.21.15.146:0/221611851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501153b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009cd0 tx=0x7f2d4802a8f0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.898+0000 7f2d56000640 1 -- 172.21.15.146:0/221611851 shutdown_connections 2024-04-24T08:30:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.898+0000 7f2d56000640 1 --2- 172.21.15.146:0/221611851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 0x7f2d50108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.898+0000 7f2d56000640 1 --2- 172.21.15.146:0/221611851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501153b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.898+0000 7f2d56000640 1 --2- 172.21.15.146:0/221611851 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5010d500 0x7f2d5010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.898+0000 7f2d56000640 1 -- 172.21.15.146:0/221611851 >> 172.21.15.146:0/221611851 conn(0x7f2d50077e20 msgr2=0x7f2d50106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.898+0000 7f2d56000640 1 -- 172.21.15.146:0/221611851 shutdown_connections 2024-04-24T08:30:42.900 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.898+0000 7f2d56000640 1 -- 172.21.15.146:0/221611851 wait complete. 2024-04-24T08:30:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.899+0000 7f2d56000640 1 Processor -- start 2024-04-24T08:30:42.901 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.899+0000 7f2d56000640 1 -- start start 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.899+0000 7f2d56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 0x7f2d501a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.899+0000 7f2d56000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.899+0000 7f2d56000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5010d500 0x7f2d5019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d56000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d50118900 con 0x7f2d50108f40 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d56000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d50118780 con 0x7f2d5010d500 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d56000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d50118a80 con 0x7f2d50108580 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.902 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 0x7f2d501a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 0x7f2d501a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47710/0 (socket says 172.21.15.146:47710) 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d55600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5010d500 0x7f2d5019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 -- 172.21.15.146:0/3791240982 learned_addr learned my addr 172.21.15.146:0/3791240982 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501a4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43736/0 (socket says 172.21.15.146:43736) 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 -- 172.21.15.146:0/3791240982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5010d500 msgr2=0x7f2d5019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5010d500 0x7f2d5019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 -- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 msgr2=0x7f2d501a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.903 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d54c00640 1 -- 172.21.15.146:0/3791240982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5019f440 con 0x7f2d50108580 2024-04-24T08:30:42.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.900+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:42.904 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.901+0000 7f2d54c00640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 0x7f2d501a4a70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400d770 tx=0x7f2d4400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.901+0000 7f2d4d600640 1 -- 172.21.15.146:0/3791240982 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44017070 con 0x7f2d50108580 2024-04-24T08:30:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.901+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5019f730 con 0x7f2d50108580 2024-04-24T08:30:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.901+0000 7f2d4d600640 1 -- 172.21.15.146:0/3791240982 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d44009c30 con 0x7f2d50108580 2024-04-24T08:30:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.901+0000 7f2d4d600640 1 -- 172.21.15.146:0/3791240982 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44003dd0 con 0x7f2d50108580 2024-04-24T08:30:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.901+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d501ab9f0 con 0x7f2d50108580 2024-04-24T08:30:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.904+0000 7f2d5a5e0740 1 Processor -- start 2024-04-24T08:30:42.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.904+0000 7f2d5a5e0740 1 -- start start 2024-04-24T08:30:42.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a279073f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279080f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55a278fd91d0 con 0x55a279074460 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a278f7f450 con 0x55a27907eb30 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d4d600640 1 -- 172.21.15.146:0/3791240982 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f2d44003f70 con 0x7f2d50108580 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a278fd5190 con 0x55a279073b40 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a279073f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a279073f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47726/0 (socket says 172.21.15.146:47726) 2024-04-24T08:30:42.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 -- 172.21.15.146:0/1458628884 learned_addr learned my addr 172.21.15.146:0/1458628884 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d54c00640 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279080f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 -- 172.21.15.146:0/1458628884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 msgr2=0x55a279080f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279080f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 -- 172.21.15.146:0/1458628884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 msgr2=0x55a27907e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.905+0000 7f2d55600640 1 -- 172.21.15.146:0/1458628884 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27903c310 con 0x55a279073b40 2024-04-24T08:30:42.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.906+0000 7f2d54c00640 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.906+0000 7f2d55600640 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a279073f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009cd0 tx=0x7f2d48030450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=517bd005e93e8dea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.906+0000 7f2d4d600640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d380a3d20 0x7f2d380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.906+0000 7f2d3dc00640 1 -- 172.21.15.146:0/1458628884 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48003970 con 0x55a279073b40 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.906+0000 7f2d3dc00640 1 -- 172.21.15.146:0/1458628884 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d48036030 con 0x55a279073b40 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.906+0000 7f2d3dc00640 1 -- 172.21.15.146:0/1458628884 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48004470 con 0x55a279073b40 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d4d600640 1 -- 172.21.15.146:0/3791240982 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f2d440d0080 con 0x7f2d50108580 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d54c00640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d380a3d20 0x7f2d380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/1458628884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 msgr2=0x55a279073f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a279073f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009cd0 tx=0x7f2d48030450 comp rx=0 tx=0).stop 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d54c00640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d380a3d20 0x7f2d380a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2d440cd870 tx=0x7f2d44000bc0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/1458628884 shutdown_connections 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279080f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/1458628884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a279073f20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/1458628884 >> 172.21.15.146:0/1458628884 conn(0x55a27906fe60 msgr2=0x55a279070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/1458628884 shutdown_connections 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.907+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/1458628884 wait complete. 2024-04-24T08:30:42.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 -- start start 2024-04-24T08:30:42.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 Processor -- start 2024-04-24T08:30:42.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27907e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a279074460 0x55a27907b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a27907eb30 0x55a27907bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55a279081960 con 0x55a27907eb30 2024-04-24T08:30:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55a278f92810 con 0x55a279074460 2024-04-24T08:30:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a279081ae0 con 0x55a279073b40 2024-04-24T08:30:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.908+0000 7f2d5a5e0740 10 client.0 _ll_register_callbacks cb 0x55a278f60140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d55600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27907e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d55600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a27907eb30 0x55a27907bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a279074460 0x55a27907b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a279074460 0x55a27907b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52062/0 (socket says 172.21.15.146:52062) 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 -- 172.21.15.146:0/3416615957 learned_addr learned my addr 172.21.15.146:0/3416615957 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 -- 172.21.15.146:0/3416615957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 msgr2=0x55a27907e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27907e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 -- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a27907eb30 msgr2=0x55a27907bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a27907eb30 0x55a27907bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d4fe00640 1 -- 172.21.15.146:0/3416615957 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a278fa33f0 con 0x55a279074460 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.909+0000 7f2d55600640 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a27907eb30 0x55a27907bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a279074460 0x55a27907b890 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2d40011740 tx=0x7f2d40011c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d07556dc4d61578e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d32000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d32000640 1 -- 172.21.15.146:0/3416615957 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40010ba0 con 0x55a279074460 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d32000640 1 -- 172.21.15.146:0/3416615957 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f2d40010d40 con 0x55a279074460 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d48036030 con 0x55a279074460 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d32000640 1 -- 172.21.15.146:0/3416615957 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2d40005020 con 0x55a279074460 2024-04-24T08:30:42.919 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d5a5e0740 10 client.27278 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a278ceb3f0 con 0x55a279074460 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.910+0000 7f2d32000640 1 -- 172.21.15.146:0/3416615957 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d48036030 con 0x55a279074460 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d32000640 1 -- 172.21.15.146:0/3416615957 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2d4001a410 con 0x55a279074460 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d5a5e0740 10 client.27278 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d5a5e0740 10 client.27278 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d5a5e0740 20 client.27278 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a27908ed80 0x55a2790911a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.920 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d5a5e0740 4 client.27278 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a278fe6bf0 con 0x55a27908ed80 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.911+0000 7f2d55600640 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a27908ed80 0x55a2790911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.912+0000 7f2d55600640 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a27908ed80 0x55a2790911a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6af41c67551ae25f server_cookie=2e8e1fabaf4a9d20 in_seq=0 out_seq=0 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.912+0000 7f2d32000640 10 client.27278 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.912+0000 7f2d32000640 1 -- 172.21.15.146:0/3416615957 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a278fe6bf0 con 0x55a27908ed80 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.912+0000 7f2d32000640 10 client.27278 handle_command_reply: tid=0 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.912+0000 7f2d5a5e0740 1 client.27278 shutdown 2024-04-24T08:30:42.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a279074460 msgr2=0x55a27907b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a279074460 0x55a27907b890 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2d40011740 tx=0x7f2d40011c00 comp rx=0 tx=0).stop 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 shutdown_connections 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a27908ed80 0x55a2790911a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a27907eb30 0x55a27907bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a279074460 0x55a27907b890 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d32000640 0 client.27278 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/3416615957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27907e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 >> 172.21.15.146:0/3416615957 conn(0x55a27906fe60 msgr2=0x55a279075c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 shutdown_connections 2024-04-24T08:30:42.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/3416615957 wait complete. 2024-04-24T08:30:42.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.913+0000 7f2d5a5e0740 20 client.27278 trim_cache size 0 max 16384 2024-04-24T08:30:42.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 Processor -- start 2024-04-24T08:30:42.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 -- start start 2024-04-24T08:30:42.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27908a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a27907e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d501b6ab0 con 0x55a279074460 2024-04-24T08:30:42.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d501b4380 con 0x55a27907eb30 2024-04-24T08:30:42.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d501b21f0 con 0x55a279073b40 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d4fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43762/0 (socket says 172.21.15.146:43762) 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d4fe00640 1 -- 172.21.15.146:0/2015226418 learned_addr learned my addr 172.21.15.146:0/2015226418 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d55600640 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27908a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.928+0000 7f2d55600640 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a27907e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d4fe00640 1 -- 172.21.15.146:0/2015226418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 msgr2=0x55a27908a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27908a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d4fe00640 1 -- 172.21.15.146:0/2015226418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 msgr2=0x55a27907e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a27907e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d4fe00640 1 -- 172.21.15.146:0/2015226418 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27907c340 con 0x55a279074460 2024-04-24T08:30:42.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907a870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2d4001a830 tx=0x7f2d4000bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e895bbf45a79f13d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d3d200640 1 -- 172.21.15.146:0/2015226418 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4001aaa0 con 0x55a279074460 2024-04-24T08:30:42.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d3d200640 1 -- 172.21.15.146:0/2015226418 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d40011e50 con 0x55a279074460 2024-04-24T08:30:42.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.929+0000 7f2d3d200640 1 -- 172.21.15.146:0/2015226418 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d400051c0 con 0x55a279074460 2024-04-24T08:30:42.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/2015226418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 msgr2=0x55a27907a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907a870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2d4001a830 tx=0x7f2d4000bef0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/2015226418 shutdown_connections 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a27907e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279074460 0x55a27907a870 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/2015226418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279073b40 0x55a27908a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/2015226418 >> 172.21.15.146:0/2015226418 conn(0x55a27906fe60 msgr2=0x55a27907a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/2015226418 shutdown_connections 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/2015226418 wait complete. 2024-04-24T08:30:42.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.930+0000 7f2d5a5e0740 1 -- start start 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 1 Processor -- start 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279073b40 0x55a27907f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279074460 0x55a27907f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279085d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2d501b2840 con 0x55a279073b40 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d501b3cb0 con 0x55a27907eb30 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d501b2f10 con 0x55a279074460 2024-04-24T08:30:42.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d5a5e0740 10 client.0 _ll_register_callbacks cb 0x7f2d4000bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:42.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d55600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279073b40 0x55a27907f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d55600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279073b40 0x55a27907f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43772/0 (socket says 172.21.15.146:43772) 2024-04-24T08:30:42.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d55600640 1 -- 172.21.15.146:0/671688833 learned_addr learned my addr 172.21.15.146:0/671688833 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:42.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279074460 0x55a27907f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.931+0000 7f2d55600640 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279085d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d55600640 1 -- 172.21.15.146:0/671688833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279074460 msgr2=0x55a27907f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d55600640 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279074460 0x55a27907f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d55600640 1 -- 172.21.15.146:0/671688833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 msgr2=0x55a279085d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d55600640 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279085d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d55600640 1 -- 172.21.15.146:0/671688833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a278db57d0 con 0x55a279073b40 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d55600640 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279073b40 0x55a27907f300 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2d48000f80 tx=0x7f2d48005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c601bc001fbf651 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d32000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d32000640 1 -- 172.21.15.146:0/671688833 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4804a040 con 0x55a279073b40 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d40011e50 con 0x55a279073b40 2024-04-24T08:30:42.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d32000640 1 -- 172.21.15.146:0/671688833 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f2d48030c10 con 0x55a279073b40 2024-04-24T08:30:42.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.932+0000 7f2d32000640 1 -- 172.21.15.146:0/671688833 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2d48004b40 con 0x55a279073b40 2024-04-24T08:30:42.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 10 client.17139 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:42.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a2790871d0 con 0x55a279073b40 2024-04-24T08:30:42.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d32000640 1 -- 172.21.15.146:0/671688833 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d40011e50 con 0x55a279073b40 2024-04-24T08:30:42.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d32000640 1 -- 172.21.15.146:0/671688833 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2d48043070 con 0x55a279073b40 2024-04-24T08:30:42.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 10 client.17139 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:42.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 10 client.17139 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:30:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 20 client.17139 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a2790f52e0 0x55a2790f76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 4 client.17139 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:30:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a278fe0a60 con 0x55a2790f52e0 2024-04-24T08:30:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.933+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a2790f52e0 0x55a2790f76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.934+0000 7f2d32000640 10 client.17139 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:42.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.934+0000 7f2d4fe00640 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a2790f52e0 0x55a2790f76d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9d408f5f7e09bcf3 server_cookie=6b3da5060b457bfb in_seq=0 out_seq=0 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.934+0000 7f2d32000640 1 -- 172.21.15.146:0/671688833 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a278fe0a60 con 0x55a2790f52e0 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.934+0000 7f2d32000640 10 client.17139 handle_command_reply: tid=0 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.934+0000 7f2d5a5e0740 1 client.17139 shutdown 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.934+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279073b40 msgr2=0x55a27907f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.934+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279073b40 0x55a27907f300 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2d48000f80 tx=0x7f2d48005df0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 shutdown_connections 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55a2790f52e0 0x55a2790f76d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a279074460 0x55a27907f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55a27907eb30 0x55a279085d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d32000640 0 client.17139 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:30:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 --2- 172.21.15.146:0/671688833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a279073b40 0x55a27907f300 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 >> 172.21.15.146:0/671688833 conn(0x55a27906fe60 msgr2=0x55a27907a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 shutdown_connections 2024-04-24T08:30:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 1 -- 172.21.15.146:0/671688833 wait complete. 2024-04-24T08:30:42.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d5a5e0740 20 client.17139 trim_cache size 0 max 16384 2024-04-24T08:30:42.945 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:30:42.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.935+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d380a3d20 msgr2=0x7f2d380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d380a3d20 0x7f2d380a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2d440cd870 tx=0x7f2d44000bc0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 msgr2=0x7f2d501a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 0x7f2d501a4a70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400d770 tx=0x7f2d4400dc30 comp rx=0 tx=0).stop 2024-04-24T08:30:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 shutdown_connections 2024-04-24T08:30:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2d380a3d20 0x7f2d380a61e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d5010d500 0x7f2d5019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d50108f40 0x7f2d501a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 --2- 172.21.15.146:0/3791240982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d50108580 0x7f2d501a4a70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:42.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 >> 172.21.15.146:0/3791240982 conn(0x7f2d50077e20 msgr2=0x7f2d50078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:42.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 shutdown_connections 2024-04-24T08:30:42.947 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:42.936+0000 7f2d56000640 1 -- 172.21.15.146:0/3791240982 wait complete. 2024-04-24T08:30:42.954 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:30:43.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.144+0000 7fa4a3200640 1 -- 172.21.15.146:0/2720310206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108f40 msgr2=0x7fa4a41153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.144+0000 7fa4a3200640 1 --2- 172.21.15.146:0/2720310206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108f40 0x7fa4a41153b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009ca0 tx=0x7fa49c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.144+0000 7fa4a3200640 1 -- 172.21.15.146:0/2720310206 shutdown_connections 2024-04-24T08:30:43.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.144+0000 7fa4a3200640 1 --2- 172.21.15.146:0/2720310206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4a4108580 0x7fa4a4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.144+0000 7fa4a3200640 1 --2- 172.21.15.146:0/2720310206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108f40 0x7fa4a41153b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.144+0000 7fa4a3200640 1 --2- 172.21.15.146:0/2720310206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a410d500 0x7fa4a410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.144+0000 7fa4a3200640 1 -- 172.21.15.146:0/2720310206 >> 172.21.15.146:0/2720310206 conn(0x7fa4a4077e20 msgr2=0x7fa4a4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:43.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.145+0000 7fa4a3200640 1 -- 172.21.15.146:0/2720310206 shutdown_connections 2024-04-24T08:30:43.147 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.145+0000 7fa4a3200640 1 -- 172.21.15.146:0/2720310206 wait complete. 2024-04-24T08:30:43.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 Processor -- start 2024-04-24T08:30:43.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 -- start start 2024-04-24T08:30:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108580 0x7fa4a41a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4108f40 0x7fa4a41a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4a410d500 0x7fa4a419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a4118920 con 0x7fa4a4108f40 2024-04-24T08:30:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a41187a0 con 0x7fa4a4108580 2024-04-24T08:30:43.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a4118aa0 con 0x7fa4a410d500 2024-04-24T08:30:43.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa499400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4108f40 0x7fa4a41a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.146+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4a410d500 0x7fa4a419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa4a1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108580 0x7fa4a41a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4a410d500 0x7fa4a419eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47768/0 (socket says 172.21.15.146:47768) 2024-04-24T08:30:43.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa4a2800640 1 -- 172.21.15.146:0/3024196528 learned_addr learned my addr 172.21.15.146:0/3024196528 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:43.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa499400640 1 -- 172.21.15.146:0/3024196528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4a410d500 msgr2=0x7fa4a419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa499400640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4a410d500 0x7fa4a419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa499400640 1 -- 172.21.15.146:0/3024196528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108580 msgr2=0x7fa4a41a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa499400640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108580 0x7fa4a41a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa499400640 1 -- 172.21.15.146:0/3024196528 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a419f280 con 0x7fa4a4108f40 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa499400640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4108f40 0x7fa4a41a4e40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9f0 tx=0x7fa48c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa49b400640 1 -- 172.21.15.146:0/3024196528 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00ccc0 con 0x7fa4a4108f40 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.147+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a419f570 con 0x7fa4a4108f40 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.148+0000 7fa49b400640 1 -- 172.21.15.146:0/3024196528 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa48c005020 con 0x7fa4a4108f40 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.148+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a41ab7c0 con 0x7fa4a4108f40 2024-04-24T08:30:43.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.148+0000 7fa49b400640 1 -- 172.21.15.146:0/3024196528 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c01c070 con 0x7fa4a4108f40 2024-04-24T08:30:43.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 Processor -- start 2024-04-24T08:30:43.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa49b400640 1 -- 172.21.15.146:0/3024196528 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fa48c004a00 con 0x7fa4a4108f40 2024-04-24T08:30:43.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 -- start start 2024-04-24T08:30:43.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8feaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8ff55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bea8f501d0 con 0x55bea8ff5b30 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bea8ef6450 con 0x55bea8feb460 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.151+0000 7fa4abc48740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bea8f4c190 con 0x55bea8feab40 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa499400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8feaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8ff55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8feaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47784/0 (socket says 172.21.15.146:47784) 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8ff55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52094/0 (socket says 172.21.15.146:52094) 2024-04-24T08:30:43.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 -- 172.21.15.146:0/3468458627 learned_addr learned my addr 172.21.15.146:0/3468458627 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 -- 172.21.15.146:0/3468458627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 msgr2=0x55bea8ff55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 --2- 172.21.15.146:0/3468458627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8ff55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 -- 172.21.15.146:0/3468458627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 msgr2=0x55bea8ff7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 --2- 172.21.15.146:0/3468458627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.152+0000 7fa4a2800640 1 -- 172.21.15.146:0/3468458627 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bea8fb3310 con 0x55bea8feab40 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.153+0000 7fa4a2800640 1 --2- 172.21.15.146:0/3468458627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8feaf20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009a40 tx=0x7fa49c02f310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f47484f649be6ce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.153+0000 7fa49b400640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4840a3d20 0x7fa4840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.153+0000 7fa49b400640 1 -- 172.21.15.146:0/3024196528 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7fa48c0d0770 con 0x7fa4a4108f40 2024-04-24T08:30:43.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.153+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4840a3d20 0x7fa4840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.153+0000 7fa493400640 1 -- 172.21.15.146:0/3468458627 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c031e60 con 0x55bea8feab40 2024-04-24T08:30:43.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.153+0000 7fa493400640 1 -- 172.21.15.146:0/3468458627 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa49c036030 con 0x55bea8feab40 2024-04-24T08:30:43.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.153+0000 7fa493400640 1 -- 172.21.15.146:0/3468458627 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0043a0 con 0x55bea8feab40 2024-04-24T08:30:43.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 -- 172.21.15.146:0/3468458627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 msgr2=0x55bea8feaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4840a3d20 0x7fa4840a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55bea8fec100 tx=0x7fa494005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:43.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 --2- 172.21.15.146:0/3468458627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8feaf20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009a40 tx=0x7fa49c02f310 comp rx=0 tx=0).stop 2024-04-24T08:30:43.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 -- 172.21.15.146:0/3468458627 shutdown_connections 2024-04-24T08:30:43.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 --2- 172.21.15.146:0/3468458627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 --2- 172.21.15.146:0/3468458627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8ff55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 --2- 172.21.15.146:0/3468458627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8feaf20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 -- 172.21.15.146:0/3468458627 >> 172.21.15.146:0/3468458627 conn(0x55bea8fe6e60 msgr2=0x55bea8fe7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:43.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 -- 172.21.15.146:0/3468458627 shutdown_connections 2024-04-24T08:30:43.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.154+0000 7fa4abc48740 1 -- 172.21.15.146:0/3468458627 wait complete. 2024-04-24T08:30:43.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 -- start start 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 Processor -- start 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea8ff5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8ff2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55bea8ff87e0 con 0x55bea8feb460 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55bea8f09810 con 0x55bea8feab40 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bea8ff8960 con 0x55bea8ff5b30 2024-04-24T08:30:43.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4abc48740 10 client.0 _ll_register_callbacks cb 0x55bea8ed7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:43.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.155+0000 7fa499400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8ff2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47796/0 (socket says 172.21.15.146:47796) 2024-04-24T08:30:43.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa4a2800640 1 -- 172.21.15.146:0/2378685193 learned_addr learned my addr 172.21.15.146:0/2378685193 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:43.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea8ff5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa499400640 1 -- 172.21.15.146:0/2378685193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 msgr2=0x55bea8ff2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa499400640 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa499400640 1 -- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 msgr2=0x55bea8ff5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa499400640 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea8ff5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa499400640 1 -- 172.21.15.146:0/2378685193 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bea8f1a3f0 con 0x55bea8feb460 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa499400640 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8ff2870 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0cde50 tx=0x7fa48c098a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721c2bb966864d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa48be00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa48be00640 1 -- 172.21.15.146:0/2378685193 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c09cda0 con 0x55bea8feb460 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa48be00640 1 -- 172.21.15.146:0/2378685193 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7fa48c0ce0d0 con 0x55bea8feb460 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.156+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa49c036030 con 0x55bea8feb460 2024-04-24T08:30:43.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa48be00640 1 -- 172.21.15.146:0/2378685193 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa48c0ce2d0 con 0x55bea8feb460 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 10 client.17151 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bea8c623f0 con 0x55bea8feb460 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa48be00640 1 -- 172.21.15.146:0/2378685193 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa49c036030 con 0x55bea8feb460 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa48be00640 1 -- 172.21.15.146:0/2378685193 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa48c01f030 con 0x55bea8feb460 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 10 client.17151 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 10 client.17151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 20 client.17151 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea9005b70 0x55bea9007f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.166 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 4 client.17151 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.157+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bea8f5dbf0 con 0x55bea9005b70 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.158+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea9005b70 0x55bea9007f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.158+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea9005b70 0x55bea9007f80 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=987e67c9c3a8c65b server_cookie=e0d9c5e06f39d3d in_seq=0 out_seq=0 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.158+0000 7fa48be00640 10 client.17151 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa48be00640 1 -- 172.21.15.146:0/2378685193 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bea8f5dbf0 con 0x55bea9005b70 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa48be00640 10 client.17151 handle_command_reply: tid=0 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 client.17151 shutdown 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 msgr2=0x55bea8ff2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8ff2870 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0cde50 tx=0x7fa48c098a00 comp rx=0 tx=0).stop 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 shutdown_connections 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea9005b70 0x55bea9007f80 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8ff2870 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa48be00640 0 client.17151 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2378685193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea8ff5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.159+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 >> 172.21.15.146:0/2378685193 conn(0x55bea8fe6e60 msgr2=0x55bea8fecc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:43.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.160+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 shutdown_connections 2024-04-24T08:30:43.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.160+0000 7fa4abc48740 1 -- 172.21.15.146:0/2378685193 wait complete. 2024-04-24T08:30:43.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.160+0000 7fa4abc48740 20 client.17151 trim_cache size 0 max 16384 2024-04-24T08:30:43.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.174+0000 7fa4abc48740 1 Processor -- start 2024-04-24T08:30:43.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.174+0000 7fa4abc48740 1 -- start start 2024-04-24T08:30:43.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea9001700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8fe5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4abc48740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a41b68d0 con 0x55bea8feb460 2024-04-24T08:30:43.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4abc48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a41b41a0 con 0x55bea8feab40 2024-04-24T08:30:43.176 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4abc48740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a41b2010 con 0x55bea8ff5b30 2024-04-24T08:30:43.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa499400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8fe5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea9001700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.178 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a1e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea9001700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52102/0 (socket says 172.21.15.146:52102) 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a1e00640 1 -- 172.21.15.146:0/2413101316 learned_addr learned my addr 172.21.15.146:0/2413101316 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa499400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8fe5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43814/0 (socket says 172.21.15.146:43814) 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a1e00640 1 -- 172.21.15.146:0/2413101316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 msgr2=0x55bea8ff50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/2413101316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a1e00640 1 -- 172.21.15.146:0/2413101316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 msgr2=0x55bea8fe5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.175+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/2413101316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8fe5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.176+0000 7fa4a1e00640 1 -- 172.21.15.146:0/2413101316 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bea8d2c7d0 con 0x55bea8feab40 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.176+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/2413101316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea9001700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa494013010 tx=0x7fa49401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=515ae8d1096b9976 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.176+0000 7fa492a00640 1 -- 172.21.15.146:0/2413101316 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4940043b0 con 0x55bea8feab40 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.176+0000 7fa492a00640 1 -- 172.21.15.146:0/2413101316 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa49402b900 con 0x55bea8feab40 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.176+0000 7fa492a00640 1 -- 172.21.15.146:0/2413101316 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494004890 con 0x55bea8feab40 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 -- 172.21.15.146:0/2413101316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 msgr2=0x55bea9001700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2413101316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea9001700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa494013010 tx=0x7fa49401b000 comp rx=0 tx=0).stop 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 -- 172.21.15.146:0/2413101316 shutdown_connections 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2413101316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8ff5b30 0x55bea8ff50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2413101316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8feb460 0x55bea8fe5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 --2- 172.21.15.146:0/2413101316 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feab40 0x55bea9001700 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 -- 172.21.15.146:0/2413101316 >> 172.21.15.146:0/2413101316 conn(0x55bea8fe6e60 msgr2=0x55bea8fe56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:43.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 -- 172.21.15.146:0/2413101316 shutdown_connections 2024-04-24T08:30:43.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 -- 172.21.15.146:0/2413101316 wait complete. 2024-04-24T08:30:43.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.177+0000 7fa4abc48740 1 -- start start 2024-04-24T08:30:43.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 1 Processor -- start 2024-04-24T08:30:43.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8ffe0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8fefd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a41b2660 con 0x55bea8ff5b30 2024-04-24T08:30:43.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a41b3ad0 con 0x55bea8feb460 2024-04-24T08:30:43.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a41b2d30 con 0x55bea8feab40 2024-04-24T08:30:43.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4abc48740 10 client.0 _ll_register_callbacks cb 0x7fa48c0d6220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:43.185 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa499400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8fefd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4a1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8ffe0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4a1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8ffe0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47816/0 (socket says 172.21.15.146:47816) 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa499400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8fefd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52118/0 (socket says 172.21.15.146:52118) 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4a2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43822/0 (socket says 172.21.15.146:43822) 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.178+0000 7fa4a1e00640 1 -- 172.21.15.146:0/163641339 learned_addr learned my addr 172.21.15.146:0/163641339 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa499400640 1 -- 172.21.15.146:0/163641339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 msgr2=0x55bea8ffe0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.186 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa499400640 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8ffe0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa499400640 1 -- 172.21.15.146:0/163641339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 msgr2=0x55bea8ff0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa499400640 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa499400640 1 -- 172.21.15.146:0/163641339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bea8ff6ba0 con 0x55bea8feb460 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa499400640 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8fefd80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00c6a0 tx=0x7fa48c0cdf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25eaf33a3064378 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa48be00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa48be00640 1 -- 172.21.15.146:0/163641339 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c09c1e0 con 0x55bea8feb460 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa48be00640 1 -- 172.21.15.146:0/163641339 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7fa48c09bcf0 con 0x55bea8feb460 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.179+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa49402b900 con 0x55bea8feb460 2024-04-24T08:30:43.187 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa48be00640 1 -- 172.21.15.146:0/163641339 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa48c0ce2e0 con 0x55bea8feb460 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 10 client.27302 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bea906c520 con 0x55bea8feb460 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa48be00640 1 -- 172.21.15.146:0/163641339 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa49402b900 con 0x55bea8feb460 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa48be00640 1 -- 172.21.15.146:0/163641339 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa48c01f070 con 0x55bea8feb460 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 10 client.27302 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 10 client.27302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 20 client.27302 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea906ca20 0x55bea906ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 4 client.27302 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:30:43.188 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bea8f57a60 con 0x55bea906ca20 2024-04-24T08:30:43.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.180+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea906ca20 0x55bea906ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:43.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4a1e00640 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea906ca20 0x55bea906ee40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=540bd11457694a3c server_cookie=d0c4e9ca9c919aab in_seq=0 out_seq=0 2024-04-24T08:30:43.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa48be00640 10 client.27302 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:43.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa48be00640 1 -- 172.21.15.146:0/163641339 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bea8f57a60 con 0x55bea906ca20 2024-04-24T08:30:43.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa48be00640 10 client.27302 handle_command_reply: tid=0 2024-04-24T08:30:43.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 client.27302 shutdown 2024-04-24T08:30:43.189 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 msgr2=0x55bea8fefd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8fefd80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00c6a0 tx=0x7fa48c0cdf30 comp rx=0 tx=0).stop 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 shutdown_connections 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55bea906ca20 0x55bea906ee40 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55bea8ff5b30 0x55bea8ff0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55bea8feb460 0x55bea8fefd80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa48be00640 0 client.27302 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 --2- 172.21.15.146:0/163641339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bea8feab40 0x55bea8ffe0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 >> 172.21.15.146:0/163641339 conn(0x55bea8fe6e60 msgr2=0x55bea8fe5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:43.190 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.181+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 shutdown_connections 2024-04-24T08:30:43.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4abc48740 1 -- 172.21.15.146:0/163641339 wait complete. 2024-04-24T08:30:43.191 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4abc48740 20 client.27302 trim_cache size 0 max 16384 2024-04-24T08:30:43.191 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:30:43.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4840a3d20 msgr2=0x7fa4840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4840a3d20 0x7fa4840a61e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55bea8fec100 tx=0x7fa494005f90 comp rx=0 tx=0).stop 2024-04-24T08:30:43.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4108f40 msgr2=0x7fa4a41a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:43.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4108f40 0x7fa4a41a4e40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9f0 tx=0x7fa48c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 shutdown_connections 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa4840a3d20 0x7fa4840a61e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4a410d500 0x7fa4a419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4a4108f40 0x7fa4a41a4e40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 --2- 172.21.15.146:0/3024196528 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa4a4108580 0x7fa4a41a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.182+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 >> 172.21.15.146:0/3024196528 conn(0x7fa4a4077e20 msgr2=0x7fa4a40782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.183+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 shutdown_connections 2024-04-24T08:30:43.193 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:43.183+0000 7fa4a3200640 1 -- 172.21.15.146:0/3024196528 wait complete. 2024-04-24T08:30:44.201 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:30:44.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:30:44.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:30:44.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 -- 172.21.15.146:0/3174224088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 msgr2=0x7fef54115130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 --2- 172.21.15.146:0/3174224088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 0x7fef54115130 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009cd0 tx=0x7fef4c02a8e0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 -- 172.21.15.146:0/3174224088 shutdown_connections 2024-04-24T08:30:44.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 --2- 172.21.15.146:0/3174224088 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 0x7fef54115130 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 --2- 172.21.15.146:0/3174224088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 0x7fef54112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 --2- 172.21.15.146:0/3174224088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef54076530 0x7fef54076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 -- 172.21.15.146:0/3174224088 >> 172.21.15.146:0/3174224088 conn(0x7fef540ffb10 msgr2=0x7fef54101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 -- 172.21.15.146:0/3174224088 shutdown_connections 2024-04-24T08:30:44.423 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.419+0000 7fef5ae00640 1 -- 172.21.15.146:0/3174224088 wait complete. 2024-04-24T08:30:44.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 Processor -- start 2024-04-24T08:30:44.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 -- start start 2024-04-24T08:30:44.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef54076530 0x7fef5419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 0x7fef5419f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 0x7fef541a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fef541188d0 con 0x7fef54076e50 2024-04-24T08:30:44.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef54118750 con 0x7fef54112d40 2024-04-24T08:30:44.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.421+0000 7fef5ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef54118a50 con 0x7fef54076530 2024-04-24T08:30:44.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 0x7fef5419f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef54076530 0x7fef5419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 0x7fef5419f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43830/0 (socket says 172.21.15.146:43830) 2024-04-24T08:30:44.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59000640 1 -- 172.21.15.146:0/427013321 learned_addr learned my addr 172.21.15.146:0/427013321 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef5a400640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 0x7fef541a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59a00640 1 -- 172.21.15.146:0/427013321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 msgr2=0x7fef541a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59a00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 0x7fef541a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59a00640 1 -- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 msgr2=0x7fef5419f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59a00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 0x7fef5419f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59a00640 1 -- 172.21.15.146:0/427013321 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef541a3e20 con 0x7fef54076530 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.422+0000 7fef59000640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 0x7fef5419f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.423+0000 7fef59a00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef54076530 0x7fef5419ee30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef4800e9f0 tx=0x7fef4800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.423+0000 7fef52000640 1 -- 172.21.15.146:0/427013321 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4800ccc0 con 0x7fef54076530 2024-04-24T08:30:44.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.423+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef541a4110 con 0x7fef54076530 2024-04-24T08:30:44.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.423+0000 7fef52000640 1 -- 172.21.15.146:0/427013321 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef48005020 con 0x7fef54076530 2024-04-24T08:30:44.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.423+0000 7fef52000640 1 -- 172.21.15.146:0/427013321 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef480049e0 con 0x7fef54076530 2024-04-24T08:30:44.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.423+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef541ab9a0 con 0x7fef54076530 2024-04-24T08:30:44.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.426+0000 7fef5f1fc740 1 Processor -- start 2024-04-24T08:30:44.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.426+0000 7fef5f1fc740 1 -- start start 2024-04-24T08:30:44.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68c270 0x55ba5e68c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e696d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.430 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e697260 0x55ba5e699650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55ba5e5f1e70 con 0x55ba5e68cb90 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ba5e598380 con 0x55ba5e68c270 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ba5e5ede30 con 0x55ba5e697260 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef52000640 1 -- 172.21.15.146:0/427013321 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fef48005560 con 0x7fef54076530 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef59000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e697260 0x55ba5e699650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef5a400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68c270 0x55ba5e68c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef59000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e697260 0x55ba5e699650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47848/0 (socket says 172.21.15.146:47848) 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef59000640 1 -- 172.21.15.146:0/2642669359 learned_addr learned my addr 172.21.15.146:0/2642669359 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef59a00640 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e696d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.427+0000 7fef59000640 1 -- 172.21.15.146:0/2642669359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68c270 msgr2=0x55ba5e68c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef59000640 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68c270 0x55ba5e68c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef59000640 1 -- 172.21.15.146:0/2642669359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 msgr2=0x55ba5e696d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef59000640 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e696d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef59000640 1 -- 172.21.15.146:0/2642669359 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba5e3cc7e0 con 0x55ba5e697260 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef5a400640 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68c270 0x55ba5e68c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef59000640 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e697260 0x55ba5e699650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef40011510 tx=0x7fef400119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79e0d996ca58efe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef46a00640 1 -- 172.21.15.146:0/2642669359 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4001e070 con 0x55ba5e697260 2024-04-24T08:30:44.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef46a00640 1 -- 172.21.15.146:0/2642669359 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef4000faa0 con 0x55ba5e697260 2024-04-24T08:30:44.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef46a00640 1 -- 172.21.15.146:0/2642669359 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40019b80 con 0x55ba5e697260 2024-04-24T08:30:44.433 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.428+0000 7fef52000640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fef3c0a3d20 0x7fef3c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.437 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef52000640 1 -- 172.21.15.146:0/427013321 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7fef480d07a0 con 0x7fef54076530 2024-04-24T08:30:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5a400640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fef3c0a3d20 0x7fef3c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2642669359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e697260 msgr2=0x55ba5e699650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e697260 0x55ba5e699650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef40011510 tx=0x7fef400119d0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2642669359 shutdown_connections 2024-04-24T08:30:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e697260 0x55ba5e699650 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e696d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2642669359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68c270 0x55ba5e68c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2642669359 >> 172.21.15.146:0/2642669359 conn(0x55ba5e688590 msgr2=0x55ba5e688970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2642669359 shutdown_connections 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.429+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2642669359 wait complete. 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.430+0000 7fef5a400640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fef3c0a3d20 0x7fef3c0a61e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fef4c033040 tx=0x7fef4c03d040 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.430+0000 7fef5f1fc740 1 -- start start 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.430+0000 7fef5f1fc740 1 Processor -- start 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.430+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68c270 0x55ba5e694210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.430+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68cb90 0x55ba5e694750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.439 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.430+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e694c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.430+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55ba5e69a050 con 0x55ba5e68c270 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55ba5e5ab660 con 0x55ba5e697260 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ba5e69a1d0 con 0x55ba5e68cb90 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef5f1fc740 10 client.0 _ll_register_callbacks cb 0x55ba5e599aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68c270 0x55ba5e694210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68cb90 0x55ba5e694750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68c270 0x55ba5e694210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43852/0 (socket says 172.21.15.146:43852) 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59a00640 1 -- 172.21.15.146:0/2555325894 learned_addr learned my addr 172.21.15.146:0/2555325894 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef5a400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e694c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59000640 1 -- 172.21.15.146:0/2555325894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 msgr2=0x55ba5e694c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59000640 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e694c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59000640 1 -- 172.21.15.146:0/2555325894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68c270 msgr2=0x55ba5e694210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59000640 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68c270 0x55ba5e694210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59000640 1 -- 172.21.15.146:0/2555325894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba5e42cf70 con 0x55ba5e68cb90 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.431+0000 7fef59a00640 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68c270 0x55ba5e694210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef59000640 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68cb90 0x55ba5e694750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fef40011ed0 tx=0x7fef4000f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=decaf3c2c16ab463 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef3b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef3b400640 1 -- 172.21.15.146:0/2555325894 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4001e070 con 0x55ba5e68cb90 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef3b400640 1 -- 172.21.15.146:0/2555325894 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7fef40019b10 con 0x55ba5e68cb90 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef4000faa0 con 0x55ba5e68cb90 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef3b400640 1 -- 172.21.15.146:0/2555325894 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef40030050 con 0x55ba5e68cb90 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef5f1fc740 10 client.27319 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ba5e2c4170 con 0x55ba5e68cb90 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.432+0000 7fef3b400640 1 -- 172.21.15.146:0/2555325894 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef4000faa0 con 0x55ba5e68cb90 2024-04-24T08:30:44.442 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef3b400640 1 -- 172.21.15.146:0/2555325894 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fef4002b030 con 0x55ba5e68cb90 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef5f1fc740 10 client.27319 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef5f1fc740 10 client.27319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef5f1fc740 20 client.27319 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e6a75a0 0x55ba5e6a9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef5f1fc740 4 client.27319 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba5e5ff890 con 0x55ba5e6a75a0 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.433+0000 7fef59a00640 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e6a75a0 0x55ba5e6a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.434+0000 7fef59a00640 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e6a75a0 0x55ba5e6a9990 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad9cdffb760efca server_cookie=d5155daa871ebdd8 in_seq=0 out_seq=0 2024-04-24T08:30:44.443 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.434+0000 7fef3b400640 10 client.27319 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef3b400640 1 -- 172.21.15.146:0/2555325894 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba5e5ff890 con 0x55ba5e6a75a0 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef3b400640 10 client.27319 handle_command_reply: tid=0 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 client.27319 shutdown 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68cb90 msgr2=0x55ba5e694750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68cb90 0x55ba5e694750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fef40011ed0 tx=0x7fef4000f7f0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 shutdown_connections 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e6a75a0 0x55ba5e6a9990 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.444 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68cb90 0x55ba5e694750 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e694c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef3b400640 0 client.27319 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2555325894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68c270 0x55ba5e694210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 >> 172.21.15.146:0/2555325894 conn(0x55ba5e688590 msgr2=0x55ba5e68e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 shutdown_connections 2024-04-24T08:30:44.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2555325894 wait complete. 2024-04-24T08:30:44.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.435+0000 7fef5f1fc740 20 client.27319 trim_cache size 0 max 16384 2024-04-24T08:30:44.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 Processor -- start 2024-04-24T08:30:44.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 -- start start 2024-04-24T08:30:44.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e6a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68cb90 0x55ba5e6948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e697260 0x55ba5e6af100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fef541b6b50 con 0x55ba5e697260 2024-04-24T08:30:44.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef541b4420 con 0x55ba5e68cb90 2024-04-24T08:30:44.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.455+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef541b2290 con 0x55ba5e68c270 2024-04-24T08:30:44.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e6a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e6a2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47878/0 (socket says 172.21.15.146:47878) 2024-04-24T08:30:44.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 -- 172.21.15.146:0/2442811583 learned_addr learned my addr 172.21.15.146:0/2442811583 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef5a400640 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e697260 0x55ba5e6af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59000640 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68cb90 0x55ba5e6948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 -- 172.21.15.146:0/2442811583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68cb90 msgr2=0x55ba5e6948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68cb90 0x55ba5e6948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 -- 172.21.15.146:0/2442811583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e697260 msgr2=0x55ba5e6af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e697260 0x55ba5e6af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 -- 172.21.15.146:0/2442811583 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba5e699f30 con 0x55ba5e68c270 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.456+0000 7fef59a00640 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e6a2f10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fef48099100 tx=0x7fef480d2cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dcf5f247ba2f758 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef46000640 1 -- 172.21.15.146:0/2442811583 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4801c040 con 0x55ba5e68c270 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef46000640 1 -- 172.21.15.146:0/2442811583 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef480051d0 con 0x55ba5e68c270 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef46000640 1 -- 172.21.15.146:0/2442811583 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4809a7d0 con 0x55ba5e68c270 2024-04-24T08:30:44.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2442811583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 msgr2=0x55ba5e6a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e6a2f10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fef48099100 tx=0x7fef480d2cb0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2442811583 shutdown_connections 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e697260 0x55ba5e6af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e68cb90 0x55ba5e6948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2442811583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e6a2f10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.457+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2442811583 >> 172.21.15.146:0/2442811583 conn(0x55ba5e688590 msgr2=0x55ba5e6941d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.458+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2442811583 shutdown_connections 2024-04-24T08:30:44.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.458+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2442811583 wait complete. 2024-04-24T08:30:44.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.458+0000 7fef5f1fc740 1 -- start start 2024-04-24T08:30:44.463 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.458+0000 7fef5f1fc740 1 Processor -- start 2024-04-24T08:30:44.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e69f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e690a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5f1fc740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e690f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fef541b28e0 con 0x55ba5e68cb90 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fef541b3d50 con 0x55ba5e697260 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5f1fc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef541b2fb0 con 0x55ba5e68c270 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5f1fc740 10 client.0 _ll_register_callbacks cb 0x7fef4001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e690a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e69f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e690a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:43878/0 (socket says 172.21.15.146:43878) 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 -- 172.21.15.146:0/2458415661 learned_addr learned my addr 172.21.15.146:0/2458415661 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef5a400640 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e690f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 -- 172.21.15.146:0/2458415661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 msgr2=0x55ba5e69f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e69f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 -- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 msgr2=0x55ba5e690f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e690f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59000640 1 -- 172.21.15.146:0/2458415661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba5e6915c0 con 0x55ba5e68cb90 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.459+0000 7fef59a00640 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e69f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef59000640 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e690a40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55ba5e6957e0 tx=0x7fef40009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e08489debb5746 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef3b400640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:30:44.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef3b400640 1 -- 172.21.15.146:0/2458415661 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4001e070 con 0x55ba5e68cb90 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef3b400640 1 -- 172.21.15.146:0/2458415661 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7fef40011040 con 0x55ba5e68cb90 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef480051d0 con 0x55ba5e68cb90 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef3b400640 1 -- 172.21.15.146:0/2458415661 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fef4000faa0 con 0x55ba5e68cb90 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef5f1fc740 10 client.17175 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ba5e70d810 con 0x55ba5e68cb90 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef3b400640 1 -- 172.21.15.146:0/2458415661 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef480051d0 con 0x55ba5e68cb90 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.460+0000 7fef3b400640 1 -- 172.21.15.146:0/2458415661 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fef4002b040 con 0x55ba5e68cb90 2024-04-24T08:30:44.469 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef5f1fc740 10 client.17175 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef5f1fc740 10 client.17175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef5f1fc740 20 client.17175 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e70e010 0x55ba5e710430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef5f1fc740 4 client.17175 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ba5e5f9700 con 0x55ba5e70e010 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef59a00640 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e70e010 0x55ba5e710430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef59a00640 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e70e010 0x55ba5e710430 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20703b1d4eb191f4 server_cookie=63849e4f2159de29 in_seq=0 out_seq=0 2024-04-24T08:30:44.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.461+0000 7fef3b400640 10 client.17175 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef3b400640 1 -- 172.21.15.146:0/2458415661 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ba5e5f9700 con 0x55ba5e70e010 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef3b400640 10 client.17175 handle_command_reply: tid=0 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 client.17175 shutdown 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 msgr2=0x55ba5e690a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e690a40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55ba5e6957e0 tx=0x7fef40009530 comp rx=0 tx=0).stop 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 shutdown_connections 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55ba5e70e010 0x55ba5e710430 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55ba5e697260 0x55ba5e690f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ba5e68cb90 0x55ba5e690a40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef3b400640 0 client.17175 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 --2- 172.21.15.146:0/2458415661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5e68c270 0x55ba5e69f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 >> 172.21.15.146:0/2458415661 conn(0x55ba5e688590 msgr2=0x55ba5e694a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 shutdown_connections 2024-04-24T08:30:44.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 1 -- 172.21.15.146:0/2458415661 wait complete. 2024-04-24T08:30:44.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.463+0000 7fef5f1fc740 20 client.17175 trim_cache size 0 max 16384 2024-04-24T08:30:44.473 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:30:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:30:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "abb816fc-29b7-4770-9eee-9cb220250648", 2024-04-24T08:30:44.473 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:30:44.473 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:30:44.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fef3c0a3d20 msgr2=0x7fef3c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fef3c0a3d20 0x7fef3c0a61e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fef4c033040 tx=0x7fef4c03d040 comp rx=0 tx=0).stop 2024-04-24T08:30:44.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef54076530 msgr2=0x7fef5419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef54076530 0x7fef5419ee30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fef4800e9f0 tx=0x7fef4800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 shutdown_connections 2024-04-24T08:30:44.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fef3c0a3d20 0x7fef3c0a61e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef54112d40 0x7fef541a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef54076e50 0x7fef5419f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 --2- 172.21.15.146:0/427013321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef54076530 0x7fef5419ee30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 >> 172.21.15.146:0/427013321 conn(0x7fef540ffb10 msgr2=0x7fef541143f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 shutdown_connections 2024-04-24T08:30:44.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.464+0000 7fef5ae00640 1 -- 172.21.15.146:0/427013321 wait complete. 2024-04-24T08:30:44.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:30:44.487 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:30:44.546 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:44 smithi146 ceph-mon[27805]: pgmap v295: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:44.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:44 smithi153 ceph-mon[32746]: pgmap v295: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:44.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.736+0000 7f577ec00640 1 -- 172.21.15.146:0/3255096144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 msgr2=0x7f5778104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.736+0000 7f577ec00640 1 --2- 172.21.15.146:0/3255096144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 0x7f5778104830 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5768009900 tx=0x7f576802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.736+0000 7f577ec00640 1 -- 172.21.15.146:0/3255096144 shutdown_connections 2024-04-24T08:30:44.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.736+0000 7f577ec00640 1 --2- 172.21.15.146:0/3255096144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5778109520 0x7f57781110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.736+0000 7f577ec00640 1 --2- 172.21.15.146:0/3255096144 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778104e00 0x7f5778108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.736+0000 7f577ec00640 1 --2- 172.21.15.146:0/3255096144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 0x7f5778104830 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.737+0000 7f577ec00640 1 -- 172.21.15.146:0/3255096144 >> 172.21.15.146:0/3255096144 conn(0x7f57780ffc70 msgr2=0x7f5778102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.737+0000 7f577ec00640 1 -- 172.21.15.146:0/3255096144 shutdown_connections 2024-04-24T08:30:44.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.737+0000 7f577ec00640 1 -- 172.21.15.146:0/3255096144 wait complete. 2024-04-24T08:30:44.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 Processor -- start 2024-04-24T08:30:44.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 -- start start 2024-04-24T08:30:44.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 0x7f5778111090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778109520 0x7f577810e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f577810e8f0 0x7f57781ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5778114600 con 0x7f5778104450 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5778114480 con 0x7f5778109520 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.739+0000 7f577ec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5778114780 con 0x7f577810e8f0 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778109520 0x7f577810e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778109520 0x7f577810e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52202/0 (socket says 172.21.15.146:52202) 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 -- 172.21.15.146:0/754350310 learned_addr learned my addr 172.21.15.146:0/754350310 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777e00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 0x7f5778111090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f577cc00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f577810e8f0 0x7f57781ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 -- 172.21.15.146:0/754350310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f577810e8f0 msgr2=0x7f57781ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.743 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f577810e8f0 0x7f57781ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 -- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 msgr2=0x7f5778111090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 0x7f5778111090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f5777400640 1 -- 172.21.15.146:0/754350310 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57781ab880 con 0x7f5778109520 2024-04-24T08:30:44.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.740+0000 7f577cc00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f577810e8f0 0x7f57781ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:44.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.741+0000 7f5777400640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778109520 0x7f577810e250 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5764005fd0 tx=0x7f576400dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.741+0000 7f5774c00640 1 -- 172.21.15.146:0/754350310 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764017070 con 0x7f5778109520 2024-04-24T08:30:44.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.741+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57781aba50 con 0x7f5778109520 2024-04-24T08:30:44.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.741+0000 7f5774c00640 1 -- 172.21.15.146:0/754350310 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f576400de50 con 0x7f5778109520 2024-04-24T08:30:44.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.741+0000 7f5774c00640 1 -- 172.21.15.146:0/754350310 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764003e00 con 0x7f5778109520 2024-04-24T08:30:44.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.742+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57781abeb0 con 0x7f5778109520 2024-04-24T08:30:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.749+0000 7f5774c00640 1 -- 172.21.15.146:0/754350310 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f5764003fa0 con 0x7f5778109520 2024-04-24T08:30:44.752 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.750+0000 7f5781b99740 1 Processor -- start 2024-04-24T08:30:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.751+0000 7f5781b99740 1 -- start start 2024-04-24T08:30:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.751+0000 7f5781b99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a891c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.753 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.751+0000 7f5781b99740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a89c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.751+0000 7f5781b99740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a89c850 0x55b01a89ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.751+0000 7f5781b99740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55b01a7f71f0 con 0x55b01a89c850 2024-04-24T08:30:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.751+0000 7f5781b99740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b01a79d350 con 0x55b01a892180 2024-04-24T08:30:44.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.751+0000 7f5781b99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b01a7f31b0 con 0x55b01a891860 2024-04-24T08:30:44.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a89c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a89c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52204/0 (socket says 172.21.15.146:52204) 2024-04-24T08:30:44.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 -- 172.21.15.146:0/1680083845 learned_addr learned my addr 172.21.15.146:0/1680083845 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f577cc00640 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a891c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 -- 172.21.15.146:0/1680083845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 msgr2=0x55b01a891c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a891c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 -- 172.21.15.146:0/1680083845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a89c850 msgr2=0x55b01a89ec40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:30:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5774c00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f575c0a3e20 0x7f575c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a89c850 0x55b01a89ec40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.752+0000 7f5777e00640 1 -- 172.21.15.146:0/1680083845 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b01a7f1f50 con 0x55b01a892180 2024-04-24T08:30:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.753+0000 7f5777e00640 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a89c310 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5768009900 tx=0x7f5768004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a80c0f803f6db490 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.753+0000 7f5774c00640 1 -- 172.21.15.146:0/754350310 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f57640d0390 con 0x7f5778109520 2024-04-24T08:30:44.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.753+0000 7f576d200640 1 -- 172.21.15.146:0/1680083845 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768035040 con 0x55b01a892180 2024-04-24T08:30:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.753+0000 7f576d200640 1 -- 172.21.15.146:0/1680083845 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5768004430 con 0x55b01a892180 2024-04-24T08:30:44.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.753+0000 7f576d200640 1 -- 172.21.15.146:0/1680083845 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768003850 con 0x55b01a892180 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.754+0000 7f577cc00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f575c0a3e20 0x7f575c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.754+0000 7f5781b99740 1 -- 172.21.15.146:0/1680083845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 msgr2=0x55b01a89c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.754+0000 7f5781b99740 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a89c310 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5768009900 tx=0x7f5768004040 comp rx=0 tx=0).stop 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f577cc00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f575c0a3e20 0x7f575c0a62e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f57700099a0 tx=0x7f5770009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f5781b99740 1 -- 172.21.15.146:0/1680083845 shutdown_connections 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f5781b99740 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a89c850 0x55b01a89ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f5781b99740 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a89c310 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f5781b99740 1 --2- 172.21.15.146:0/1680083845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a891c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f5781b99740 1 -- 172.21.15.146:0/1680083845 >> 172.21.15.146:0/1680083845 conn(0x55b01a88db80 msgr2=0x55b01a88df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f5781b99740 1 -- 172.21.15.146:0/1680083845 shutdown_connections 2024-04-24T08:30:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.755+0000 7f5781b99740 1 -- 172.21.15.146:0/1680083845 wait complete. 2024-04-24T08:30:44.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.756+0000 7f5781b99740 1 -- start start 2024-04-24T08:30:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.757+0000 7f5781b99740 1 Processor -- start 2024-04-24T08:30:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5781b99740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a891860 0x55b01a899780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5781b99740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a899cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5781b99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a89c850 0x55b01a89a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5781b99740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55b01a89f5e0 con 0x55b01a891860 2024-04-24T08:30:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5781b99740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b01a7b0690 con 0x55b01a892180 2024-04-24T08:30:44.760 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5781b99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b01a89f760 con 0x55b01a89c850 2024-04-24T08:30:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5777400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a899cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5777e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a891860 0x55b01a899780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5777400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a899cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:52218/0 (socket says 172.21.15.146:52218) 2024-04-24T08:30:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5777400640 1 -- 172.21.15.146:0/939353921 learned_addr learned my addr 172.21.15.146:0/939353921 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f5781b99740 10 client.0 _ll_register_callbacks cb 0x55b01a7ec650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.758+0000 7f577cc00640 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a89c850 0x55b01a89a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5777400640 1 -- 172.21.15.146:0/939353921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a89c850 msgr2=0x55b01a89a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5777400640 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a89c850 0x55b01a89a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5777400640 1 -- 172.21.15.146:0/939353921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a891860 msgr2=0x55b01a899780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5777400640 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a891860 0x55b01a899780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5777400640 1 -- 172.21.15.146:0/939353921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b01a7af920 con 0x55b01a892180 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5777e00640 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a891860 0x55b01a899780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5777400640 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a899cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5764009610 tx=0x7f576409bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea88283f77f7e390 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.763 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.759+0000 7f5761600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.760+0000 7f5761600640 1 -- 172.21.15.146:0/939353921 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57640b5060 con 0x55b01a892180 2024-04-24T08:30:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.760+0000 7f5761600640 1 -- 172.21.15.146:0/939353921 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f57640cdcf0 con 0x55b01a892180 2024-04-24T08:30:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.760+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5768004430 con 0x55b01a892180 2024-04-24T08:30:44.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.760+0000 7f5761600640 1 -- 172.21.15.146:0/939353921 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f57640cdef0 con 0x55b01a892180 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.761+0000 7f5781b99740 10 client.27332 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.761+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b01a5d37d0 con 0x55b01a892180 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.761+0000 7f5761600640 1 -- 172.21.15.146:0/939353921 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5768004430 con 0x55b01a892180 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5761600640 1 -- 172.21.15.146:0/939353921 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5764004290 con 0x55b01a892180 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5781b99740 10 client.27332 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5781b99740 10 client.27332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5781b99740 20 client.27332 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5781b99740 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a8acbf0 0x55b01a8aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5781b99740 4 client.27332 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:30:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b01a804c10 con 0x55b01a8acbf0 2024-04-24T08:30:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.762+0000 7f5777e00640 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a8acbf0 0x55b01a8aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.763+0000 7f5777e00640 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a8acbf0 0x55b01a8aefe0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3abd6111e39fd86 server_cookie=bf23436a4bb3cd89 in_seq=0 out_seq=0 2024-04-24T08:30:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.763+0000 7f5761600640 10 client.27332 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.764+0000 7f5761600640 1 -- 172.21.15.146:0/939353921 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b01a804c10 con 0x55b01a8acbf0 2024-04-24T08:30:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.764+0000 7f5761600640 10 client.27332 handle_command_reply: tid=0 2024-04-24T08:30:44.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.764+0000 7f5781b99740 1 client.27332 shutdown 2024-04-24T08:30:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 msgr2=0x55b01a899cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a899cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5764009610 tx=0x7f576409bef0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 shutdown_connections 2024-04-24T08:30:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a8acbf0 0x55b01a8aefe0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a89c850 0x55b01a89a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a892180 0x55b01a899cc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 --2- 172.21.15.146:0/939353921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a891860 0x55b01a899780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 >> 172.21.15.146:0/939353921 conn(0x55b01a88db80 msgr2=0x55b01a893980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.765+0000 7f5761600640 0 client.27332 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.766+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 shutdown_connections 2024-04-24T08:30:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.766+0000 7f5781b99740 1 -- 172.21.15.146:0/939353921 wait complete. 2024-04-24T08:30:44.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.766+0000 7f5781b99740 20 client.27332 trim_cache size 0 max 16384 2024-04-24T08:30:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 Processor -- start 2024-04-24T08:30:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 -- start start 2024-04-24T08:30:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a898700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a894000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a89a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f57781b6d60 con 0x55b01a892180 2024-04-24T08:30:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57781b4630 con 0x55b01a89c850 2024-04-24T08:30:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5781b99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f57781b24a0 con 0x55b01a891860 2024-04-24T08:30:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5777e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a898700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5777e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a898700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47924/0 (socket says 172.21.15.146:47924) 2024-04-24T08:30:44.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5777e00640 1 -- 172.21.15.146:0/3433269853 learned_addr learned my addr 172.21.15.146:0/3433269853 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f5777400640 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a894000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.799 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.795+0000 7f577cc00640 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a89a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5777e00640 1 -- 172.21.15.146:0/3433269853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 msgr2=0x55b01a89a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5777e00640 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a89a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5777e00640 1 -- 172.21.15.146:0/3433269853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 msgr2=0x55b01a894000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5777e00640 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a894000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5777e00640 1 -- 172.21.15.146:0/3433269853 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b01a5093f0 con 0x55b01a891860 2024-04-24T08:30:44.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5777e00640 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a898700 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5768009900 tx=0x7f5768004c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d6ae9dc1694839a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5763e00640 1 -- 172.21.15.146:0/3433269853 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768035040 con 0x55b01a891860 2024-04-24T08:30:44.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5763e00640 1 -- 172.21.15.146:0/3433269853 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5768053020 con 0x55b01a891860 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.796+0000 7f5763e00640 1 -- 172.21.15.146:0/3433269853 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5768031810 con 0x55b01a891860 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 -- 172.21.15.146:0/3433269853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 msgr2=0x55b01a898700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a898700 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5768009900 tx=0x7f5768004c00 comp rx=0 tx=0).stop 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 -- 172.21.15.146:0/3433269853 shutdown_connections 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a89a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a894000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 --2- 172.21.15.146:0/3433269853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a898700 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 -- 172.21.15.146:0/3433269853 >> 172.21.15.146:0/3433269853 conn(0x55b01a88db80 msgr2=0x55b01a88cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 -- 172.21.15.146:0/3433269853 shutdown_connections 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.797+0000 7f5781b99740 1 -- 172.21.15.146:0/3433269853 wait complete. 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 -- start start 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 Processor -- start 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a8a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a8a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a895b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f57781b2af0 con 0x55b01a892180 2024-04-24T08:30:44.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f57781b3f60 con 0x55b01a89c850 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f57781b31c0 con 0x55b01a891860 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.798+0000 7f5781b99740 10 client.0 _ll_register_callbacks cb 0x7f5764009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f5777e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a8a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f577cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a895b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f5777400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a8a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f5777e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a8a49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:47936/0 (socket says 172.21.15.146:47936) 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f5777e00640 1 -- 172.21.15.146:0/1938117199 learned_addr learned my addr 172.21.15.146:0/1938117199 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f577cc00640 1 -- 172.21.15.146:0/1938117199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 msgr2=0x55b01a8a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f577cc00640 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a8a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f577cc00640 1 -- 172.21.15.146:0/1938117199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 msgr2=0x55b01a8a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f577cc00640 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a8a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f577cc00640 1 -- 172.21.15.146:0/1938117199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b01a4c90f0 con 0x55b01a89c850 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f577cc00640 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a895b70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55b01a892d60 tx=0x7f5770015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30889b986f1cb8aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.799+0000 7f5761600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5761600640 1 -- 172.21.15.146:0/1938117199 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577002a610 con 0x55b01a89c850 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5768053020 con 0x55b01a89c850 2024-04-24T08:30:44.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5761600640 1 -- 172.21.15.146:0/1938117199 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8542+0+0 (secure 0 0 0) 0x7f577002a7b0 con 0x55b01a89c850 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5761600640 1 -- 172.21.15.146:0/1938117199 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5770038050 con 0x55b01a89c850 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 10 client.27338 fetch_fsmap learned FSMap version 26 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b01a7bc810 con 0x55b01a89c850 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5761600640 1 -- 172.21.15.146:0/1938117199 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5768053020 con 0x55b01a89c850 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5761600640 1 -- 172.21.15.146:0/1938117199 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f57700333b0 con 0x55b01a89c850 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 10 client.27338 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 10 client.27338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 20 client.27338 populate_metadata read hostname 'smithi146' 2024-04-24T08:30:44.808 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a912ef0 0x55b01a915310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:30:44.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 4 client.27338 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:30:44.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.800+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b01a7fea80 con 0x55b01a912ef0 2024-04-24T08:30:44.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5777e00640 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a912ef0 0x55b01a915310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:30:44.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5777e00640 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a912ef0 0x55b01a915310 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ad549e9e67c8601 server_cookie=6a9c9fc0f707a664 in_seq=0 out_seq=0 2024-04-24T08:30:44.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5761600640 10 client.27338 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5761600640 1 -- 172.21.15.146:0/1938117199 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b01a7fea80 con 0x55b01a912ef0 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5761600640 10 client.27338 handle_command_reply: tid=0 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5781b99740 1 client.27338 shutdown 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 msgr2=0x55b01a895b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5781b99740 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a895b70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55b01a892d60 tx=0x7f5770015040 comp rx=0 tx=0).stop 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 shutdown_connections 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.801+0000 7f5781b99740 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55b01a912ef0 0x55b01a915310 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5781b99740 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b01a89c850 0x55b01a895b70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5781b99740 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b01a892180 0x55b01a8a4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5781b99740 1 --2- 172.21.15.146:0/1938117199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b01a891860 0x55b01a8a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 >> 172.21.15.146:0/1938117199 conn(0x55b01a88db80 msgr2=0x55b01a89afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5761600640 0 client.27338 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:30:44.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 shutdown_connections 2024-04-24T08:30:44.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5781b99740 1 -- 172.21.15.146:0/1938117199 wait complete. 2024-04-24T08:30:44.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f5781b99740 20 client.27338 trim_cache size 0 max 16384 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout: "abb816fc-29b7-4770-9eee-9cb220250648": { 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "abb816fc-29b7-4770-9eee-9cb220250648", 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:30:44.813 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:30:44.814 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:30:44.814 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:30:44.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f575c0a3e20 msgr2=0x7f575c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f577ec00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f575c0a3e20 0x7f575c0a62e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f57700099a0 tx=0x7f5770009500 comp rx=0 tx=0).stop 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778109520 msgr2=0x7f577810e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.802+0000 7f577ec00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778109520 0x7f577810e250 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5764005fd0 tx=0x7f576400dd20 comp rx=0 tx=0).stop 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 shutdown_connections 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f575c0a3e20 0x7f575c0a62e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f577810e8f0 0x7f57781ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5778109520 0x7f577810e250 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 --2- 172.21.15.146:0/754350310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5778104450 0x7f5778111090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:30:44.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 >> 172.21.15.146:0/754350310 conn(0x7f57780ffc70 msgr2=0x7f5778101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:30:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 shutdown_connections 2024-04-24T08:30:44.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:30:44.803+0000 7f577ec00640 1 -- 172.21.15.146:0/754350310 wait complete. 2024-04-24T08:30:44.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:44 smithi177 ceph-mon[33046]: pgmap v295: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:30:44.822 INFO:tasks.cephfs.filesystem:scrub status for tag:abb816fc-29b7-4770-9eee-9cb220250648 - {'path': '/', 'tag': 'abb816fc-29b7-4770-9eee-9cb220250648', 'options': 'recursive,force'} 2024-04-24T08:30:45.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:30:45.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:30:45.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:30:45.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:30:45.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.744 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:45 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:30:45.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:30:45.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:30:45.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:30:45.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:45 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:30:45.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:30:45.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:30:45.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:30:45.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:45.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:45 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:30:46.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:46 smithi146 ceph-mon[27805]: pgmap v296: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:46.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:46 smithi177 ceph-mon[33046]: pgmap v296: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:46.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:46 smithi153 ceph-mon[32746]: pgmap v296: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:30:48.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:48 smithi146 ceph-mon[27805]: pgmap v297: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:48.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:48 smithi177 ceph-mon[33046]: pgmap v297: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:48.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:48 smithi153 ceph-mon[32746]: pgmap v297: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:50.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:50 smithi146 ceph-mon[27805]: pgmap v298: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:50.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:50 smithi177 ceph-mon[33046]: pgmap v298: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:50.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:50 smithi153 ceph-mon[32746]: pgmap v298: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:52 smithi146 ceph-mon[27805]: pgmap v299: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:52 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:30:52.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:52 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:30:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:52 smithi177 ceph-mon[33046]: pgmap v299: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:52 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:30:52.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:52 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:30:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:52 smithi153 ceph-mon[32746]: pgmap v299: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:52 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:30:52.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:52 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:30:54.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:54 smithi146 ceph-mon[27805]: pgmap v300: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:54.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:54 smithi177 ceph-mon[33046]: pgmap v300: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:54.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:54 smithi153 ceph-mon[32746]: pgmap v300: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:56 smithi146 ceph-mon[27805]: pgmap v301: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:56 smithi177 ceph-mon[33046]: pgmap v301: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:56.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:56 smithi153 ceph-mon[32746]: pgmap v301: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:30:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:30:58 smithi177 ceph-mon[33046]: pgmap v302: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:58.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:30:58 smithi153 ceph-mon[32746]: pgmap v302: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:30:58.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:30:58 smithi146 ceph-mon[27805]: pgmap v302: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:31:00.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:00 smithi177 ceph-mon[33046]: pgmap v303: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:00.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:00 smithi153 ceph-mon[32746]: pgmap v303: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:00.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:00 smithi146 ceph-mon[27805]: pgmap v303: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:02 smithi177 ceph-mon[33046]: pgmap v304: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:02.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:02 smithi153 ceph-mon[32746]: pgmap v304: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:02.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:02 smithi146 ceph-mon[27805]: pgmap v304: 97 pgs: 97 active+clean; 74 MiB data, 697 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:04 smithi177 ceph-mon[33046]: pgmap v305: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:31:04.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:04 smithi153 ceph-mon[32746]: pgmap v305: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:31:04.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:04 smithi146 ceph-mon[27805]: pgmap v305: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:31:06.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:06 smithi177 ceph-mon[33046]: pgmap v306: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:31:06.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:06 smithi153 ceph-mon[32746]: pgmap v306: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:31:06.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:06 smithi146 ceph-mon[27805]: pgmap v306: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:31:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:08 smithi177 ceph-mon[33046]: pgmap v307: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:31:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:08 smithi177 ceph-mon[33046]: osdmap e98: 12 total, 12 up, 12 in 2024-04-24T08:31:08.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:08 smithi153 ceph-mon[32746]: pgmap v307: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:31:08.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:08 smithi153 ceph-mon[32746]: osdmap e98: 12 total, 12 up, 12 in 2024-04-24T08:31:08.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:08 smithi146 ceph-mon[27805]: pgmap v307: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:31:08.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:08 smithi146 ceph-mon[27805]: osdmap e98: 12 total, 12 up, 12 in 2024-04-24T08:31:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:10 smithi177 ceph-mon[33046]: pgmap v309: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:31:10.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:10 smithi153 ceph-mon[32746]: pgmap v309: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:31:10.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:10 smithi146 ceph-mon[27805]: pgmap v309: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:31:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:12 smithi177 ceph-mon[33046]: pgmap v310: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:31:12.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:12 smithi153 ceph-mon[32746]: pgmap v310: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:31:12.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:12 smithi146 ceph-mon[27805]: pgmap v310: 97 pgs: 97 active+clean; 74 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:31:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:14 smithi177 ceph-mon[33046]: pgmap v311: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:14.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:31:14.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:31:14.902 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:14 smithi146 ceph-mon[27805]: pgmap v311: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:14.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:14 smithi153 ceph-mon[32746]: pgmap v311: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:15.043 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 -- 172.21.15.146:0/3514262116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108580 msgr2=0x7fa794108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 --2- 172.21.15.146:0/3514262116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108580 0x7fa794108a00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa780009900 tx=0x7fa78002a400 comp rx=0 tx=0).stop 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 -- 172.21.15.146:0/3514262116 shutdown_connections 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 --2- 172.21.15.146:0/3514262116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa794108f40 0x7fa7941153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 --2- 172.21.15.146:0/3514262116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108580 0x7fa794108a00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 --2- 172.21.15.146:0/3514262116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa79410d500 0x7fa79410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 -- 172.21.15.146:0/3514262116 >> 172.21.15.146:0/3514262116 conn(0x7fa794077e20 msgr2=0x7fa794106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 -- 172.21.15.146:0/3514262116 shutdown_connections 2024-04-24T08:31:15.044 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.042+0000 7fa79b000640 1 -- 172.21.15.146:0/3514262116 wait complete. 2024-04-24T08:31:15.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.043+0000 7fa79b000640 1 Processor -- start 2024-04-24T08:31:15.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.043+0000 7fa79b000640 1 -- start start 2024-04-24T08:31:15.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa79b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa794108580 0x7fa79419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa79b000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108f40 0x7fa79419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa79b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79410d500 0x7fa7941a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa79b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa794118920 con 0x7fa79410d500 2024-04-24T08:31:15.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa79b000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941187a0 con 0x7fa794108f40 2024-04-24T08:31:15.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa79b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa794118aa0 con 0x7fa794108580 2024-04-24T08:31:15.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa794108580 0x7fa79419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa794108580 0x7fa79419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46146/0 (socket says 172.21.15.146:46146) 2024-04-24T08:31:15.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 -- 172.21.15.146:0/2418755269 learned_addr learned my addr 172.21.15.146:0/2418755269 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799200640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108f40 0x7fa79419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa79a600640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79410d500 0x7fa7941a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 -- 172.21.15.146:0/2418755269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108f40 msgr2=0x7fa79419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108f40 0x7fa79419f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 -- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79410d500 msgr2=0x7fa7941a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79410d500 0x7fa7941a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.050 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.044+0000 7fa799c00640 1 -- 172.21.15.146:0/2418755269 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7941a3de0 con 0x7fa794108580 2024-04-24T08:31:15.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.045+0000 7fa799c00640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa794108580 0x7fa79419ee40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa78800e9c0 tx=0x7fa78800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.045+0000 7fa792000640 1 -- 172.21.15.146:0/2418755269 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78800cd80 con 0x7fa794108580 2024-04-24T08:31:15.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.045+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7941a40d0 con 0x7fa794108580 2024-04-24T08:31:15.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.045+0000 7fa792000640 1 -- 172.21.15.146:0/2418755269 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7880035f0 con 0x7fa794108580 2024-04-24T08:31:15.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.045+0000 7fa792000640 1 -- 172.21.15.146:0/2418755269 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788010690 con 0x7fa794108580 2024-04-24T08:31:15.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.045+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7941ab960 con 0x7fa794108580 2024-04-24T08:31:15.056 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.048+0000 7fa79f399740 1 Processor -- start 2024-04-24T08:31:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.048+0000 7fa79f399740 1 -- start start 2024-04-24T08:31:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa792000640 1 -- 172.21.15.146:0/2418755269 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fa7880037a0 con 0x7fa794108580 2024-04-24T08:31:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79f399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6488a30 0x560ad6488e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79f399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6489350 0x560ad64934e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79f399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6495e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79f399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560ad63ee1f0 con 0x560ad6493a20 2024-04-24T08:31:15.057 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79f399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560ad6394350 con 0x560ad6489350 2024-04-24T08:31:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79f399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560ad63ea1b0 con 0x560ad6488a30 2024-04-24T08:31:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa799c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6489350 0x560ad64934e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6488a30 0x560ad6488e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa799200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6495e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa799c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6489350 0x560ad64934e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57692/0 (socket says 172.21.15.146:57692) 2024-04-24T08:31:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa79a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6488a30 0x560ad6488e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46160/0 (socket says 172.21.15.146:46160) 2024-04-24T08:31:15.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.049+0000 7fa799c00640 1 -- 172.21.15.146:0/253570005 learned_addr learned my addr 172.21.15.146:0/253570005 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa799c00640 1 -- 172.21.15.146:0/253570005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6488a30 msgr2=0x560ad6488e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa799c00640 1 --2- 172.21.15.146:0/253570005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6488a30 0x560ad6488e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa799c00640 1 -- 172.21.15.146:0/253570005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 msgr2=0x560ad6495e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa799c00640 1 --2- 172.21.15.146:0/253570005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6495e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa799c00640 1 -- 172.21.15.146:0/253570005 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ad63e8f50 con 0x560ad6489350 2024-04-24T08:31:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa799c00640 1 --2- 172.21.15.146:0/253570005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6489350 0x560ad64934e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa7880ce0f0 tx=0x7fa7880d4ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e79b6b98fb0024ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.059 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa786a00640 1 -- 172.21.15.146:0/253570005 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7880d8040 con 0x560ad6489350 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa786a00640 1 -- 172.21.15.146:0/253570005 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7880e2070 con 0x560ad6489350 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa786a00640 1 -- 172.21.15.146:0/253570005 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7880ddc40 con 0x560ad6489350 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.050+0000 7fa792000640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa77c0a3da0 0x7fa77c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa792000640 1 -- 172.21.15.146:0/2418755269 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8586+0+0 (secure 0 0 0) 0x7fa7880d0b00 con 0x7fa794108580 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79a600640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa77c0a3da0 0x7fa77c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79f399740 1 -- 172.21.15.146:0/253570005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6489350 msgr2=0x560ad64934e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79f399740 1 --2- 172.21.15.146:0/253570005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6489350 0x560ad64934e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa7880ce0f0 tx=0x7fa7880d4ba0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.060 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79f399740 1 -- 172.21.15.146:0/253570005 shutdown_connections 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79f399740 1 --2- 172.21.15.146:0/253570005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6495e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79f399740 1 --2- 172.21.15.146:0/253570005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6489350 0x560ad64934e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79f399740 1 --2- 172.21.15.146:0/253570005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6488a30 0x560ad6488e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.051+0000 7fa79f399740 1 -- 172.21.15.146:0/253570005 >> 172.21.15.146:0/253570005 conn(0x560ad6484d50 msgr2=0x560ad6485130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.052+0000 7fa79f399740 1 -- 172.21.15.146:0/253570005 shutdown_connections 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.052+0000 7fa79f399740 1 -- 172.21.15.146:0/253570005 wait complete. 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.052+0000 7fa79a600640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa77c0a3da0 0x7fa77c0a6260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa78c005fd0 tx=0x7fa78c009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.061 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.052+0000 7fa79f399740 1 -- start start 2024-04-24T08:31:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.052+0000 7fa79f399740 1 Processor -- start 2024-04-24T08:31:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79f399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad6490a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79f399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6490f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79f399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad6491480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79f399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x560ad64967b0 con 0x560ad6488a30 2024-04-24T08:31:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79f399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x560ad63a7690 con 0x560ad6493a20 2024-04-24T08:31:15.062 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79f399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560ad6496930 con 0x560ad6489350 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79f399740 10 client.0 _ll_register_callbacks cb 0x560ad63e3650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad6490a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6490f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79a600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad6491480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6490f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46162/0 (socket says 172.21.15.146:46162) 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 -- 172.21.15.146:0/1178956884 learned_addr learned my addr 172.21.15.146:0/1178956884 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 -- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 msgr2=0x560ad6491480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad6491480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 -- 172.21.15.146:0/1178956884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 msgr2=0x560ad6490a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad6490a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799200640 1 -- 172.21.15.146:0/1178956884 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ad63a6920 con 0x560ad6489350 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa799c00640 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad6490a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.053+0000 7fa79a600640 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad6491480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa799200640 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6490f40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa780033000 tx=0x7fa780004440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c466151f025451e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa77b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa77b400640 1 -- 172.21.15.146:0/1178956884 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa780035040 con 0x560ad6489350 2024-04-24T08:31:15.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7880e2070 con 0x560ad6489350 2024-04-24T08:31:15.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa77b400640 1 -- 172.21.15.146:0/1178956884 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8586+0+0 (secure 0 0 0) 0x7fa780005510 con 0x560ad6489350 2024-04-24T08:31:15.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa77b400640 1 -- 172.21.15.146:0/1178956884 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa7800046c0 con 0x560ad6489350 2024-04-24T08:31:15.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa79f399740 10 client.27352 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:15.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.054+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560ad61ca7d0 con 0x560ad6489350 2024-04-24T08:31:15.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa77b400640 1 -- 172.21.15.146:0/1178956884 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7880e2070 con 0x560ad6489350 2024-04-24T08:31:15.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa77b400640 1 -- 172.21.15.146:0/1178956884 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa780003c10 con 0x560ad6489350 2024-04-24T08:31:15.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa79f399740 10 client.27352 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa79f399740 10 client.27352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa79f399740 20 client.27352 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa79f399740 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad64a3d30 0x560ad64a6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa79f399740 4 client.27352 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.055+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ad63fbc10 con 0x560ad64a3d30 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.056+0000 7fa799c00640 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad64a3d30 0x560ad64a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.056+0000 7fa799c00640 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad64a3d30 0x560ad64a6140 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cef31d9f3a7251ba server_cookie=ef69ba66cef65e7d in_seq=0 out_seq=0 2024-04-24T08:31:15.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.056+0000 7fa77b400640 10 client.27352 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:15.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa77b400640 1 -- 172.21.15.146:0/1178956884 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ad63fbc10 con 0x560ad64a3d30 2024-04-24T08:31:15.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa77b400640 10 client.27352 handle_command_reply: tid=0 2024-04-24T08:31:15.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 client.27352 shutdown 2024-04-24T08:31:15.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 msgr2=0x560ad6490f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6490f40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa780033000 tx=0x7fa780004440 comp rx=0 tx=0).stop 2024-04-24T08:31:15.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 shutdown_connections 2024-04-24T08:31:15.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad64a3d30 0x560ad64a6140 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6490f40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa77b400640 0 client.27352 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:15.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad6491480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 --2- 172.21.15.146:0/1178956884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad6490a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 >> 172.21.15.146:0/1178956884 conn(0x560ad6484d50 msgr2=0x560ad6483fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 shutdown_connections 2024-04-24T08:31:15.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 1 -- 172.21.15.146:0/1178956884 wait complete. 2024-04-24T08:31:15.070 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.057+0000 7fa79f399740 20 client.27352 trim_cache size 0 max 16384 2024-04-24T08:31:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.076+0000 7fa79f399740 1 Processor -- start 2024-04-24T08:31:15.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.076+0000 7fa79f399740 1 -- start start 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa79f399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad649f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa79f399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad64910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa79f399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad64aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa79f399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b6b10 con 0x560ad6488a30 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa79f399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b43e0 con 0x560ad6493a20 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa79f399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b2250 con 0x560ad6489350 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad649f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad649f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50916/0 (socket says 172.21.15.146:50916) 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad64910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 -- 172.21.15.146:0/1883802455 learned_addr learned my addr 172.21.15.146:0/1883802455 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa79a600640 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad64aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 -- 172.21.15.146:0/1883802455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 msgr2=0x560ad64910d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad64910d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 -- 172.21.15.146:0/1883802455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 msgr2=0x560ad64aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad64aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.077+0000 7fa799c00640 1 -- 172.21.15.146:0/1883802455 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ad61003f0 con 0x560ad6488a30 2024-04-24T08:31:15.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.078+0000 7fa799c00640 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad649f670 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa7880993d0 tx=0x7fa788003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45513bac9b06981 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.078+0000 7fa786000640 1 -- 172.21.15.146:0/1883802455 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78809c7d0 con 0x560ad6488a30 2024-04-24T08:31:15.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.078+0000 7fa786000640 1 -- 172.21.15.146:0/1883802455 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7880e2040 con 0x560ad6488a30 2024-04-24T08:31:15.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.078+0000 7fa786000640 1 -- 172.21.15.146:0/1883802455 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7880d4480 con 0x560ad6488a30 2024-04-24T08:31:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.078+0000 7fa79f399740 1 -- 172.21.15.146:0/1883802455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 msgr2=0x560ad649f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.078+0000 7fa79f399740 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad649f670 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa7880993d0 tx=0x7fa788003f10 comp rx=0 tx=0).stop 2024-04-24T08:31:15.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 -- 172.21.15.146:0/1883802455 shutdown_connections 2024-04-24T08:31:15.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad64910d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6493a20 0x560ad64aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 --2- 172.21.15.146:0/1883802455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6488a30 0x560ad649f670 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 -- 172.21.15.146:0/1883802455 >> 172.21.15.146:0/1883802455 conn(0x560ad6484d50 msgr2=0x560ad64909c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 -- 172.21.15.146:0/1883802455 shutdown_connections 2024-04-24T08:31:15.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 -- 172.21.15.146:0/1883802455 wait complete. 2024-04-24T08:31:15.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 -- start start 2024-04-24T08:31:15.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.079+0000 7fa79f399740 1 Processor -- start 2024-04-24T08:31:15.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79f399740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6488a30 0x560ad649c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79f399740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6497430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79f399740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6497b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79f399740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b28a0 con 0x560ad6493a20 2024-04-24T08:31:15.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79f399740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b3d10 con 0x560ad6488a30 2024-04-24T08:31:15.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79f399740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7941b2f70 con 0x560ad6489350 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79f399740 10 client.0 _ll_register_callbacks cb 0x7fa780009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6488a30 0x560ad649c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6488a30 0x560ad649c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57720/0 (socket says 172.21.15.146:57720) 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 -- 172.21.15.146:0/1747434356 learned_addr learned my addr 172.21.15.146:0/1747434356 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799200640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6497430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79a600640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6497b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 -- 172.21.15.146:0/1747434356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 msgr2=0x560ad6497430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6497430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 -- 172.21.15.146:0/1747434356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 msgr2=0x560ad6497b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6497b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa799c00640 1 -- 172.21.15.146:0/1747434356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ad60c00f0 con 0x560ad6488a30 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.080+0000 7fa79a600640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6497b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa799c00640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6488a30 0x560ad649c0c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa7880d0040 tx=0x7fa7880d4770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=855310530b5f854b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa77b400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa77b400640 1 -- 172.21.15.146:0/1747434356 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7880b5070 con 0x560ad6488a30 2024-04-24T08:31:15.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa77b400640 1 -- 172.21.15.146:0/1747434356 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8586+0+0 (secure 0 0 0) 0x7fa7880d7070 con 0x560ad6488a30 2024-04-24T08:31:15.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7880e2040 con 0x560ad6488a30 2024-04-24T08:31:15.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa77b400640 1 -- 172.21.15.146:0/1747434356 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa78809c360 con 0x560ad6488a30 2024-04-24T08:31:15.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa79f399740 10 client.27356 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:15.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560ad63b3810 con 0x560ad6488a30 2024-04-24T08:31:15.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa77b400640 1 -- 172.21.15.146:0/1747434356 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7880e2040 con 0x560ad6488a30 2024-04-24T08:31:15.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.081+0000 7fa77b400640 1 -- 172.21.15.146:0/1747434356 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa7880ce430 con 0x560ad6488a30 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa79f399740 10 client.27356 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa79f399740 10 client.27356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa79f399740 20 client.27356 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa79f399740 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad650a370 0x560ad650c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa79f399740 4 client.27356 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560ad63f5a80 con 0x560ad650a370 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa799200640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad650a370 0x560ad650c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa799200640 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad650a370 0x560ad650c760 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f993ad1eb64aba1a server_cookie=6256443c91dc5c1a in_seq=0 out_seq=0 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa77b400640 10 client.27356 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.082+0000 7fa77b400640 1 -- 172.21.15.146:0/1747434356 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560ad63f5a80 con 0x560ad650a370 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa77b400640 10 client.27356 handle_command_reply: tid=0 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 client.27356 shutdown 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6488a30 msgr2=0x560ad649c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6488a30 0x560ad649c0c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa7880d0040 tx=0x7fa7880d4770 comp rx=0 tx=0).stop 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 shutdown_connections 2024-04-24T08:31:15.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x560ad650a370 0x560ad650c760 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560ad6489350 0x560ad6497430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ad6493a20 0x560ad6497b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 --2- 172.21.15.146:0/1747434356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x560ad6488a30 0x560ad649c0c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa77b400640 0 client.27356 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 >> 172.21.15.146:0/1747434356 conn(0x560ad6484d50 msgr2=0x560ad6491230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 shutdown_connections 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 1 -- 172.21.15.146:0/1747434356 wait complete. 2024-04-24T08:31:15.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.083+0000 7fa79f399740 20 client.27356 trim_cache size 0 max 16384 2024-04-24T08:31:15.094 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:31:15.094 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:31:15.094 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:31:15.094 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:31:15.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa77c0a3da0 msgr2=0x7fa77c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa77c0a3da0 0x7fa77c0a6260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa78c005fd0 tx=0x7fa78c009500 comp rx=0 tx=0).stop 2024-04-24T08:31:15.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa794108580 msgr2=0x7fa79419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa794108580 0x7fa79419ee40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa78800e9c0 tx=0x7fa78800ee80 comp rx=0 tx=0).stop 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 shutdown_connections 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa77c0a3da0 0x7fa77c0a6260 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa79410d500 0x7fa7941a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa794108f40 0x7fa79419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 --2- 172.21.15.146:0/2418755269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa794108580 0x7fa79419ee40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 >> 172.21.15.146:0/2418755269 conn(0x7fa794077e20 msgr2=0x7fa794078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 shutdown_connections 2024-04-24T08:31:15.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.084+0000 7fa79b000640 1 -- 172.21.15.146:0/2418755269 wait complete. 2024-04-24T08:31:15.103 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:31:15.104 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:31:15.298 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 -- 172.21.15.146:0/1974172028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 msgr2=0x7f080c104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 --2- 172.21.15.146:0/1974172028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 0x7f080c104650 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f07fc009be0 tx=0x7f07fc02a6c0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 -- 172.21.15.146:0/1974172028 shutdown_connections 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 --2- 172.21.15.146:0/1974172028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c109340 0x7f080c110f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 --2- 172.21.15.146:0/1974172028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f080c104c20 0x7f080c108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 --2- 172.21.15.146:0/1974172028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 0x7f080c104650 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 -- 172.21.15.146:0/1974172028 >> 172.21.15.146:0/1974172028 conn(0x7f080c0ffad0 msgr2=0x7f080c101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 -- 172.21.15.146:0/1974172028 shutdown_connections 2024-04-24T08:31:15.299 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.297+0000 7f080b200640 1 -- 172.21.15.146:0/1974172028 wait complete. 2024-04-24T08:31:15.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.298+0000 7f080b200640 1 Processor -- start 2024-04-24T08:31:15.301 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.298+0000 7f080b200640 1 -- start start 2024-04-24T08:31:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 0x7f080c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f080c109340 0x7f080c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c19a780 0x7f080c19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f080c1143a0 con 0x7f080c19a780 2024-04-24T08:31:15.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f080c114220 con 0x7f080c109340 2024-04-24T08:31:15.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f080c114520 con 0x7f080c104270 2024-04-24T08:31:15.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c19a780 0x7f080c19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f0801400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f080c109340 0x7f080c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c19a780 0x7f080c19ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50946/0 (socket says 172.21.15.146:50946) 2024-04-24T08:31:15.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f080a800640 1 -- 172.21.15.146:0/3809961378 learned_addr learned my addr 172.21.15.146:0/3809961378 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.299+0000 7f0809e00640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 0x7f080c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0801400640 1 -- 172.21.15.146:0/3809961378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 msgr2=0x7f080c1a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0801400640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 0x7f080c1a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0801400640 1 -- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c19a780 msgr2=0x7f080c19ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0801400640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c19a780 0x7f080c19ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0801400640 1 -- 172.21.15.146:0/3809961378 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080c19b380 con 0x7f080c109340 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f080a800640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c19a780 0x7f080c19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:15.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0801400640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f080c109340 0x7f080c1a0bb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f07f4005fd0 tx=0x7f07f400ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0803400640 1 -- 172.21.15.146:0/3809961378 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f400cc90 con 0x7f080c109340 2024-04-24T08:31:15.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f080c1a7410 con 0x7f080c109340 2024-04-24T08:31:15.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0803400640 1 -- 172.21.15.146:0/3809961378 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07f400ee70 con 0x7f080c109340 2024-04-24T08:31:15.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f0803400640 1 -- 172.21.15.146:0/3809961378 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4004a00 con 0x7f080c109340 2024-04-24T08:31:15.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.300+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f080c1a7810 con 0x7f080c109340 2024-04-24T08:31:15.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.302+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f080c105740 con 0x7f080c109340 2024-04-24T08:31:15.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.304+0000 7f0803400640 1 -- 172.21.15.146:0/3809961378 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f07f4004ba0 con 0x7f080c109340 2024-04-24T08:31:15.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.305+0000 7f0803400640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f07f00a3d20 0x7f07f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.305+0000 7f0803400640 1 -- 172.21.15.146:0/3809961378 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8586+0+0 (secure 0 0 0) 0x7f07f40d07e0 con 0x7f080c109340 2024-04-24T08:31:15.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.306+0000 7f0809e00640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f07f00a3d20 0x7f07f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.306+0000 7f0809e00640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f07f00a3d20 0x7f07f00a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f07fc000f80 tx=0x7f07fc005c10 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.307+0000 7f0803400640 1 -- 172.21.15.146:0/3809961378 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07f409ce70 con 0x7f080c109340 2024-04-24T08:31:15.559 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.558+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f080c069190 con 0x7f080c109340 2024-04-24T08:31:15.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.559+0000 7f0803400640 1 -- 172.21.15.146:0/3809961378 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f07f409c770 con 0x7f080c109340 2024-04-24T08:31:15.560 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:31:15.561 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:31:15.561 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:31:15.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f07f00a3d20 msgr2=0x7f07f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f07f00a3d20 0x7f07f00a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f07fc000f80 tx=0x7f07fc005c10 comp rx=0 tx=0).stop 2024-04-24T08:31:15.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f080c109340 msgr2=0x7f080c1a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f080c109340 0x7f080c1a0bb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f07f4005fd0 tx=0x7f07f400ec20 comp rx=0 tx=0).stop 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 shutdown_connections 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f07f00a3d20 0x7f07f00a61e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f080c19a780 0x7f080c19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f080c109340 0x7f080c1a0bb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 --2- 172.21.15.146:0/3809961378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f080c104270 0x7f080c1a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.562+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 >> 172.21.15.146:0/3809961378 conn(0x7f080c0ffad0 msgr2=0x7f080c100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.564 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.563+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 shutdown_connections 2024-04-24T08:31:15.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.563+0000 7f080b200640 1 -- 172.21.15.146:0/3809961378 wait complete. 2024-04-24T08:31:15.583 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:31:15.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.788+0000 7fd1cb200640 1 -- 172.21.15.146:0/2742645341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 msgr2=0x7fd1cc1153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.788+0000 7fd1cb200640 1 --2- 172.21.15.146:0/2742645341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1153b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009cd0 tx=0x7fd1c402a8f0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.788+0000 7fd1cb200640 1 -- 172.21.15.146:0/2742645341 shutdown_connections 2024-04-24T08:31:15.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.789+0000 7fd1cb200640 1 --2- 172.21.15.146:0/2742645341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1cc108580 0x7fd1cc108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.789+0000 7fd1cb200640 1 --2- 172.21.15.146:0/2742645341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1153b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.789+0000 7fd1cb200640 1 --2- 172.21.15.146:0/2742645341 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1cc10d500 0x7fd1cc10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.789+0000 7fd1cb200640 1 -- 172.21.15.146:0/2742645341 >> 172.21.15.146:0/2742645341 conn(0x7fd1cc077e20 msgr2=0x7fd1cc106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.789+0000 7fd1cb200640 1 -- 172.21.15.146:0/2742645341 shutdown_connections 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.789+0000 7fd1cb200640 1 -- 172.21.15.146:0/2742645341 wait complete. 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 Processor -- start 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 -- start start 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1cc108580 0x7fd1cc1a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1cc10d500 0x7fd1cc19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc118900 con 0x7fd1cc108f40 2024-04-24T08:31:15.793 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc118780 con 0x7fd1cc108580 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1cb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc118a80 con 0x7fd1cc10d500 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1c1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1c1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1a4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50964/0 (socket says 172.21.15.146:50964) 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1c1400640 1 -- 172.21.15.146:0/518302116 learned_addr learned my addr 172.21.15.146:0/518302116 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.790+0000 7fd1ca800640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1cc10d500 0x7fd1cc19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1cc108580 0x7fd1cc1a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c1400640 1 -- 172.21.15.146:0/518302116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1cc10d500 msgr2=0x7fd1cc19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c1400640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1cc10d500 0x7fd1cc19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c1400640 1 -- 172.21.15.146:0/518302116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1cc108580 msgr2=0x7fd1cc1a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c1400640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1cc108580 0x7fd1cc1a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c1400640 1 -- 172.21.15.146:0/518302116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc19f440 con 0x7fd1cc108f40 2024-04-24T08:31:15.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c1400640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1a4fb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400d950 tx=0x7fd1b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c3400640 1 -- 172.21.15.146:0/518302116 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4017030 con 0x7fd1cc108f40 2024-04-24T08:31:15.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c3400640 1 -- 172.21.15.146:0/518302116 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1b4004020 con 0x7fd1cc108f40 2024-04-24T08:31:15.796 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.791+0000 7fd1c3400640 1 -- 172.21.15.146:0/518302116 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40047a0 con 0x7fd1cc108f40 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.792+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1cc19f730 con 0x7fd1cc108f40 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.792+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1cc1ab9f0 con 0x7fd1cc108f40 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 Processor -- start 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 -- start start 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca13f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55b9cc9791d0 con 0x55b9cca14460 2024-04-24T08:31:15.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b9cc91f450 con 0x55b9cca1eb30 2024-04-24T08:31:15.798 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.795+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b9cc975190 con 0x55b9cca13b40 2024-04-24T08:31:15.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca13f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1c3400640 1 -- 172.21.15.146:0/518302116 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fd1b40041d0 con 0x7fd1cc108f40 2024-04-24T08:31:15.800 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca13f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46224/0 (socket says 172.21.15.146:46224) 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 -- 172.21.15.146:0/4206138857 learned_addr learned my addr 172.21.15.146:0/4206138857 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 -- 172.21.15.146:0/4206138857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 msgr2=0x55b9cca20f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 -- 172.21.15.146:0/4206138857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 msgr2=0x55b9cca1e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.796+0000 7fd1ca800640 1 -- 172.21.15.146:0/4206138857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9cc9dc310 con 0x55b9cca13b40 2024-04-24T08:31:15.801 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.797+0000 7fd1ca800640 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca13f20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009cd0 tx=0x7fd1c4005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d17907a97c1b50 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.797+0000 7fd1bb000640 1 -- 172.21.15.146:0/4206138857 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4003970 con 0x55b9cca13b40 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.797+0000 7fd1bb000640 1 -- 172.21.15.146:0/4206138857 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c4036030 con 0x55b9cca13b40 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.797+0000 7fd1bb000640 1 -- 172.21.15.146:0/4206138857 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40044a0 con 0x55b9cca13b40 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.797+0000 7fd1c3400640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd1b00a3da0 0x7fd1b00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.797+0000 7fd1c3400640 1 -- 172.21.15.146:0/518302116 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8586+0+0 (secure 0 0 0) 0x7fd1b40d0140 con 0x7fd1cc108f40 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd1b00a3da0 0x7fd1b00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/4206138857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 msgr2=0x55b9cca13f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca13f20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009cd0 tx=0x7fd1c4005fa0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/4206138857 shutdown_connections 2024-04-24T08:31:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/4206138857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca13f20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/4206138857 >> 172.21.15.146:0/4206138857 conn(0x55b9cca0fe60 msgr2=0x55b9cca10240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/4206138857 shutdown_connections 2024-04-24T08:31:15.803 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/4206138857 wait complete. 2024-04-24T08:31:15.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd1b00a3da0 0x7fd1b00a6260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55b9cca15100 tx=0x7fd1bc005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.798+0000 7fd1d3a2a740 1 -- start start 2024-04-24T08:31:15.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 1 Processor -- start 2024-04-24T08:31:15.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca13b40 0x55b9cca1e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca14460 0x55b9cca1b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca1eb30 0x55b9cca1c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55b9cca21960 con 0x55b9cca1eb30 2024-04-24T08:31:15.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55b9cc932810 con 0x55b9cca13b40 2024-04-24T08:31:15.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b9cca21ae0 con 0x55b9cca14460 2024-04-24T08:31:15.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1d3a2a740 10 client.0 _ll_register_callbacks cb 0x55b9cc900140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:15.805 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1c9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca13b40 0x55b9cca1e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1ca800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca1eb30 0x55b9cca1c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca14460 0x55b9cca1b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.799+0000 7fd1c9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca13b40 0x55b9cca1e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57776/0 (socket says 172.21.15.146:57776) 2024-04-24T08:31:15.806 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c9e00640 1 -- 172.21.15.146:0/1900093824 learned_addr learned my addr 172.21.15.146:0/1900093824 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c1400640 1 -- 172.21.15.146:0/1900093824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca13b40 msgr2=0x55b9cca1e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c1400640 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca13b40 0x55b9cca1e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c1400640 1 -- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca1eb30 msgr2=0x55b9cca1c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c1400640 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca1eb30 0x55b9cca1c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c1400640 1 -- 172.21.15.146:0/1900093824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9cc9433f0 con 0x55b9cca14460 2024-04-24T08:31:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1ca800640 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca1eb30 0x55b9cca1c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:15.809 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca13b40 0x55b9cca1e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1c1400640 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca14460 0x55b9cca1b8d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cd850 tx=0x7fd1b409bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55e4bf1ae1eb6d82 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1af400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.800+0000 7fd1af400640 1 -- 172.21.15.146:0/1900093824 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b409c0a0 con 0x55b9cca14460 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c4036030 con 0x55b9cca14460 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1af400640 1 -- 172.21.15.146:0/1900093824 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8586+0+0 (secure 0 0 0) 0x7fd1b40cdad0 con 0x55b9cca14460 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1af400640 1 -- 172.21.15.146:0/1900093824 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd1b40a0450 con 0x55b9cca14460 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 10 client.27376 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:15.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b9cc68b3f0 con 0x55b9cca14460 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1af400640 1 -- 172.21.15.146:0/1900093824 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c4036030 con 0x55b9cca14460 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1af400640 1 -- 172.21.15.146:0/1900093824 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd1b40b2ce0 con 0x55b9cca14460 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 10 client.27376 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 10 client.27376 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 20 client.27376 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca2ed30 0x55b9cca31120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 4 client.27376 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:15.811 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.801+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9cc986bf0 con 0x55b9cca2ed30 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.802+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca2ed30 0x55b9cca31120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.802+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca2ed30 0x55b9cca31120 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aadca7154d4ac785 server_cookie=e94fdd97cf7e55bf in_seq=0 out_seq=0 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.802+0000 7fd1af400640 10 client.27376 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1af400640 1 -- 172.21.15.146:0/1900093824 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9cc986bf0 con 0x55b9cca2ed30 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1af400640 10 client.27376 handle_command_reply: tid=0 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 client.27376 shutdown 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca14460 msgr2=0x55b9cca1b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca14460 0x55b9cca1b8d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40cd850 tx=0x7fd1b409bfa0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 shutdown_connections 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca2ed30 0x55b9cca31120 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca14460 0x55b9cca1b8d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca1eb30 0x55b9cca1c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1900093824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca13b40 0x55b9cca1e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 >> 172.21.15.146:0/1900093824 conn(0x55b9cca0fe60 msgr2=0x55b9cca15b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.803+0000 7fd1af400640 0 client.27376 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:15.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.804+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 shutdown_connections 2024-04-24T08:31:15.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.804+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1900093824 wait complete. 2024-04-24T08:31:15.814 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.804+0000 7fd1d3a2a740 20 client.27376 trim_cache size 0 max 16384 2024-04-24T08:31:15.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 Processor -- start 2024-04-24T08:31:15.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 -- start start 2024-04-24T08:31:15.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca13b40 0x55b9cca2a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca14460 0x55b9cca1e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca1eb30 0x55b9cca368e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b69d0 con 0x55b9cca13b40 2024-04-24T08:31:15.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b42a0 con 0x55b9cca14460 2024-04-24T08:31:15.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b2110 con 0x55b9cca1eb30 2024-04-24T08:31:15.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1c9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca13b40 0x55b9cca2a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1c1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca14460 0x55b9cca1e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1ca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca1eb30 0x55b9cca368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1c9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca13b40 0x55b9cca2a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51004/0 (socket says 172.21.15.146:51004) 2024-04-24T08:31:15.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1c1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca14460 0x55b9cca1e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57790/0 (socket says 172.21.15.146:57790) 2024-04-24T08:31:15.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.819+0000 7fd1c9e00640 1 -- 172.21.15.146:0/3330061729 learned_addr learned my addr 172.21.15.146:0/3330061729 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1c1400640 1 -- 172.21.15.146:0/3330061729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca1eb30 msgr2=0x55b9cca368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1c1400640 1 --2- 172.21.15.146:0/3330061729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca1eb30 0x55b9cca368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1c1400640 1 -- 172.21.15.146:0/3330061729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca13b40 msgr2=0x55b9cca2a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1c1400640 1 --2- 172.21.15.146:0/3330061729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca13b40 0x55b9cca2a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.826 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1c1400640 1 -- 172.21.15.146:0/3330061729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9cca14290 con 0x55b9cca14460 2024-04-24T08:31:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1c1400640 1 --2- 172.21.15.146:0/3330061729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca14460 0x55b9cca1e110 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40d0110 tx=0x7fd1b40045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ef0a3fc157b0479 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1ba600640 1 -- 172.21.15.146:0/3330061729 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4098150 con 0x55b9cca14460 2024-04-24T08:31:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1ba600640 1 -- 172.21.15.146:0/3330061729 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1b409c0a0 con 0x55b9cca14460 2024-04-24T08:31:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.820+0000 7fd1ba600640 1 -- 172.21.15.146:0/3330061729 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40b0b40 con 0x55b9cca14460 2024-04-24T08:31:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/3330061729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca14460 msgr2=0x55b9cca1e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/3330061729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca14460 0x55b9cca1e110 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40d0110 tx=0x7fd1b40045b0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/3330061729 shutdown_connections 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/3330061729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca1eb30 0x55b9cca368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/3330061729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca14460 0x55b9cca1e110 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/3330061729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca13b40 0x55b9cca2a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/3330061729 >> 172.21.15.146:0/3330061729 conn(0x55b9cca0fe60 msgr2=0x55b9cca16bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/3330061729 shutdown_connections 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.821+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/3330061729 wait complete. 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 -- start start 2024-04-24T08:31:15.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 Processor -- start 2024-04-24T08:31:15.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca1f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b2760 con 0x55b9cca14460 2024-04-24T08:31:15.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b3bd0 con 0x55b9cca1eb30 2024-04-24T08:31:15.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.822+0000 7fd1d3a2a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b2e30 con 0x55b9cca13b40 2024-04-24T08:31:15.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1d3a2a740 10 client.0 _ll_register_callbacks cb 0x7fd1b4098410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1c9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca1f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1c1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1c9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca1f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46252/0 (socket says 172.21.15.146:46252) 2024-04-24T08:31:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1ca800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1c1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51010/0 (socket says 172.21.15.146:51010) 2024-04-24T08:31:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1c9e00640 1 -- 172.21.15.146:0/1674636012 learned_addr learned my addr 172.21.15.146:0/1674636012 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:15.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1ca800640 1 -- 172.21.15.146:0/1674636012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 msgr2=0x55b9cca1f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1ca800640 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca1f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1ca800640 1 -- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 msgr2=0x55b9cca1fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:15.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1ca800640 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1ca800640 1 -- 172.21.15.146:0/1674636012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9cc7557d0 con 0x55b9cca1eb30 2024-04-24T08:31:15.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.823+0000 7fd1c1400640 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:15.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1ca800640 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20260 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4005e30 tx=0x7fd1c40040e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f55b02f47ef9acd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1af400640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1af400640 1 -- 172.21.15.146:0/1674636012 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4030cb0 con 0x55b9cca1eb30 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1b409c0a0 con 0x55b9cca1eb30 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1af400640 1 -- 172.21.15.146:0/1674636012 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8586+0+0 (secure 0 0 0) 0x7fd1c4003970 con 0x55b9cca1eb30 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1af400640 1 -- 172.21.15.146:0/1674636012 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd1c4030e50 con 0x55b9cca1eb30 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1d3a2a740 10 client.27377 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b9cca951a0 con 0x55b9cca1eb30 2024-04-24T08:31:15.832 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.824+0000 7fd1af400640 1 -- 172.21.15.146:0/1674636012 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1b409c0a0 con 0x55b9cca1eb30 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1af400640 1 -- 172.21.15.146:0/1674636012 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd1c4043070 con 0x55b9cca1eb30 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1d3a2a740 10 client.27377 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1d3a2a740 10 client.27377 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1d3a2a740 20 client.27377 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca95500 0x55b9cca97920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1d3a2a740 4 client.27377 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9cc980a60 con 0x55b9cca95500 2024-04-24T08:31:15.833 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.825+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca95500 0x55b9cca97920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:15.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1af400640 10 client.27377 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:15.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1c9e00640 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca95500 0x55b9cca97920 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ce6b068e6f725d6 server_cookie=e91e0782c089b47a in_seq=0 out_seq=0 2024-04-24T08:31:15.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1af400640 1 -- 172.21.15.146:0/1674636012 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b9cc980a60 con 0x55b9cca95500 2024-04-24T08:31:15.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1af400640 10 client.27377 handle_command_reply: tid=0 2024-04-24T08:31:15.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 client.27377 shutdown 2024-04-24T08:31:15.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 msgr2=0x55b9cca20260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.835 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20260 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4005e30 tx=0x7fd1c40040e0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 shutdown_connections 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x55b9cca95500 0x55b9cca97920 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55b9cca1eb30 0x55b9cca20260 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b9cca14460 0x55b9cca1fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 --2- 172.21.15.146:0/1674636012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b9cca13b40 0x55b9cca1f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1af400640 0 client.27377 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 >> 172.21.15.146:0/1674636012 conn(0x55b9cca0fe60 msgr2=0x55b9cca0e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.836 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 shutdown_connections 2024-04-24T08:31:15.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.826+0000 7fd1d3a2a740 1 -- 172.21.15.146:0/1674636012 wait complete. 2024-04-24T08:31:15.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1d3a2a740 20 client.27377 trim_cache size 0 max 16384 2024-04-24T08:31:15.837 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:31:15.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd1b00a3da0 msgr2=0x7fd1b00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd1b00a3da0 0x7fd1b00a6260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55b9cca15100 tx=0x7fd1bc005f90 comp rx=0 tx=0).stop 2024-04-24T08:31:15.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 msgr2=0x7fd1cc1a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:15.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1a4fb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400d950 tx=0x7fd1b400de10 comp rx=0 tx=0).stop 2024-04-24T08:31:15.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 shutdown_connections 2024-04-24T08:31:15.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fd1b00a3da0 0x7fd1b00a6260 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1cc10d500 0x7fd1cc19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1cc108f40 0x7fd1cc1a4fb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 --2- 172.21.15.146:0/518302116 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1cc108580 0x7fd1cc1a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:15.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 >> 172.21.15.146:0/518302116 conn(0x7fd1cc077e20 msgr2=0x7fd1cc078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:15.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.827+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 shutdown_connections 2024-04-24T08:31:15.839 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:15.828+0000 7fd1cb200640 1 -- 172.21.15.146:0/518302116 wait complete. 2024-04-24T08:31:15.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:31:16.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 -- 172.21.15.146:0/2625173784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1064108fd0 msgr2=0x7f1064115490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 --2- 172.21.15.146:0/2625173784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1064108fd0 0x7f1064115490 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f105c009cd0 tx=0x7f105c02a8e0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 -- 172.21.15.146:0/2625173784 shutdown_connections 2024-04-24T08:31:16.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 --2- 172.21.15.146:0/2625173784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1064108fd0 0x7f1064115490 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 --2- 172.21.15.146:0/2625173784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 0x7f1064108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 --2- 172.21.15.146:0/2625173784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f106410e660 0x7f106410ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 -- 172.21.15.146:0/2625173784 >> 172.21.15.146:0/2625173784 conn(0x7f1064076540 msgr2=0x7f1064078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 -- 172.21.15.146:0/2625173784 shutdown_connections 2024-04-24T08:31:16.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.058+0000 7f106b000640 1 -- 172.21.15.146:0/2625173784 wait complete. 2024-04-24T08:31:16.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 Processor -- start 2024-04-24T08:31:16.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 -- start start 2024-04-24T08:31:16.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 0x7f106419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064108fd0 0x7f106419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106410e660 0x7f10641a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f10641189a0 con 0x7f1064108610 2024-04-24T08:31:16.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1064118820 con 0x7f1064108fd0 2024-04-24T08:31:16.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.059+0000 7f106b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1064118b20 con 0x7f106410e660 2024-04-24T08:31:16.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 0x7f106419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f105b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064108fd0 0x7f106419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 0x7f106419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51032/0 (socket says 172.21.15.146:51032) 2024-04-24T08:31:16.067 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 -- 172.21.15.146:0/1310949856 learned_addr learned my addr 172.21.15.146:0/1310949856 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1069000640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106410e660 0x7f10641a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 -- 172.21.15.146:0/1310949856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106410e660 msgr2=0x7f10641a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106410e660 0x7f10641a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 -- 172.21.15.146:0/1310949856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064108fd0 msgr2=0x7f106419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064108fd0 0x7f106419f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 -- 172.21.15.146:0/1310949856 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10641a3de0 con 0x7f1064108610 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f105b400640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064108fd0 0x7f106419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:16.068 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.060+0000 7f1063e00640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 0x7f106419ee40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f105400e9c0 tx=0x7f105400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.061+0000 7f1061600640 1 -- 172.21.15.146:0/1310949856 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105400cd80 con 0x7f1064108610 2024-04-24T08:31:16.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.061+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10641a40d0 con 0x7f1064108610 2024-04-24T08:31:16.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.061+0000 7f1061600640 1 -- 172.21.15.146:0/1310949856 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10540035f0 con 0x7f1064108610 2024-04-24T08:31:16.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.061+0000 7f1061600640 1 -- 172.21.15.146:0/1310949856 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054010690 con 0x7f1064108610 2024-04-24T08:31:16.069 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.061+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10641ab960 con 0x7f1064108610 2024-04-24T08:31:16.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.064+0000 7f106de1c740 1 Processor -- start 2024-04-24T08:31:16.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.064+0000 7f106de1c740 1 -- start start 2024-04-24T08:31:16.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f106de1c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f106de1c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f106de1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f106de1c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c3184131d0 con 0x55c3184ad950 2024-04-24T08:31:16.076 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f106de1c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c3183b9450 con 0x55c3184ae270 2024-04-24T08:31:16.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f106de1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c31840f190 con 0x55c3184b8940 2024-04-24T08:31:16.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f1061600640 1 -- 172.21.15.146:0/1310949856 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f10540037a0 con 0x7f1064108610 2024-04-24T08:31:16.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51040/0 (socket says 172.21.15.146:51040) 2024-04-24T08:31:16.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f1069000640 1 -- 172.21.15.146:0/9362102 learned_addr learned my addr 172.21.15.146:0/9362102 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f1063e00640 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.077 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.065+0000 7f105b400640 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1069000640 1 -- 172.21.15.146:0/9362102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 msgr2=0x55c3184bad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1069000640 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1069000640 1 -- 172.21.15.146:0/9362102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 msgr2=0x55c3184b8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1069000640 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1069000640 1 -- 172.21.15.146:0/9362102 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c318476310 con 0x55c3184ad950 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f105b400640 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1069000640 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184add30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f105c009cd0 tx=0x7f105c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90789e977c3b8b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1059600640 1 -- 172.21.15.146:0/9362102 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c048070 con 0x55c3184ad950 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1059600640 1 -- 172.21.15.146:0/9362102 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f105c036070 con 0x55c3184ad950 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.066+0000 7f1059600640 1 -- 172.21.15.146:0/9362102 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c0043e0 con 0x55c3184ad950 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f1061600640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f10480a3da0 0x7f10480a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f1061600640 1 -- 172.21.15.146:0/1310949856 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f10540d0b70 con 0x7f1064108610 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f1063e00640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f10480a3da0 0x7f10480a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 -- 172.21.15.146:0/9362102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 msgr2=0x55c3184add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184add30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f105c009cd0 tx=0x7f105c009a20 comp rx=0 tx=0).stop 2024-04-24T08:31:16.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 -- 172.21.15.146:0/9362102 shutdown_connections 2024-04-24T08:31:16.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 --2- 172.21.15.146:0/9362102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184add30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 -- 172.21.15.146:0/9362102 >> 172.21.15.146:0/9362102 conn(0x55c3184a9c70 msgr2=0x55c3184aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.067+0000 7f106de1c740 1 -- 172.21.15.146:0/9362102 shutdown_connections 2024-04-24T08:31:16.080 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.068+0000 7f106de1c740 1 -- 172.21.15.146:0/9362102 wait complete. 2024-04-24T08:31:16.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.068+0000 7f1063e00640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f10480a3da0 0x7f10480a6260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55c3184aef10 tx=0x7f10540d4620 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.068+0000 7f106de1c740 1 -- start start 2024-04-24T08:31:16.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.068+0000 7f106de1c740 1 Processor -- start 2024-04-24T08:31:16.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f106de1c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f106de1c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f106de1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.081 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f106de1c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55c3184bb7c0 con 0x55c3184ad950 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f106de1c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c3183cc810 con 0x55c3184ae270 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f106de1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c3184bb940 con 0x55c3184b8940 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f106de1c740 10 client.0 _ll_register_callbacks cb 0x55c31839a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f105b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51050/0 (socket says 172.21.15.146:51050) 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 -- 172.21.15.146:0/1212256315 learned_addr learned my addr 172.21.15.146:0/1212256315 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 -- 172.21.15.146:0/1212256315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 msgr2=0x55c3184b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 -- 172.21.15.146:0/1212256315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 msgr2=0x55c3184b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f1069000640 1 -- 172.21.15.146:0/1212256315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3183dd3f0 con 0x55c3184ad950 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.069+0000 7f105b400640 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f1069000640 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b82c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f105c006010 tx=0x7f105c003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f4457062276ce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f1052000640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f1052000640 1 -- 172.21.15.146:0/1212256315 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c054020 con 0x55c3184ad950 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f1052000640 1 -- 172.21.15.146:0/1212256315 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f105c004120 con 0x55c3184ad950 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f105c036070 con 0x55c3184ad950 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f1052000640 1 -- 172.21.15.146:0/1212256315 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f105c05f050 con 0x55c3184ad950 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f106de1c740 10 client.17244 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.070+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c3181253f0 con 0x55c3184ad950 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f1052000640 1 -- 172.21.15.146:0/1212256315 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f105c036070 con 0x55c3184ad950 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f1052000640 1 -- 172.21.15.146:0/1212256315 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f105c042030 con 0x55c3184ad950 2024-04-24T08:31:16.084 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f106de1c740 10 client.17244 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:16.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f106de1c740 10 client.17244 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:31:16.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f106de1c740 20 client.17244 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:16.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f106de1c740 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c3184c8900 0x55c3184cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f106de1c740 4 client.17244 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:16.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c318420bf0 con 0x55c3184c8900 2024-04-24T08:31:16.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.071+0000 7f105b400640 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c3184c8900 0x55c3184cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.085 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.072+0000 7f105b400640 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c3184c8900 0x55c3184cacf0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7847d262056cb9f3 server_cookie=1809c8d57499a886 in_seq=0 out_seq=0 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.072+0000 7f1052000640 10 client.17244 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f1052000640 1 -- 172.21.15.146:0/1212256315 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c318420bf0 con 0x55c3184c8900 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f1052000640 10 client.17244 handle_command_reply: tid=0 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 client.17244 shutdown 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 msgr2=0x55c3184b82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b82c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f105c006010 tx=0x7f105c003f40 comp rx=0 tx=0).stop 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 shutdown_connections 2024-04-24T08:31:16.086 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c3184c8900 0x55c3184cacf0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f1052000640 0 client.17244 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 --2- 172.21.15.146:0/1212256315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b82c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 >> 172.21.15.146:0/1212256315 conn(0x55c3184a9c70 msgr2=0x55c3184af970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 shutdown_connections 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 1 -- 172.21.15.146:0/1212256315 wait complete. 2024-04-24T08:31:16.087 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.073+0000 7f106de1c740 20 client.17244 trim_cache size 0 max 16384 2024-04-24T08:31:16.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.089+0000 7f106de1c740 1 Processor -- start 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f106de1c740 1 -- start start 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f106de1c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f106de1c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f106de1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f106de1c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f10641b6b00 con 0x55c3184ad950 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f106de1c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10641b43d0 con 0x55c3184ae270 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f106de1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10641b2240 con 0x55c3184b8940 2024-04-24T08:31:16.092 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51058/0 (socket says 172.21.15.146:51058) 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f105b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f1069000640 1 -- 172.21.15.146:0/4128506119 learned_addr learned my addr 172.21.15.146:0/4128506119 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.090+0000 7f1069000640 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f105b400640 1 -- 172.21.15.146:0/4128506119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 msgr2=0x55c3184d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f105b400640 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f105b400640 1 -- 172.21.15.146:0/4128506119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 msgr2=0x55c3184c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f105b400640 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f105b400640 1 -- 172.21.15.146:0/4128506119 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3181ef7d0 con 0x55c3184ae270 2024-04-24T08:31:16.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f1069000640 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:16.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f105b400640 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b7df0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55c3184cb290 tx=0x7f104c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=663643725034860a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f1058c00640 1 -- 172.21.15.146:0/4128506119 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00fc80 con 0x55c3184ae270 2024-04-24T08:31:16.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f1058c00640 1 -- 172.21.15.146:0/4128506119 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f104c00fe20 con 0x55c3184ae270 2024-04-24T08:31:16.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.091+0000 7f1058c00640 1 -- 172.21.15.146:0/4128506119 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c0228e0 con 0x55c3184ae270 2024-04-24T08:31:16.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 -- 172.21.15.146:0/4128506119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 msgr2=0x55c3184b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b7df0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55c3184cb290 tx=0x7f104c005f90 comp rx=0 tx=0).stop 2024-04-24T08:31:16.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 -- 172.21.15.146:0/4128506119 shutdown_connections 2024-04-24T08:31:16.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b7df0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 --2- 172.21.15.146:0/4128506119 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 -- 172.21.15.146:0/4128506119 >> 172.21.15.146:0/4128506119 conn(0x55c3184a9c70 msgr2=0x55c3184aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 -- 172.21.15.146:0/4128506119 shutdown_connections 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.092+0000 7f106de1c740 1 -- 172.21.15.146:0/4128506119 wait complete. 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 -- start start 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 Processor -- start 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f10641b2890 con 0x55c3184ad950 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10641b3d00 con 0x55c3184ae270 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10641b2f60 con 0x55c3184b8940 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f106de1c740 10 client.0 _ll_register_callbacks cb 0x7f105c046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.093+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f105b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f1069000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51068/0 (socket says 172.21.15.146:51068) 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f1069000640 1 -- 172.21.15.146:0/2499450484 learned_addr learned my addr 172.21.15.146:0/2499450484 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f1069000640 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f105b400640 1 -- 172.21.15.146:0/2499450484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 msgr2=0x55c3184bf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f105b400640 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f105b400640 1 -- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 msgr2=0x55c3184b99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f105b400640 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f105b400640 1 -- 172.21.15.146:0/2499450484 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3184bf6d0 con 0x55c3184ae270 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f1069000640 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:16.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f105b400640 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b9f30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55c3184cb200 tx=0x7f104c00ffd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1cfe07285f4e04d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.094+0000 7f1052000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.095+0000 7f1052000640 1 -- 172.21.15.146:0/2499450484 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c030020 con 0x55c3184ae270 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.095+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f104c00fe20 con 0x55c3184ae270 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.095+0000 7f1052000640 1 -- 172.21.15.146:0/2499450484 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f104c0225e0 con 0x55c3184ae270 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.095+0000 7f1052000640 1 -- 172.21.15.146:0/2499450484 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f104c03b050 con 0x55c3184ae270 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.095+0000 7f106de1c740 10 client.27395 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.095+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c31852f2e0 con 0x55c3184ae270 2024-04-24T08:31:16.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f1052000640 1 -- 172.21.15.146:0/2499450484 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f104c00fe20 con 0x55c3184ae270 2024-04-24T08:31:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f1052000640 1 -- 172.21.15.146:0/2499450484 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f104c003830 con 0x55c3184ae270 2024-04-24T08:31:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f106de1c740 10 client.27395 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f106de1c740 10 client.27395 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:31:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f106de1c740 20 client.27395 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f106de1c740 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c31852fa30 0x55c318531e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f106de1c740 4 client.27395 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:31:16.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c31841aa60 con 0x55c31852fa30 2024-04-24T08:31:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.096+0000 7f1069000640 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c31852fa30 0x55c318531e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f1069000640 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c31852fa30 0x55c318531e50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=878bc6edb5355fbb server_cookie=965eda7501376a9b in_seq=0 out_seq=0 2024-04-24T08:31:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f1052000640 10 client.27395 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f1052000640 1 -- 172.21.15.146:0/2499450484 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c31841aa60 con 0x55c31852fa30 2024-04-24T08:31:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f1052000640 10 client.27395 handle_command_reply: tid=0 2024-04-24T08:31:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 client.27395 shutdown 2024-04-24T08:31:16.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 msgr2=0x55c3184b9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b9f30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55c3184cb200 tx=0x7f104c00ffd0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 shutdown_connections 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x55c31852fa30 0x55c318531e50 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3184b8940 0x55c3184bf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c3184ae270 0x55c3184b9f30 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 --2- 172.21.15.146:0/2499450484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c3184ad950 0x55c3184b99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f1052000640 0 client.27395 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.097+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 >> 172.21.15.146:0/2499450484 conn(0x55c3184a9c70 msgr2=0x55c3184af0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 shutdown_connections 2024-04-24T08:31:16.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106de1c740 1 -- 172.21.15.146:0/2499450484 wait complete. 2024-04-24T08:31:16.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106de1c740 20 client.27395 trim_cache size 0 max 16384 2024-04-24T08:31:16.106 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:31:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f10480a3da0 msgr2=0x7f10480a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f10480a3da0 0x7f10480a6260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55c3184aef10 tx=0x7f10540d4620 comp rx=0 tx=0).stop 2024-04-24T08:31:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 msgr2=0x7f106419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 0x7f106419ee40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f105400e9c0 tx=0x7f105400ee80 comp rx=0 tx=0).stop 2024-04-24T08:31:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 shutdown_connections 2024-04-24T08:31:16.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f10480a3da0 0x7f10480a6260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f106410e660 0x7f10641a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1064108fd0 0x7f106419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 --2- 172.21.15.146:0/1310949856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1064108610 0x7f106419ee40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.098+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 >> 172.21.15.146:0/1310949856 conn(0x7f1064076540 msgr2=0x7f1064077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.108 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.099+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 shutdown_connections 2024-04-24T08:31:16.109 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.099+0000 7f106b000640 1 -- 172.21.15.146:0/1310949856 wait complete. 2024-04-24T08:31:16.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:31:16.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 -- 172.21.15.146:0/2740826917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c1093e0 msgr2=0x7fbf0c110fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 --2- 172.21.15.146:0/2740826917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c110fb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009ca0 tx=0x7fbf0402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 -- 172.21.15.146:0/2740826917 shutdown_connections 2024-04-24T08:31:16.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 --2- 172.21.15.146:0/2740826917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c110fb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 --2- 172.21.15.146:0/2740826917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c104cc0 0x7fbf0c108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 --2- 172.21.15.146:0/2740826917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c104310 0x7fbf0c1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 -- 172.21.15.146:0/2740826917 >> 172.21.15.146:0/2740826917 conn(0x7fbf0c0ffb70 msgr2=0x7fbf0c101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 -- 172.21.15.146:0/2740826917 shutdown_connections 2024-04-24T08:31:16.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.324+0000 7fbf10e00640 1 -- 172.21.15.146:0/2740826917 wait complete. 2024-04-24T08:31:16.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.325+0000 7fbf10e00640 1 Processor -- start 2024-04-24T08:31:16.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.325+0000 7fbf10e00640 1 -- start start 2024-04-24T08:31:16.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf10e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c104310 0x7fbf0c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf10e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf10e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c19a770 0x7fbf0c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf10e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1144f0 con 0x7fbf0c1093e0 2024-04-24T08:31:16.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf10e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c114370 con 0x7fbf0c104310 2024-04-24T08:31:16.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf10e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c114670 con 0x7fbf0c19a770 2024-04-24T08:31:16.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51084/0 (socket says 172.21.15.146:51084) 2024-04-24T08:31:16.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 -- 172.21.15.146:0/883404633 learned_addr learned my addr 172.21.15.146:0/883404633 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0b400640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c104310 0x7fbf0c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0be00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c19a770 0x7fbf0c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 -- 172.21.15.146:0/883404633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c19a770 msgr2=0x7fbf0c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c19a770 0x7fbf0c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 -- 172.21.15.146:0/883404633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c104310 msgr2=0x7fbf0c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c104310 0x7fbf0c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.326+0000 7fbf0aa00640 1 -- 172.21.15.146:0/883404633 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c19b370 con 0x7fbf0c1093e0 2024-04-24T08:31:16.335 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.327+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c1a0bf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbef400e9f0 tx=0x7fbef400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.327+0000 7fbf03e00640 1 -- 172.21.15.146:0/883404633 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef400ccc0 con 0x7fbf0c1093e0 2024-04-24T08:31:16.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.327+0000 7fbf03e00640 1 -- 172.21.15.146:0/883404633 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef4005500 con 0x7fbf0c1093e0 2024-04-24T08:31:16.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.327+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c1a7450 con 0x7fbf0c1093e0 2024-04-24T08:31:16.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.327+0000 7fbf03e00640 1 -- 172.21.15.146:0/883404633 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4010690 con 0x7fbf0c1093e0 2024-04-24T08:31:16.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.327+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c1a7800 con 0x7fbf0c1093e0 2024-04-24T08:31:16.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.330+0000 7fbf15216740 1 Processor -- start 2024-04-24T08:31:16.336 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.330+0000 7fbf03e00640 1 -- 172.21.15.146:0/883404633 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fbef40046c0 con 0x7fbf0c1093e0 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf15216740 1 -- start start 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf15216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be8ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf15216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf15216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf15216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f58bdf51d0 con 0x55f58be90460 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf15216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f58bd9b450 con 0x55f58be8fb40 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf15216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f58bdf1190 con 0x55f58be9ab30 2024-04-24T08:31:16.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be8ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf0b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf0be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be8ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57910/0 (socket says 172.21.15.146:57910) 2024-04-24T08:31:16.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.331+0000 7fbf0be00640 1 -- 172.21.15.146:0/1810474009 learned_addr learned my addr 172.21.15.146:0/1810474009 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf0be00640 1 -- 172.21.15.146:0/1810474009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 msgr2=0x55f58be9cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf0be00640 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf0be00640 1 -- 172.21.15.146:0/1810474009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 msgr2=0x55f58be9a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf0be00640 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf0be00640 1 -- 172.21.15.146:0/1810474009 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f58be58310 con 0x55f58be8fb40 2024-04-24T08:31:16.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:16.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf0be00640 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be8ff20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009ca0 tx=0x7fbf040093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc59e1a5b0a0299 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.343 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf03e00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbef00a3d20 0x7fbef00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.332+0000 7fbf00c00640 1 -- 172.21.15.146:0/1810474009 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04031db0 con 0x55f58be8fb40 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf00c00640 1 -- 172.21.15.146:0/1810474009 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf04036030 con 0x55f58be8fb40 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf00c00640 1 -- 172.21.15.146:0/1810474009 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf040042a0 con 0x55f58be8fb40 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf03e00640 1 -- 172.21.15.146:0/883404633 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fbef40d0b30 con 0x7fbf0c1093e0 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf0b400640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbef00a3d20 0x7fbef00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf15216740 1 -- 172.21.15.146:0/1810474009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 msgr2=0x55f58be8ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf15216740 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be8ff20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009ca0 tx=0x7fbf040093e0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.344 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf0b400640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbef00a3d20 0x7fbef00a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55f58be91100 tx=0x7fbefc005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf15216740 1 -- 172.21.15.146:0/1810474009 shutdown_connections 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf15216740 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf15216740 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf15216740 1 --2- 172.21.15.146:0/1810474009 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be8ff20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.333+0000 7fbf15216740 1 -- 172.21.15.146:0/1810474009 >> 172.21.15.146:0/1810474009 conn(0x55f58be8be60 msgr2=0x55f58be8c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.334+0000 7fbf15216740 1 -- 172.21.15.146:0/1810474009 shutdown_connections 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.334+0000 7fbf15216740 1 -- 172.21.15.146:0/1810474009 wait complete. 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.334+0000 7fbf15216740 1 -- start start 2024-04-24T08:31:16.345 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.334+0000 7fbf15216740 1 Processor -- start 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf15216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58be9a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf15216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be97830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf15216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58be97e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf15216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x55f58be9d7e0 con 0x55f58be90460 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf15216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55f58bdae810 con 0x55f58be9ab30 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf15216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f58be9d960 con 0x55f58be8fb40 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf15216740 10 client.0 _ll_register_callbacks cb 0x55f58bd7c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58be9a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be97830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58be9a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46344/0 (socket says 172.21.15.146:46344) 2024-04-24T08:31:16.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0b400640 1 -- 172.21.15.146:0/3516374352 learned_addr learned my addr 172.21.15.146:0/3516374352 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0be00640 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58be97e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0aa00640 1 -- 172.21.15.146:0/3516374352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 msgr2=0x55f58be9a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58be9a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.347 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0aa00640 1 -- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 msgr2=0x55f58be97e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58be97e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.335+0000 7fbf0aa00640 1 -- 172.21.15.146:0/3516374352 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f58bdbf3f0 con 0x55f58be90460 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be97830 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbef40ce210 tx=0x7fbef40096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2cbed5349884ec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbef9600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbef9600640 1 -- 172.21.15.146:0/3516374352 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef409c5a0 con 0x55f58be90460 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbef9600640 1 -- 172.21.15.146:0/3516374352 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fbef40a5440 con 0x55f58be90460 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf04036030 con 0x55f58be90460 2024-04-24T08:31:16.348 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbef9600640 1 -- 172.21.15.146:0/3516374352 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbef40ce6a0 con 0x55f58be90460 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbf15216740 10 client.17262 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f58bb073f0 con 0x55f58be90460 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.336+0000 7fbef9600640 1 -- 172.21.15.146:0/3516374352 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf04036030 con 0x55f58be90460 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbef9600640 1 -- 172.21.15.146:0/3516374352 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbef409beb0 con 0x55f58be90460 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbf15216740 10 client.17262 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbf15216740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbf15216740 20 client.17262 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbf15216740 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58beaabe0 0x55f58beacfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.349 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbf15216740 4 client.17262 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f58be02bf0 con 0x55f58beaabe0 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.337+0000 7fbf0b400640 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58beaabe0 0x55f58beacfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.338+0000 7fbf0b400640 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58beaabe0 0x55f58beacfd0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aefa337e9caf5ebd server_cookie=c450fcb0b0bab661 in_seq=0 out_seq=0 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.338+0000 7fbef9600640 10 client.17262 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbef9600640 1 -- 172.21.15.146:0/3516374352 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f58be02bf0 con 0x55f58beaabe0 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbef9600640 10 client.17262 handle_command_reply: tid=0 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 client.17262 shutdown 2024-04-24T08:31:16.350 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 msgr2=0x55f58be97830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be97830 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbef40ce210 tx=0x7fbef40096c0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 shutdown_connections 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58beaabe0 0x55f58beacfd0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58be97e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be97830 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 --2- 172.21.15.146:0/3516374352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58be9a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbef9600640 0 client.17262 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 >> 172.21.15.146:0/3516374352 conn(0x55f58be8be60 msgr2=0x55f58be91b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.351 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 shutdown_connections 2024-04-24T08:31:16.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 1 -- 172.21.15.146:0/3516374352 wait complete. 2024-04-24T08:31:16.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.339+0000 7fbf15216740 20 client.17262 trim_cache size 0 max 16384 2024-04-24T08:31:16.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 Processor -- start 2024-04-24T08:31:16.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 -- start start 2024-04-24T08:31:16.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58bea66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58beb28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1b2610 con 0x55f58be90460 2024-04-24T08:31:16.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1afee0 con 0x55f58be9ab30 2024-04-24T08:31:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.354+0000 7fbf15216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1add50 con 0x55f58be8fb40 2024-04-24T08:31:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51124/0 (socket says 172.21.15.146:51124) 2024-04-24T08:31:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 -- 172.21.15.146:0/4057998146 learned_addr learned my addr 172.21.15.146:0/4057998146 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0be00640 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58beb28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0b400640 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58bea66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 -- 172.21.15.146:0/4057998146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 msgr2=0x55f58bea66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58bea66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 -- 172.21.15.146:0/4057998146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 msgr2=0x55f58beb28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58beb28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 -- 172.21.15.146:0/4057998146 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f58bbd17d0 con 0x55f58be90460 2024-04-24T08:31:16.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0be00640 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58beb28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.355+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a070 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbef409ad10 tx=0x7fbef40d0db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b75cf47d92072b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbefbe00640 1 -- 172.21.15.146:0/4057998146 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef40056b0 con 0x55f58be90460 2024-04-24T08:31:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbefbe00640 1 -- 172.21.15.146:0/4057998146 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef40d6060 con 0x55f58be90460 2024-04-24T08:31:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbefbe00640 1 -- 172.21.15.146:0/4057998146 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef409c3c0 con 0x55f58be90460 2024-04-24T08:31:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 -- 172.21.15.146:0/4057998146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 msgr2=0x55f58be9a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a070 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbef409ad10 tx=0x7fbef40d0db0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 -- 172.21.15.146:0/4057998146 shutdown_connections 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be9ab30 0x55f58beb28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9a070 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 --2- 172.21.15.146:0/4057998146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be8fb40 0x55f58bea66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 -- 172.21.15.146:0/4057998146 >> 172.21.15.146:0/4057998146 conn(0x55f58be8be60 msgr2=0x55f58be92a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.356+0000 7fbf15216740 1 -- 172.21.15.146:0/4057998146 shutdown_connections 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.357+0000 7fbf15216740 1 -- 172.21.15.146:0/4057998146 wait complete. 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.357+0000 7fbf15216740 1 -- start start 2024-04-24T08:31:16.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.357+0000 7fbf15216740 1 Processor -- start 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.357+0000 7fbf15216740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be9b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.357+0000 7fbf15216740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.357+0000 7fbf15216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf15216740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1ae3a0 con 0x55f58be90460 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf15216740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1af810 con 0x55f58be8fb40 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf15216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c1aea70 con 0x55f58be9ab30 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf15216740 10 client.0 _ll_register_callbacks cb 0x7fbef4004aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:16.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51134/0 (socket says 172.21.15.146:51134) 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 -- 172.21.15.146:0/3286369207 learned_addr learned my addr 172.21.15.146:0/3286369207 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0be00640 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0b400640 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be9b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 -- 172.21.15.146:0/3286369207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 msgr2=0x55f58be9c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 -- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 msgr2=0x55f58be9b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:16.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be9b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 -- 172.21.15.146:0/3286369207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f58bea11a0 con 0x55f58be90460 2024-04-24T08:31:16.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbf0aa00640 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9bf30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbef4099430 tx=0x7fbef40d46f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fb26dc017c24b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:16.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.358+0000 7fbef9600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:16.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbef9600640 1 -- 172.21.15.146:0/3286369207 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef409c3c0 con 0x55f58be90460 2024-04-24T08:31:16.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbef9600640 1 -- 172.21.15.146:0/3286369207 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fbef40a5240 con 0x55f58be90460 2024-04-24T08:31:16.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbef40d6060 con 0x55f58be90460 2024-04-24T08:31:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbef9600640 1 -- 172.21.15.146:0/3286369207 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbef4098d00 con 0x55f58be90460 2024-04-24T08:31:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 10 client.17268 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f58bea11a0 con 0x55f58be90460 2024-04-24T08:31:16.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbef9600640 1 -- 172.21.15.146:0/3286369207 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef40d6060 con 0x55f58be90460 2024-04-24T08:31:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbef9600640 1 -- 172.21.15.146:0/3286369207 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbef4087050 con 0x55f58be90460 2024-04-24T08:31:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 10 client.17268 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:16.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 20 client.17268 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:16.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58bf1a110 0x55f58bf1c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:16.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 4 client.17268 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:31:16.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.359+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f58bdfca60 con 0x55f58bf1a110 2024-04-24T08:31:16.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.360+0000 7fbf0b400640 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58bf1a110 0x55f58bf1c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:16.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.360+0000 7fbf0b400640 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58bf1a110 0x55f58bf1c530 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d942462437106365 server_cookie=9c851a40daeb45e2 in_seq=0 out_seq=0 2024-04-24T08:31:16.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.360+0000 7fbef9600640 10 client.17268 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:16.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.360+0000 7fbef9600640 1 -- 172.21.15.146:0/3286369207 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f58bdfca60 con 0x55f58bf1a110 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.360+0000 7fbef9600640 10 client.17268 handle_command_reply: tid=0 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 client.17268 shutdown 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 msgr2=0x55f58be9bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9bf30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbef4099430 tx=0x7fbef40d46f0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 shutdown_connections 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x55f58bf1a110 0x55f58bf1c530 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f58be9ab30 0x55f58be9c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f58be90460 0x55f58be9bf30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 --2- 172.21.15.146:0/3286369207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55f58be8fb40 0x55f58be9b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbef9600640 0 client.17268 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:16.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 >> 172.21.15.146:0/3286369207 conn(0x55f58be8be60 msgr2=0x55f58be8a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 shutdown_connections 2024-04-24T08:31:16.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 1 -- 172.21.15.146:0/3286369207 wait complete. 2024-04-24T08:31:16.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.361+0000 7fbf15216740 20 client.17268 trim_cache size 0 max 16384 2024-04-24T08:31:16.373 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:31:16.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbef00a3d20 msgr2=0x7fbef00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbef00a3d20 0x7fbef00a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55f58be91100 tx=0x7fbefc005f90 comp rx=0 tx=0).stop 2024-04-24T08:31:16.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c1093e0 msgr2=0x7fbf0c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:16.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c1a0bf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbef400e9f0 tx=0x7fbef400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 shutdown_connections 2024-04-24T08:31:16.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fbef00a3d20 0x7fbef00a61e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf0c19a770 0x7fbf0c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf0c1093e0 0x7fbf0c1a0bf0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 --2- 172.21.15.146:0/883404633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf0c104310 0x7fbf0c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:16.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 >> 172.21.15.146:0/883404633 conn(0x7fbf0c0ffb70 msgr2=0x7fbf0c100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:16.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 shutdown_connections 2024-04-24T08:31:16.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:16.362+0000 7fbf10e00640 1 -- 172.21.15.146:0/883404633 wait complete. 2024-04-24T08:31:16.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:16 smithi146 ceph-mon[27805]: pgmap v312: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:16.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:16 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3809961378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:31:16.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:16 smithi146 ceph-mon[27805]: osdmap e99: 12 total, 12 up, 12 in 2024-04-24T08:31:16.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:16 smithi177 ceph-mon[33046]: pgmap v312: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:16.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:16 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3809961378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:31:16.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:16 smithi177 ceph-mon[33046]: osdmap e99: 12 total, 12 up, 12 in 2024-04-24T08:31:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:16 smithi153 ceph-mon[32746]: pgmap v312: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:16 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3809961378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:31:16.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:16 smithi153 ceph-mon[32746]: osdmap e99: 12 total, 12 up, 12 in 2024-04-24T08:31:17.381 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:31:17.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:31:17.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:31:17.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 -- 172.21.15.146:0/4054326471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c076e50 msgr2=0x7f932c1129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 --2- 172.21.15.146:0/4054326471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c076e50 0x7f932c1129a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9314009930 tx=0x7f931402a420 comp rx=0 tx=0).stop 2024-04-24T08:31:17.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 -- 172.21.15.146:0/4054326471 shutdown_connections 2024-04-24T08:31:17.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 --2- 172.21.15.146:0/4054326471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c112ee0 0x7f932c1152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 --2- 172.21.15.146:0/4054326471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c076e50 0x7f932c1129a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 --2- 172.21.15.146:0/4054326471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c076530 0x7f932c076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 -- 172.21.15.146:0/4054326471 >> 172.21.15.146:0/4054326471 conn(0x7f932c0ffd90 msgr2=0x7f932c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 -- 172.21.15.146:0/4054326471 shutdown_connections 2024-04-24T08:31:17.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.599+0000 7f9332600640 1 -- 172.21.15.146:0/4054326471 wait complete. 2024-04-24T08:31:17.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 Processor -- start 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 -- start start 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c076530 0x7f932c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c076e50 0x7f932c19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c112ee0 0x7f932c1a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f932c118a90 con 0x7f932c076e50 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f932c118910 con 0x7f932c112ee0 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f9332600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f932c118c10 con 0x7f932c076530 2024-04-24T08:31:17.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f932aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c076e50 0x7f932c19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.601+0000 7f932b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c076530 0x7f932c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c076530 0x7f932c19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46380/0 (socket says 172.21.15.146:46380) 2024-04-24T08:31:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 -- 172.21.15.146:0/2822055256 learned_addr learned my addr 172.21.15.146:0/2822055256 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932be00640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c112ee0 0x7f932c1a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 -- 172.21.15.146:0/2822055256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c112ee0 msgr2=0x7f932c1a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c112ee0 0x7f932c1a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 -- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c076e50 msgr2=0x7f932c19f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c076e50 0x7f932c19f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 -- 172.21.15.146:0/2822055256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c1a3ff0 con 0x7f932c076530 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.602+0000 7f932b400640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c076530 0x7f932c19f050 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e9f0 tx=0x7f931c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.603+0000 7f9323e00640 1 -- 172.21.15.146:0/2822055256 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c00cd80 con 0x7f932c076530 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.603+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932c1a42e0 con 0x7f932c076530 2024-04-24T08:31:17.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.603+0000 7f9323e00640 1 -- 172.21.15.146:0/2822055256 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f931c010910 con 0x7f932c076530 2024-04-24T08:31:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.603+0000 7f9323e00640 1 -- 172.21.15.146:0/2822055256 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0188f0 con 0x7f932c076530 2024-04-24T08:31:17.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.603+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932c1abb70 con 0x7f932c076530 2024-04-24T08:31:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 Processor -- start 2024-04-24T08:31:17.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 -- start start 2024-04-24T08:31:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9323e00640 1 -- 172.21.15.146:0/2822055256 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f931c010ac0 con 0x7f932c076530 2024-04-24T08:31:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288cab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x559f28825e70 con 0x559f288c0090 2024-04-24T08:31:17.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559f287cc380 con 0x559f288cb080 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.606+0000 7f9335588740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559f28821e30 con 0x559f288c09b0 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46396/0 (socket says 172.21.15.146:46396) 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 -- 172.21.15.146:0/483955660 learned_addr learned my addr 172.21.15.146:0/483955660 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51166/0 (socket says 172.21.15.146:51166) 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932aa00640 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 -- 172.21.15.146:0/483955660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 msgr2=0x559f288cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 -- 172.21.15.146:0/483955660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 msgr2=0x559f288c0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932b400640 1 -- 172.21.15.146:0/483955660 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f286007e0 con 0x559f288c09b0 2024-04-24T08:31:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.607+0000 7f932be00640 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.608+0000 7f932b400640 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288cab40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f931c0ce1c0 tx=0x7f931c0d9660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62a89a41cbadc8e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.608+0000 7f9320c00640 1 -- 172.21.15.146:0/483955660 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0e0020 con 0x559f288c09b0 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.608+0000 7f9323e00640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93100a3da0 0x7f93100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.608+0000 7f9320c00640 1 -- 172.21.15.146:0/483955660 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f931c0d5040 con 0x559f288c09b0 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.608+0000 7f9320c00640 1 -- 172.21.15.146:0/483955660 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0db810 con 0x559f288c09b0 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.608+0000 7f9323e00640 1 -- 172.21.15.146:0/2822055256 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f931c0d0ba0 con 0x7f932c076530 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.608+0000 7f932be00640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93100a3da0 0x7f93100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 -- 172.21.15.146:0/483955660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 msgr2=0x559f288cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288cab40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f931c0ce1c0 tx=0x7f931c0d9660 comp rx=0 tx=0).stop 2024-04-24T08:31:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 -- 172.21.15.146:0/483955660 shutdown_connections 2024-04-24T08:31:17.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288cab40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 --2- 172.21.15.146:0/483955660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 -- 172.21.15.146:0/483955660 >> 172.21.15.146:0/483955660 conn(0x559f288bc3b0 msgr2=0x559f288bc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f932be00640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93100a3da0 0x7f93100a6260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9324009fd0 tx=0x7f9324009500 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 -- 172.21.15.146:0/483955660 shutdown_connections 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 -- 172.21.15.146:0/483955660 wait complete. 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.609+0000 7f9335588740 1 -- start start 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 1 Processor -- start 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288c8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x559f288cde70 con 0x559f288c0090 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x559f287df660 con 0x559f288cb080 2024-04-24T08:31:17.619 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559f288cdff0 con 0x559f288c09b0 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f9335588740 10 client.0 _ll_register_callbacks cb 0x559f287cdaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f932b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f932b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51174/0 (socket says 172.21.15.146:51174) 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f932b400640 1 -- 172.21.15.146:0/2605494479 learned_addr learned my addr 172.21.15.146:0/2605494479 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f932aa00640 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.610+0000 7f932be00640 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f932b400640 1 -- 172.21.15.146:0/2605494479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 msgr2=0x559f288c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f932b400640 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288c8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f932b400640 1 -- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 msgr2=0x559f288c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f932b400640 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f932b400640 1 -- 172.21.15.146:0/2605494479 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f28660f70 con 0x559f288c0090 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f932b400640 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c7fe0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f931c099550 tx=0x7f931c0dcf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4897611e75c505 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f9319600640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f9319600640 1 -- 172.21.15.146:0/2605494479 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0e0030 con 0x559f288c0090 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f9319600640 1 -- 172.21.15.146:0/2605494479 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f931c0bd0f0 con 0x559f288c0090 2024-04-24T08:31:17.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.611+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f931c0d5040 con 0x559f288c0090 2024-04-24T08:31:17.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9319600640 1 -- 172.21.15.146:0/2605494479 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f931c098e20 con 0x559f288c0090 2024-04-24T08:31:17.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 10 client.17280 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:17.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559f284f8170 con 0x559f288c0090 2024-04-24T08:31:17.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9319600640 1 -- 172.21.15.146:0/2605494479 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f931c0d5040 con 0x559f288c0090 2024-04-24T08:31:17.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9319600640 1 -- 172.21.15.146:0/2605494479 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f931c0ce500 con 0x559f288c0090 2024-04-24T08:31:17.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 10 client.17280 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:17.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:17.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 20 client.17280 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:17.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f288db250 0x559f288dd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 4 client.17280 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:17.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.612+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f28833890 con 0x559f288db250 2024-04-24T08:31:17.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.613+0000 7f932aa00640 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f288db250 0x559f288dd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.613+0000 7f932aa00640 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f288db250 0x559f288dd640 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba56f2565b271cd5 server_cookie=47b53769f5be92cb in_seq=0 out_seq=0 2024-04-24T08:31:17.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.613+0000 7f9319600640 10 client.17280 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9319600640 1 -- 172.21.15.146:0/2605494479 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f28833890 con 0x559f288db250 2024-04-24T08:31:17.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9319600640 10 client.17280 handle_command_reply: tid=0 2024-04-24T08:31:17.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 client.17280 shutdown 2024-04-24T08:31:17.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 msgr2=0x559f288c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.625 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c7fe0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f931c099550 tx=0x7f931c0dcf50 comp rx=0 tx=0).stop 2024-04-24T08:31:17.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 shutdown_connections 2024-04-24T08:31:17.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f288db250 0x559f288dd640 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c09b0 0x559f288c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288cb080 0x559f288c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 --2- 172.21.15.146:0/2605494479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c0090 0x559f288c7fe0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9319600640 0 client.17280 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.626 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.614+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 >> 172.21.15.146:0/2605494479 conn(0x559f288bc3b0 msgr2=0x559f288c21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.615+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 shutdown_connections 2024-04-24T08:31:17.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.615+0000 7f9335588740 1 -- 172.21.15.146:0/2605494479 wait complete. 2024-04-24T08:31:17.627 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.615+0000 7f9335588740 20 client.17280 trim_cache size 0 max 16384 2024-04-24T08:31:17.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 Processor -- start 2024-04-24T08:31:17.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 -- start start 2024-04-24T08:31:17.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c0090 0x559f288c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c09b0 0x559f288d6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.636 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288cb080 0x559f288e0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f932c1b6d80 con 0x559f288cb080 2024-04-24T08:31:17.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f932c1b4650 con 0x559f288c09b0 2024-04-24T08:31:17.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.634+0000 7f9335588740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f932c1b24c0 con 0x559f288c0090 2024-04-24T08:31:17.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c0090 0x559f288c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.637 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c0090 0x559f288c2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46416/0 (socket says 172.21.15.146:46416) 2024-04-24T08:31:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 -- 172.21.15.146:0/4279801732 learned_addr learned my addr 172.21.15.146:0/4279801732 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932aa00640 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c09b0 0x559f288d6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932be00640 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288cb080 0x559f288e0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 -- 172.21.15.146:0/4279801732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c09b0 msgr2=0x559f288d6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c09b0 0x559f288d6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.638 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 -- 172.21.15.146:0/4279801732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288cb080 msgr2=0x559f288e0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288cb080 0x559f288e0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 -- 172.21.15.146:0/4279801732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f288c07e0 con 0x559f288c0090 2024-04-24T08:31:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932aa00640 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c09b0 0x559f288d6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:17.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f932b400640 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c0090 0x559f288c2900 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f931c09c880 tx=0x7f931c09b090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba51d9efbe319baf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.635+0000 7f931be00640 1 -- 172.21.15.146:0/4279801732 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0e0070 con 0x559f288c0090 2024-04-24T08:31:17.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f931be00640 1 -- 172.21.15.146:0/4279801732 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f931c0d9b60 con 0x559f288c0090 2024-04-24T08:31:17.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f931be00640 1 -- 172.21.15.146:0/4279801732 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c09c4e0 con 0x559f288c0090 2024-04-24T08:31:17.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 -- 172.21.15.146:0/4279801732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c0090 msgr2=0x559f288c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c0090 0x559f288c2900 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f931c09c880 tx=0x7f931c09b090 comp rx=0 tx=0).stop 2024-04-24T08:31:17.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 -- 172.21.15.146:0/4279801732 shutdown_connections 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288cb080 0x559f288e0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c09b0 0x559f288d6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 --2- 172.21.15.146:0/4279801732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288c0090 0x559f288c2900 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 -- 172.21.15.146:0/4279801732 >> 172.21.15.146:0/4279801732 conn(0x559f288bc3b0 msgr2=0x559f288d6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 -- 172.21.15.146:0/4279801732 shutdown_connections 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.636+0000 7f9335588740 1 -- 172.21.15.146:0/4279801732 wait complete. 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.637+0000 7f9335588740 1 -- start start 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.637+0000 7f9335588740 1 Processor -- start 2024-04-24T08:31:17.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9335588740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c0090 0x559f288de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9335588740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c09b0 0x559f288dea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9335588740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288cb080 0x559f28940430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9335588740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f932c1b2b10 con 0x559f288c09b0 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9335588740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f932c1b3f80 con 0x559f288c0090 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9335588740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f932c1b31e0 con 0x559f288cb080 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9335588740 10 client.0 _ll_register_callbacks cb 0x7f9324009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c0090 0x559f288de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c0090 0x559f288de4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:57998/0 (socket says 172.21.15.146:57998) 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 -- 172.21.15.146:0/1227195383 learned_addr learned my addr 172.21.15.146:0/1227195383 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932aa00640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c09b0 0x559f288dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932be00640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288cb080 0x559f28940430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 -- 172.21.15.146:0/1227195383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288cb080 msgr2=0x559f28940430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288cb080 0x559f28940430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 -- 172.21.15.146:0/1227195383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c09b0 msgr2=0x559f288dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c09b0 0x559f288dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 -- 172.21.15.146:0/1227195383 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f28940970 con 0x559f288c0090 2024-04-24T08:31:17.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932be00640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288cb080 0x559f28940430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:17.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f932b400640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c0090 0x559f288de4d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f931c098fe0 tx=0x7f931c0a5740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba0377f18b8c4f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.638+0000 7f9319600640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:31:17.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9319600640 1 -- 172.21.15.146:0/1227195383 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0e0070 con 0x559f288c0090 2024-04-24T08:31:17.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9319600640 1 -- 172.21.15.146:0/1227195383 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f931c0bd0f0 con 0x559f288c0090 2024-04-24T08:31:17.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f931c0d9b60 con 0x559f288c0090 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9319600640 1 -- 172.21.15.146:0/1227195383 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f931c0d4020 con 0x559f288c0090 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9335588740 10 client.27431 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559f28940970 con 0x559f288c0090 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9319600640 1 -- 172.21.15.146:0/1227195383 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f931c0d9b60 con 0x559f288c0090 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9319600640 1 -- 172.21.15.146:0/1227195383 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f931c0d47c0 con 0x559f288c0090 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9335588740 10 client.27431 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9335588740 10 client.27431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.639+0000 7f9335588740 20 client.27431 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:17.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.640+0000 7f9335588740 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f28941ec0 0x559f289442b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.640+0000 7f9335588740 4 client.27431 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:31:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.640+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559f2882d700 con 0x559f28941ec0 2024-04-24T08:31:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.640+0000 7f932aa00640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f28941ec0 0x559f289442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.640+0000 7f932aa00640 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f28941ec0 0x559f289442b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f142cf91ed8b7fd0 server_cookie=de3b5a009c39fbbc in_seq=0 out_seq=0 2024-04-24T08:31:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.640+0000 7f9319600640 10 client.27431 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.641+0000 7f9319600640 1 -- 172.21.15.146:0/1227195383 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559f2882d700 con 0x559f28941ec0 2024-04-24T08:31:17.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.641+0000 7f9319600640 10 client.27431 handle_command_reply: tid=0 2024-04-24T08:31:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.641+0000 7f9335588740 1 client.27431 shutdown 2024-04-24T08:31:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c0090 msgr2=0x559f288de4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c0090 0x559f288de4d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f931c098fe0 tx=0x7f931c0a5740 comp rx=0 tx=0).stop 2024-04-24T08:31:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 shutdown_connections 2024-04-24T08:31:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x559f28941ec0 0x559f289442b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559f288cb080 0x559f28940430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x559f288c09b0 0x559f288dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9319600640 0 client.27431 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 --2- 172.21.15.146:0/1227195383 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x559f288c0090 0x559f288de4d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 >> 172.21.15.146:0/1227195383 conn(0x559f288bc3b0 msgr2=0x559f288ca0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 shutdown_connections 2024-04-24T08:31:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 1 -- 172.21.15.146:0/1227195383 wait complete. 2024-04-24T08:31:17.652 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.642+0000 7f9335588740 20 client.27431 trim_cache size 0 max 16384 2024-04-24T08:31:17.652 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:31:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "return_code": 0, 2024-04-24T08:31:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_tag": "534e6d12-6bbc-439f-9ad7-508a87d11e15", 2024-04-24T08:31:17.653 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": "asynchronous" 2024-04-24T08:31:17.653 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:31:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93100a3da0 msgr2=0x7f93100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93100a3da0 0x7f93100a6260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9324009fd0 tx=0x7f9324009500 comp rx=0 tx=0).stop 2024-04-24T08:31:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c076530 msgr2=0x7f932c19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c076530 0x7f932c19f050 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f931c00e9f0 tx=0x7f931c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 shutdown_connections 2024-04-24T08:31:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f93100a3da0 0x7f93100a6260 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f932c112ee0 0x7f932c1a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c076e50 0x7f932c19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 --2- 172.21.15.146:0/2822055256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f932c076530 0x7f932c19f050 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 >> 172.21.15.146:0/2822055256 conn(0x7f932c0ffd90 msgr2=0x7f932c114670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 shutdown_connections 2024-04-24T08:31:17.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.643+0000 7f9332600640 1 -- 172.21.15.146:0/2822055256 wait complete. 2024-04-24T08:31:17.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:31:17.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:31:17.905 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.904+0000 7fcecfe00640 1 -- 172.21.15.146:0/797716672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced0103060 msgr2=0x7fced0103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.904+0000 7fcecfe00640 1 --2- 172.21.15.146:0/797716672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced0103060 0x7fced0103440 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009900 tx=0x7fcec002a480 comp rx=0 tx=0).stop 2024-04-24T08:31:17.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.905+0000 7fcecfe00640 1 -- 172.21.15.146:0/797716672 shutdown_connections 2024-04-24T08:31:17.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.905+0000 7fcecfe00640 1 --2- 172.21.15.146:0/797716672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced010c480 0x7fced010e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.905+0000 7fcecfe00640 1 --2- 172.21.15.146:0/797716672 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced0103980 0x7fced010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.905+0000 7fcecfe00640 1 --2- 172.21.15.146:0/797716672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced0103060 0x7fced0103440 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.905+0000 7fcecfe00640 1 -- 172.21.15.146:0/797716672 >> 172.21.15.146:0/797716672 conn(0x7fced00fc880 msgr2=0x7fced00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.907 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.905+0000 7fcecfe00640 1 -- 172.21.15.146:0/797716672 shutdown_connections 2024-04-24T08:31:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.905+0000 7fcecfe00640 1 -- 172.21.15.146:0/797716672 wait complete. 2024-04-24T08:31:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 Processor -- start 2024-04-24T08:31:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 -- start start 2024-04-24T08:31:17.908 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced0103060 0x7fced01a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced0103980 0x7fced01a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced010c480 0x7fced019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fced0100f70 con 0x7fced0103060 2024-04-24T08:31:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fced0100df0 con 0x7fced010c480 2024-04-24T08:31:17.909 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.907+0000 7fcecfe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fced01010f0 con 0x7fced0103980 2024-04-24T08:31:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.908+0000 7fcecf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced010c480 0x7fced019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.908+0000 7fcec7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced0103980 0x7fced01a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.908+0000 7fcecf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced010c480 0x7fced019b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58028/0 (socket says 172.21.15.146:58028) 2024-04-24T08:31:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.908+0000 7fcecf400640 1 -- 172.21.15.146:0/834158101 learned_addr learned my addr 172.21.15.146:0/834158101 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.910 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.908+0000 7fcecea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced0103060 0x7fced01a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.909+0000 7fcec7e00640 1 -- 172.21.15.146:0/834158101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced010c480 msgr2=0x7fced019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.909+0000 7fcec7e00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced010c480 0x7fced019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.909+0000 7fcec7e00640 1 -- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced0103060 msgr2=0x7fced01a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.909+0000 7fcec7e00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced0103060 0x7fced01a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.911 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.909+0000 7fcec7e00640 1 -- 172.21.15.146:0/834158101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced019b810 con 0x7fced0103980 2024-04-24T08:31:17.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.910+0000 7fcec7e00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced0103980 0x7fced01a0c50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fceb800d680 tx=0x7fceb800db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.910+0000 7fced4a00640 1 -- 172.21.15.146:0/834158101 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8017070 con 0x7fced0103980 2024-04-24T08:31:17.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.910+0000 7fced4a00640 1 -- 172.21.15.146:0/834158101 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceb80097c0 con 0x7fced0103980 2024-04-24T08:31:17.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.910+0000 7fced4a00640 1 -- 172.21.15.146:0/834158101 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8004cd0 con 0x7fced0103980 2024-04-24T08:31:17.912 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.910+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced019bb00 con 0x7fced0103980 2024-04-24T08:31:17.913 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.910+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced01a7560 con 0x7fced0103980 2024-04-24T08:31:17.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.915+0000 7fced4a00640 1 -- 172.21.15.146:0/834158101 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fceb8009970 con 0x7fced0103980 2024-04-24T08:31:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 Processor -- start 2024-04-24T08:31:17.917 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 -- start start 2024-04-24T08:31:17.918 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.921 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558b2301d1f0 con 0x558b230b7a30 2024-04-24T08:31:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558b22fc3350 con 0x558b230b8350 2024-04-24T08:31:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fced8baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558b230191b0 con 0x558b230c2a20 2024-04-24T08:31:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.917+0000 7fcecea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58032/0 (socket says 172.21.15.146:58032) 2024-04-24T08:31:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecea00640 1 -- 172.21.15.146:0/1878821825 learned_addr learned my addr 172.21.15.146:0/1878821825 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcec7e00640 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecf400640 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.923 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fced4a00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fceb40a3da0 0x7fceb40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fced4a00640 1 -- 172.21.15.146:0/834158101 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fceb80cff80 con 0x7fced0103980 2024-04-24T08:31:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecea00640 1 -- 172.21.15.146:0/1878821825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 msgr2=0x558b230c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.924 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecea00640 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecea00640 1 -- 172.21.15.146:0/1878821825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 msgr2=0x558b230b7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecea00640 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230b7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.918+0000 7fcecea00640 1 -- 172.21.15.146:0/1878821825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b23017f50 con 0x558b230b8350 2024-04-24T08:31:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.919+0000 7fcecf400640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fceb40a3da0 0x7fceb40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.919+0000 7fcecea00640 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230c24e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009900 tx=0x7fcec0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f438170d4c101db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.919+0000 7fcebfe00640 1 -- 172.21.15.146:0/1878821825 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec0035040 con 0x558b230b8350 2024-04-24T08:31:17.925 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.919+0000 7fcebfe00640 1 -- 172.21.15.146:0/1878821825 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec0004430 con 0x558b230b8350 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.919+0000 7fcebfe00640 1 -- 172.21.15.146:0/1878821825 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec00035f0 con 0x558b230b8350 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.920+0000 7fced8baa740 1 -- 172.21.15.146:0/1878821825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 msgr2=0x558b230c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.920+0000 7fced8baa740 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230c24e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009900 tx=0x7fcec0004040 comp rx=0 tx=0).stop 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fcecf400640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fceb40a3da0 0x7fceb40a6260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcec8009aa0 tx=0x7fcec8004a40 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fced8baa740 1 -- 172.21.15.146:0/1878821825 shutdown_connections 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fced8baa740 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fced8baa740 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230c24e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.926 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fced8baa740 1 --2- 172.21.15.146:0/1878821825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230b7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fced8baa740 1 -- 172.21.15.146:0/1878821825 >> 172.21.15.146:0/1878821825 conn(0x558b230b3d50 msgr2=0x558b230b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fced8baa740 1 -- 172.21.15.146:0/1878821825 shutdown_connections 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.921+0000 7fced8baa740 1 -- 172.21.15.146:0/1878821825 wait complete. 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.922+0000 7fced8baa740 1 -- start start 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.922+0000 7fced8baa740 1 Processor -- start 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.923+0000 7fced8baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b7a30 0x558b230bf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.923+0000 7fced8baa740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b8350 0x558b230bff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.927 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.923+0000 7fced8baa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230c2a20 0x558b230c0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.928 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.923+0000 7fced8baa740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x558b230c57b0 con 0x558b230b8350 2024-04-24T08:31:17.929 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.923+0000 7fced8baa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x558b22fd6690 con 0x558b230c2a20 2024-04-24T08:31:17.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.923+0000 7fced8baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558b230c5930 con 0x558b230b7a30 2024-04-24T08:31:17.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.924+0000 7fced8baa740 10 client.0 _ll_register_callbacks cb 0x558b23012650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:17.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.924+0000 7fcecea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b7a30 0x558b230bf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.924+0000 7fcecea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b7a30 0x558b230bf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:46458/0 (socket says 172.21.15.146:46458) 2024-04-24T08:31:17.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.924+0000 7fcecea00640 1 -- 172.21.15.146:0/245944418 learned_addr learned my addr 172.21.15.146:0/245944418 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.924+0000 7fcec7e00640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b8350 0x558b230bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.924+0000 7fcecf400640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230c2a20 0x558b230c0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.925+0000 7fcec7e00640 1 -- 172.21.15.146:0/245944418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b7a30 msgr2=0x558b230bf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.925+0000 7fcec7e00640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b7a30 0x558b230bf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.926+0000 7fcec7e00640 1 -- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230c2a20 msgr2=0x558b230c0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.926+0000 7fcec7e00640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230c2a20 0x558b230c0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.926+0000 7fcec7e00640 1 -- 172.21.15.146:0/245944418 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b22fd5920 con 0x558b230b8350 2024-04-24T08:31:17.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.926+0000 7fcecea00640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b7a30 0x558b230bf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.926+0000 7fcecf400640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230c2a20 0x558b230c0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.926+0000 7fcec7e00640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b8350 0x558b230bff00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fceb80cd630 tx=0x7fceb809b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea80be342a8f59b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.926+0000 7fcebcc00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.927+0000 7fcebcc00640 1 -- 172.21.15.146:0/245944418 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8020040 con 0x558b230b8350 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.927+0000 7fcebcc00640 1 -- 172.21.15.146:0/245944418 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fceb80cd8e0 con 0x558b230b8350 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.927+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec0004430 con 0x558b230b8350 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.928+0000 7fcebcc00640 1 -- 172.21.15.146:0/245944418 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fceb80cdae0 con 0x558b230b8350 2024-04-24T08:31:17.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.928+0000 7fced8baa740 10 client.17295 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:17.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.928+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558b22df97d0 con 0x558b230b8350 2024-04-24T08:31:17.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.929+0000 7fcebcc00640 1 -- 172.21.15.146:0/245944418 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec0004430 con 0x558b230b8350 2024-04-24T08:31:17.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.929+0000 7fcebcc00640 1 -- 172.21.15.146:0/245944418 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fceb809b3b0 con 0x558b230b8350 2024-04-24T08:31:17.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.929+0000 7fced8baa740 10 client.17295 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:17.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.929+0000 7fced8baa740 10 client.17295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:17.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.930+0000 7fced8baa740 20 client.17295 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:17.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.930+0000 7fced8baa740 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b230d32f0 0x558b230d5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.930+0000 7fced8baa740 4 client.17295 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:17.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.930+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b2302ac10 con 0x558b230d32f0 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.930+0000 7fcecea00640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b230d32f0 0x558b230d5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.931+0000 7fcebcc00640 10 client.17295 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.931+0000 7fcecea00640 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b230d32f0 0x558b230d5770 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2673ffc9796cb77d server_cookie=c300bf88c03d8f78 in_seq=0 out_seq=0 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.932+0000 7fcebcc00640 1 -- 172.21.15.146:0/245944418 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b2302ac10 con 0x558b230d32f0 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.932+0000 7fcebcc00640 10 client.17295 handle_command_reply: tid=0 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.932+0000 7fced8baa740 1 client.17295 shutdown 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b8350 msgr2=0x558b230bff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b8350 0x558b230bff00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fceb80cd630 tx=0x7fceb809b8c0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 shutdown_connections 2024-04-24T08:31:17.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b230d32f0 0x558b230d5770 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230c2a20 0x558b230c0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fcebcc00640 0 client.17295 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b8350 0x558b230bff00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 --2- 172.21.15.146:0/245944418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b7a30 0x558b230bf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 >> 172.21.15.146:0/245944418 conn(0x558b230b3d50 msgr2=0x558b230b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.933+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 shutdown_connections 2024-04-24T08:31:17.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.934+0000 7fced8baa740 1 -- 172.21.15.146:0/245944418 wait complete. 2024-04-24T08:31:17.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.934+0000 7fced8baa740 20 client.17295 trim_cache size 0 max 16384 2024-04-24T08:31:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 Processor -- start 2024-04-24T08:31:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 -- start start 2024-04-24T08:31:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b7a30 0x558b230ce670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b8350 0x558b230ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230c2a20 0x558b230c0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fced01b2750 con 0x558b230c2a20 2024-04-24T08:31:17.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fced01b0020 con 0x558b230b7a30 2024-04-24T08:31:17.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.967+0000 7fced8baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fced01ade90 con 0x558b230b8350 2024-04-24T08:31:17.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b7a30 0x558b230ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b7a30 0x558b230ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58056/0 (socket says 172.21.15.146:58056) 2024-04-24T08:31:17.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 -- 172.21.15.146:0/1393516553 learned_addr learned my addr 172.21.15.146:0/1393516553 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecf400640 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230c2a20 0x558b230c0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcec7e00640 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b8350 0x558b230ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 -- 172.21.15.146:0/1393516553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b8350 msgr2=0x558b230ba1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b8350 0x558b230ba1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 -- 172.21.15.146:0/1393516553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230c2a20 msgr2=0x558b230c0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230c2a20 0x558b230c0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.968+0000 7fcecea00640 1 -- 172.21.15.146:0/1393516553 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b22d2f3f0 con 0x558b230b7a30 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.969+0000 7fcecea00640 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b7a30 0x558b230ce670 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcec0004650 tx=0x7fcec00045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c5527cde3fc9c52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.969+0000 7fcebf400640 1 -- 172.21.15.146:0/1393516553 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec0035040 con 0x558b230b7a30 2024-04-24T08:31:17.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.969+0000 7fcebf400640 1 -- 172.21.15.146:0/1393516553 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec0053020 con 0x558b230b7a30 2024-04-24T08:31:17.972 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.969+0000 7fcebf400640 1 -- 172.21.15.146:0/1393516553 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec0031810 con 0x558b230b7a30 2024-04-24T08:31:17.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 -- 172.21.15.146:0/1393516553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b7a30 msgr2=0x558b230ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b7a30 0x558b230ce670 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcec0004650 tx=0x7fcec00045e0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 -- 172.21.15.146:0/1393516553 shutdown_connections 2024-04-24T08:31:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230b8350 0x558b230ba1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230c2a20 0x558b230c0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 --2- 172.21.15.146:0/1393516553 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b7a30 0x558b230ce670 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 -- 172.21.15.146:0/1393516553 >> 172.21.15.146:0/1393516553 conn(0x558b230b3d50 msgr2=0x558b230bf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 -- 172.21.15.146:0/1393516553 shutdown_connections 2024-04-24T08:31:17.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.970+0000 7fced8baa740 1 -- 172.21.15.146:0/1393516553 wait complete. 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.971+0000 7fced8baa740 1 -- start start 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.971+0000 7fced8baa740 1 Processor -- start 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fced8baa740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fced8baa740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230bc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fced8baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230bc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fced8baa740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fced01ae4e0 con 0x558b230b7a30 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fced8baa740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fced01af950 con 0x558b230b8350 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fced8baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fced01aebb0 con 0x558b230c2a20 2024-04-24T08:31:17.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fced8baa740 10 client.0 _ll_register_callbacks cb 0x7fceb80982c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcec7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230bc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcec7e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230bc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:58060/0 (socket says 172.21.15.146:58060) 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcec7e00640 1 -- 172.21.15.146:0/2434732833 learned_addr learned my addr 172.21.15.146:0/2434732833 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcecf400640 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcecea00640 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcec7e00640 1 -- 172.21.15.146:0/2434732833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 msgr2=0x558b230bc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcec7e00640 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230bc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.976 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcec7e00640 1 -- 172.21.15.146:0/2434732833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 msgr2=0x558b230cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:17.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.972+0000 7fcec7e00640 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.973+0000 7fcec7e00640 1 -- 172.21.15.146:0/2434732833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b22cef0f0 con 0x558b230b8350 2024-04-24T08:31:17.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.973+0000 7fcec7e00640 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230bc250 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fceb809bef0 tx=0x7fceb8005820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9434c631d8b08c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:17.977 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.973+0000 7fcebcc00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:31:17.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.973+0000 7fcebcc00640 1 -- 172.21.15.146:0/2434732833 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb809bd30 con 0x558b230b8350 2024-04-24T08:31:17.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.973+0000 7fcebcc00640 1 -- 172.21.15.146:0/2434732833 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fceb80052c0 con 0x558b230b8350 2024-04-24T08:31:17.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.973+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec0053020 con 0x558b230b8350 2024-04-24T08:31:17.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fcebcc00640 1 -- 172.21.15.146:0/2434732833 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fceb80a0260 con 0x558b230b8350 2024-04-24T08:31:17.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 10 client.27449 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:17.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558b22fe2810 con 0x558b230b8350 2024-04-24T08:31:17.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fcebcc00640 1 -- 172.21.15.146:0/2434732833 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec0053020 con 0x558b230b8350 2024-04-24T08:31:17.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fcebcc00640 1 -- 172.21.15.146:0/2434732833 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fceb80b5200 con 0x558b230b8350 2024-04-24T08:31:17.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 10 client.27449 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:17.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 10 client.27449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:17.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 20 client.27449 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:17.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b231420a0 0x558b231444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:17.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 4 client.27449 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:31:17.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.974+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558b23024a80 con 0x558b231420a0 2024-04-24T08:31:17.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.975+0000 7fcecea00640 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b231420a0 0x558b231444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:17.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.976+0000 7fcecea00640 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b231420a0 0x558b231444c0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1664deec173d587 server_cookie=5dd452ff45da2333 in_seq=0 out_seq=0 2024-04-24T08:31:17.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.976+0000 7fcebcc00640 10 client.27449 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.976+0000 7fcebcc00640 1 -- 172.21.15.146:0/2434732833 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x558b23024a80 con 0x558b231420a0 2024-04-24T08:31:17.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.976+0000 7fcebcc00640 10 client.27449 handle_command_reply: tid=0 2024-04-24T08:31:17.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.976+0000 7fced8baa740 1 client.27449 shutdown 2024-04-24T08:31:17.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 msgr2=0x558b230bc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230bc250 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fceb809bef0 tx=0x7fceb8005820 comp rx=0 tx=0).stop 2024-04-24T08:31:17.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 shutdown_connections 2024-04-24T08:31:17.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x558b231420a0 0x558b231444c0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b230c2a20 0x558b230bc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x558b230b8350 0x558b230bc250 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 --2- 172.21.15.146:0/2434732833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b230b7a30 0x558b230cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 >> 172.21.15.146:0/2434732833 conn(0x558b230b3d50 msgr2=0x558b230c01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fcebcc00640 0 client.27449 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:17.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 shutdown_connections 2024-04-24T08:31:17.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 1 -- 172.21.15.146:0/2434732833 wait complete. 2024-04-24T08:31:17.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.977+0000 7fced8baa740 20 client.27449 trim_cache size 0 max 16384 2024-04-24T08:31:17.985 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "scrub active (17 inodes in the stack)", 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": { 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "534e6d12-6bbc-439f-9ad7-508a87d11e15": { 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "path": "/", 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "tag": "534e6d12-6bbc-439f-9ad7-508a87d11e15", 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: "options": "recursive,force" 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:17.986 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:17.987 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:31:17.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.978+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fceb40a3da0 msgr2=0x7fceb40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.978+0000 7fcecfe00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fceb40a3da0 0x7fceb40a6260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcec8009aa0 tx=0x7fcec8004a40 comp rx=0 tx=0).stop 2024-04-24T08:31:17.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.978+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced0103980 msgr2=0x7fced01a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:17.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.978+0000 7fcecfe00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced0103980 0x7fced01a0c50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fceb800d680 tx=0x7fceb800db40 comp rx=0 tx=0).stop 2024-04-24T08:31:17.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.979+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 shutdown_connections 2024-04-24T08:31:17.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.979+0000 7fcecfe00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fceb40a3da0 0x7fceb40a6260 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.979+0000 7fcecfe00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fced0103980 0x7fced01a0c50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.979+0000 7fcecfe00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fced010c480 0x7fced019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.979+0000 7fcecfe00640 1 --2- 172.21.15.146:0/834158101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced0103060 0x7fced01a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:17.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.979+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 >> 172.21.15.146:0/834158101 conn(0x7fced00fc880 msgr2=0x7fced010c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:17.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.979+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 shutdown_connections 2024-04-24T08:31:17.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:17.980+0000 7fcecfe00640 1 -- 172.21.15.146:0/834158101 wait complete. 2024-04-24T08:31:18.006 INFO:tasks.cephfs.filesystem:scrub status for tag:534e6d12-6bbc-439f-9ad7-508a87d11e15 - {'path': '/', 'tag': '534e6d12-6bbc-439f-9ad7-508a87d11e15', 'options': 'recursive,force'} 2024-04-24T08:31:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: pgmap v314: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2024-04-24T08:31:18.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub queued for path: / 2024-04-24T08:31:18.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:31:18.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:31:18.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub summary: active paths [/] 2024-04-24T08:31:18.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:18 smithi177 ceph-mon[33046]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: pgmap v314: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2024-04-24T08:31:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub queued for path: / 2024-04-24T08:31:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:31:18.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:31:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub summary: active paths [/] 2024-04-24T08:31:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:18 smithi153 ceph-mon[32746]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: pgmap v314: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2024-04-24T08:31:18.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub queued for path: / 2024-04-24T08:31:18.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:31:18.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:31:18.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub summary: active paths [/] 2024-04-24T08:31:18.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:18.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:18 smithi146 ceph-mon[27805]: scrub summary: idle+waiting paths [/] 2024-04-24T08:31:20.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:20 smithi177 ceph-mon[33046]: pgmap v315: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:20.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:20 smithi153 ceph-mon[32746]: pgmap v315: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:20.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:20 smithi146 ceph-mon[27805]: pgmap v315: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:22 smithi146 ceph-mon[27805]: pgmap v316: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:22.938 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-24T08:31:22.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:22 smithi153 ceph-mon[32746]: pgmap v316: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:23.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:22 smithi177 ceph-mon[33046]: pgmap v316: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:31:23.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 -- 172.21.15.146:0/96374880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103020 msgr2=0x7fb2b8103400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/96374880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103020 0x7fb2b8103400 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009900 tx=0x7fb2a802a490 comp rx=0 tx=0).stop 2024-04-24T08:31:23.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 -- 172.21.15.146:0/96374880 shutdown_connections 2024-04-24T08:31:23.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/96374880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b810c3f0 0x7fb2b810e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.127 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/96374880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b8103940 0x7fb2b810be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/96374880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103020 0x7fb2b8103400 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 -- 172.21.15.146:0/96374880 >> 172.21.15.146:0/96374880 conn(0x7fb2b80fc820 msgr2=0x7fb2b80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:23.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.125+0000 7fb2bcc00640 1 -- 172.21.15.146:0/96374880 shutdown_connections 2024-04-24T08:31:23.128 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.126+0000 7fb2bcc00640 1 -- 172.21.15.146:0/96374880 wait complete. 2024-04-24T08:31:23.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.126+0000 7fb2bcc00640 1 Processor -- start 2024-04-24T08:31:23.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2bcc00640 1 -- start start 2024-04-24T08:31:23.130 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2bcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b8103020 0x7fb2b81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2bcc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103940 0x7fb2b81a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2bcc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b810c3f0 0x7fb2b819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2bcc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b8112210 con 0x7fb2b8103940 2024-04-24T08:31:23.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2bcc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b8112090 con 0x7fb2b810c3f0 2024-04-24T08:31:23.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2bcc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b8112390 con 0x7fb2b8103020 2024-04-24T08:31:23.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2b7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b8103020 0x7fb2b81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.131 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2b7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b8103020 0x7fb2b81a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:43468/0 (socket says 172.21.15.146:43468) 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2b7400640 1 -- 172.21.15.146:0/3169430437 learned_addr learned my addr 172.21.15.146:0/3169430437 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2afe00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103940 0x7fb2b81a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.127+0000 7fb2b7e00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b810c3f0 0x7fb2b819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b7400640 1 -- 172.21.15.146:0/3169430437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b810c3f0 msgr2=0x7fb2b819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b7400640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b810c3f0 0x7fb2b819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b7400640 1 -- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103940 msgr2=0x7fb2b81a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b7400640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103940 0x7fb2b81a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.132 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b7400640 1 -- 172.21.15.146:0/3169430437 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b819afe0 con 0x7fb2b8103020 2024-04-24T08:31:23.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b7400640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b8103020 0x7fb2b81a0660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8004770 tx=0x7fb2a8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:23.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b4c00640 1 -- 172.21.15.146:0/3169430437 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a803f070 con 0x7fb2b8103020 2024-04-24T08:31:23.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b4c00640 1 -- 172.21.15.146:0/3169430437 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2a8031ca0 con 0x7fb2b8103020 2024-04-24T08:31:23.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b819b270 con 0x7fb2b8103020 2024-04-24T08:31:23.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2b4c00640 1 -- 172.21.15.146:0/3169430437 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8035030 con 0x7fb2b8103020 2024-04-24T08:31:23.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.128+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b81a7400 con 0x7fb2b8103020 2024-04-24T08:31:23.133 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.129+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b8069f30 con 0x7fb2b8103020 2024-04-24T08:31:23.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.132+0000 7fb2b4c00640 1 -- 172.21.15.146:0/3169430437 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fb2a8042070 con 0x7fb2b8103020 2024-04-24T08:31:23.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.133+0000 7fb2b4c00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb29c0a3d10 0x7fb29c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.133+0000 7fb2b4c00640 1 -- 172.21.15.146:0/3169430437 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb2a80f1410 con 0x7fb2b8103020 2024-04-24T08:31:23.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.133+0000 7fb2afe00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb29c0a3d10 0x7fb29c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.135 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.134+0000 7fb2afe00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb29c0a3d10 0x7fb29c0a61d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2b819bf70 tx=0x7fb2a0005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:23.136 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.135+0000 7fb2b4c00640 1 -- 172.21.15.146:0/3169430437 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2a80bd4b0 con 0x7fb2b8103020 2024-04-24T08:31:23.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.357+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fb2b80fe320 con 0x7fb29c0a3d10 2024-04-24T08:31:23.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.368+0000 7fb2b4c00640 1 -- 172.21.15.146:0/3169430437 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fb2b80fe320 con 0x7fb29c0a3d10 2024-04-24T08:31:23.369 INFO:teuthology.orchestra.run.smithi146.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-24T00:00:00", "created": "2024-04-24T08:21:20", "first": "2024-04-24T08:22:00", "last": "2024-04-24T08:31:00", "last_pruned": "2024-04-24T08:31:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-24T08:31:23.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb29c0a3d10 msgr2=0x7fb29c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb29c0a3d10 0x7fb29c0a61d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2b819bf70 tx=0x7fb2a0005f90 comp rx=0 tx=0).stop 2024-04-24T08:31:23.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b8103020 msgr2=0x7fb2b81a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b8103020 0x7fb2b81a0660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8004770 tx=0x7fb2a8003c10 comp rx=0 tx=0).stop 2024-04-24T08:31:23.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 shutdown_connections 2024-04-24T08:31:23.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb29c0a3d10 0x7fb29c0a61d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb2b810c3f0 0x7fb2b819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b8103940 0x7fb2b81a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 --2- 172.21.15.146:0/3169430437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2b8103020 0x7fb2b81a0660 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 >> 172.21.15.146:0/3169430437 conn(0x7fb2b80fc820 msgr2=0x7fb2b80fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:23.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 shutdown_connections 2024-04-24T08:31:23.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.371+0000 7fb2bcc00640 1 -- 172.21.15.146:0/3169430437 wait complete. 2024-04-24T08:31:23.388 DEBUG:teuthology.orchestra.run.smithi146:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-24T08:31:23.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 -- 172.21.15.146:0/2567721142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac104c60 msgr2=0x7f0eac108ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 --2- 172.21.15.146:0/2567721142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac104c60 0x7f0eac108ca0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0e98009900 tx=0x7f0e9802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 -- 172.21.15.146:0/2567721142 shutdown_connections 2024-04-24T08:31:23.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 --2- 172.21.15.146:0/2567721142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac109380 0x7f0eac110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 --2- 172.21.15.146:0/2567721142 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac104c60 0x7f0eac108ca0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 --2- 172.21.15.146:0/2567721142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eac1042b0 0x7f0eac104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 -- 172.21.15.146:0/2567721142 >> 172.21.15.146:0/2567721142 conn(0x7f0eac0ffb10 msgr2=0x7f0eac101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:23.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 -- 172.21.15.146:0/2567721142 shutdown_connections 2024-04-24T08:31:23.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.565+0000 7f0eb1000640 1 -- 172.21.15.146:0/2567721142 wait complete. 2024-04-24T08:31:23.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.566+0000 7f0eb1000640 1 Processor -- start 2024-04-24T08:31:23.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.566+0000 7f0eb1000640 1 -- start start 2024-04-24T08:31:23.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eb1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac1042b0 0x7f0eac1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eb1000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac109380 0x7f0eac1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eb1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eac19a790 0x7f0eac19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.569 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eb1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac1143f0 con 0x7f0eac1042b0 2024-04-24T08:31:23.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eb1000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac114270 con 0x7f0eac109380 2024-04-24T08:31:23.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eb1000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac114570 con 0x7f0eac19a790 2024-04-24T08:31:23.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac109380 0x7f0eac1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac1042b0 0x7f0eac1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eabe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eac19a790 0x7f0eac19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eaaa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac109380 0x7f0eac1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:40896/0 (socket says 172.21.15.146:40896) 2024-04-24T08:31:23.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eaaa00640 1 -- 172.21.15.146:0/3213172673 learned_addr learned my addr 172.21.15.146:0/3213172673 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:23.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eabe00640 1 -- 172.21.15.146:0/3213172673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac109380 msgr2=0x7f0eac1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.567+0000 7f0eabe00640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac109380 0x7f0eac1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.568+0000 7f0eabe00640 1 -- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac1042b0 msgr2=0x7f0eac1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.568+0000 7f0eabe00640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac1042b0 0x7f0eac1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.568+0000 7f0eabe00640 1 -- 172.21.15.146:0/3213172673 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac19b390 con 0x7f0eac19a790 2024-04-24T08:31:23.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.568+0000 7f0eabe00640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eac19a790 0x7f0eac19ac10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009fd0 tx=0x7f0ea400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:23.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.569+0000 7f0ea3e00640 1 -- 172.21.15.146:0/3213172673 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400ce20 con 0x7f0eac19a790 2024-04-24T08:31:23.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.569+0000 7f0ea3e00640 1 -- 172.21.15.146:0/3213172673 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ea4005020 con 0x7f0eac19a790 2024-04-24T08:31:23.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.569+0000 7f0ea3e00640 1 -- 172.21.15.146:0/3213172673 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4010720 con 0x7f0eac19a790 2024-04-24T08:31:23.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.569+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac1a7420 con 0x7f0eac19a790 2024-04-24T08:31:23.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.569+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac1a7820 con 0x7f0eac19a790 2024-04-24T08:31:23.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.570+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac105780 con 0x7f0eac19a790 2024-04-24T08:31:23.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.572+0000 7f0ea3e00640 1 -- 172.21.15.146:0/3213172673 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f0ea4005540 con 0x7f0eac19a790 2024-04-24T08:31:23.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.573+0000 7f0ea3e00640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0e900a3d90 0x7f0e900a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:23.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.573+0000 7f0ea3e00640 1 -- 172.21.15.146:0/3213172673 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f0ea40d0e40 con 0x7f0eac19a790 2024-04-24T08:31:23.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.573+0000 7f0eab400640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0e900a3d90 0x7f0e900a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:23.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.574+0000 7f0eab400640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0e900a3d90 0x7f0e900a6250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0095f0 tx=0x7f0e9c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:23.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.577+0000 7f0ea3e00640 1 -- 172.21.15.146:0/3213172673 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ea409d470 con 0x7f0eac19a790 2024-04-24T08:31:23.802 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.802+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 --> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f0eac104c60 con 0x7f0e900a3d90 2024-04-24T08:31:23.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.812+0000 7f0ea3e00640 1 -- 172.21.15.146:0/3213172673 <== mgr.15087 v2:172.21.15.146:6800/827025587 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f0eac104c60 con 0x7f0e900a3d90 2024-04-24T08:31:23.813 INFO:teuthology.orchestra.run.smithi146.stdout:/ 1m 6m3h 2024-04-24T08:31:23.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0e900a3d90 msgr2=0x7f0e900a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0e900a3d90 0x7f0e900a6250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0095f0 tx=0x7f0e9c005f90 comp rx=0 tx=0).stop 2024-04-24T08:31:23.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eac19a790 msgr2=0x7f0eac19ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:23.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eac19a790 0x7f0eac19ac10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009fd0 tx=0x7f0ea400ef10 comp rx=0 tx=0).stop 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 shutdown_connections 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f0e900a3d90 0x7f0e900a6250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0eac19a790 0x7f0eac19ac10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0eac109380 0x7f0eac1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 --2- 172.21.15.146:0/3213172673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac1042b0 0x7f0eac1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 >> 172.21.15.146:0/3213172673 conn(0x7f0eac0ffb10 msgr2=0x7f0eac100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 shutdown_connections 2024-04-24T08:31:23.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:23.815+0000 7f0eb1000640 1 -- 172.21.15.146:0/3213172673 wait complete. 2024-04-24T08:31:23.834 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-24T08:31:23.845 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-24T08:31:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:24 smithi153 ceph-mon[32746]: from='client.27460 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:31:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:24 smithi153 ceph-mon[32746]: pgmap v317: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:31:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:24 smithi153 ceph-mon[32746]: from='client.27466 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:31:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:24 smithi146 ceph-mon[27805]: from='client.27460 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:31:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:24 smithi146 ceph-mon[27805]: pgmap v317: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:31:24.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:24 smithi146 ceph-mon[27805]: from='client.27466 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:31:25.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:24 smithi177 ceph-mon[33046]: from='client.27460 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:31:25.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:24 smithi177 ceph-mon[33046]: pgmap v317: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:31:25.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:24 smithi177 ceph-mon[33046]: from='client.27466 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:31:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:26 smithi153 ceph-mon[32746]: pgmap v318: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:31:26.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:26 smithi146 ceph-mon[27805]: pgmap v318: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:31:27.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:26 smithi177 ceph-mon[33046]: pgmap v318: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:31:27.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:27 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:31:27.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:27 smithi153 ceph-mon[32746]: scrub summary: idle 2024-04-24T08:31:27.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:27 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:31:27.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:27 smithi146 ceph-mon[27805]: scrub summary: idle 2024-04-24T08:31:28.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:27 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:31:28.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:27 smithi177 ceph-mon[33046]: scrub summary: idle 2024-04-24T08:31:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:28 smithi153 ceph-mon[32746]: pgmap v319: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 89 B/s wr, 10 op/s 2024-04-24T08:31:28.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:28 smithi146 ceph-mon[27805]: pgmap v319: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 89 B/s wr, 10 op/s 2024-04-24T08:31:29.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:28 smithi177 ceph-mon[33046]: pgmap v319: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 89 B/s wr, 10 op/s 2024-04-24T08:31:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:30 smithi153 ceph-mon[32746]: pgmap v320: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:30.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:30 smithi146 ceph-mon[27805]: pgmap v320: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:31.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:30 smithi177 ceph-mon[33046]: pgmap v320: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:32.854 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:32 smithi146 ceph-mon[27805]: pgmap v321: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:32 smithi153 ceph-mon[32746]: pgmap v321: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:33.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:32 smithi177 ceph-mon[33046]: pgmap v321: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:34.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:34 smithi153 ceph-mon[32746]: pgmap v322: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T08:31:34.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:34 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:31:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:34 smithi146 ceph-mon[27805]: pgmap v322: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T08:31:34.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:34 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:31:35.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:34 smithi177 ceph-mon[33046]: pgmap v322: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-24T08:31:35.043 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:34 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:31:36.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:35 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:35 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:35 smithi153 ceph-mon[32746]: pgmap v323: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:36.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:35 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:35 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:35 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:35 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:35 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:35 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:35 smithi146 ceph-mon[27805]: pgmap v323: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:36.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:35 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:35 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:35 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:35 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:35 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:35 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:35 smithi177 ceph-mon[33046]: pgmap v323: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:31:36.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:35 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:35 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:35 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:36.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:35 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:37.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-24T08:31:37.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-24T08:31:37.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:36 smithi153 ceph-mon[32746]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-24T08:31:37.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.244 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-24T08:31:37.245 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-24T08:31:37.245 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:36 smithi146 ceph-mon[27805]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-24T08:31:37.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-24T08:31:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.317 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: Unable to set osd_memory_target on smithi177 to 184727961: error parsing value: Value '184727961' is below minimum 939524096 2024-04-24T08:31:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi153 to 176.2M 2024-04-24T08:31:37.318 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:36 smithi177 ceph-mon[33046]: Unable to set osd_memory_target on smithi153 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-24T08:31:38.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:38 smithi153 ceph-mon[32746]: pgmap v324: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-24T08:31:38.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:38 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:38.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:38 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:38.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:38 smithi146 ceph-mon[27805]: pgmap v324: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-24T08:31:38.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:38 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:38.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:38 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:39.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:38 smithi177 ceph-mon[33046]: pgmap v324: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-24T08:31:39.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:38 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:39.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:38 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:39.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:31:39.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:31:39.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:39 smithi153 ceph-mon[32746]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:39.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:39.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:31:39.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:31:39.996 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:39 smithi146 ceph-mon[27805]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:40.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.066 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:31:40.067 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:31:40.068 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:31:40.068 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:39 smithi177 ceph-mon[33046]: from='mgr.15087 ' entity='mgr.x' 2024-04-24T08:31:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:40 smithi153 ceph-mon[32746]: Adjusting osd_memory_target on smithi146 to 176.4M 2024-04-24T08:31:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:40 smithi153 ceph-mon[32746]: Unable to set osd_memory_target on smithi146 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-24T08:31:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:40 smithi153 ceph-mon[32746]: pgmap v325: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:40.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:40 smithi146 ceph-mon[27805]: Adjusting osd_memory_target on smithi146 to 176.4M 2024-04-24T08:31:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:40 smithi146 ceph-mon[27805]: Unable to set osd_memory_target on smithi146 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-24T08:31:40.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:40 smithi146 ceph-mon[27805]: pgmap v325: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:40 smithi177 ceph-mon[33046]: Adjusting osd_memory_target on smithi146 to 176.4M 2024-04-24T08:31:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:40 smithi177 ceph-mon[33046]: Unable to set osd_memory_target on smithi146 to 185064857: error parsing value: Value '185064857' is below minimum 939524096 2024-04-24T08:31:41.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:40 smithi177 ceph-mon[33046]: pgmap v325: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:42.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:42 smithi153 ceph-mon[32746]: pgmap v326: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:42.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:42 smithi146 ceph-mon[27805]: pgmap v326: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:43.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:42 smithi177 ceph-mon[33046]: pgmap v326: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:44.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:44 smithi153 ceph-mon[32746]: pgmap v327: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:44.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:44 smithi146 ceph-mon[27805]: pgmap v327: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:45.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:44 smithi177 ceph-mon[33046]: pgmap v327: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:46.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:46 smithi153 ceph-mon[32746]: pgmap v328: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:46.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:46 smithi146 ceph-mon[27805]: pgmap v328: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:47.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:46 smithi177 ceph-mon[33046]: pgmap v328: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:48.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:31:48.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:31:48.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 -- 172.21.15.146:0/2528108522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108f40 msgr2=0x7f46941153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 --2- 172.21.15.146:0/2528108522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108f40 0x7f46941153b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f468c009ca0 tx=0x7f468c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 -- 172.21.15.146:0/2528108522 shutdown_connections 2024-04-24T08:31:48.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 --2- 172.21.15.146:0/2528108522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108f40 0x7f46941153b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 --2- 172.21.15.146:0/2528108522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108580 0x7f4694108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 --2- 172.21.15.146:0/2528108522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f469410d500 0x7f469410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 -- 172.21.15.146:0/2528108522 >> 172.21.15.146:0/2528108522 conn(0x7f4694077e20 msgr2=0x7f4694106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 -- 172.21.15.146:0/2528108522 shutdown_connections 2024-04-24T08:31:48.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.224+0000 7f4693200640 1 -- 172.21.15.146:0/2528108522 wait complete. 2024-04-24T08:31:48.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.225+0000 7f4693200640 1 Processor -- start 2024-04-24T08:31:48.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.225+0000 7f4693200640 1 -- start start 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.225+0000 7f4693200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108580 0x7f46941a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4693200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108f40 0x7f46941a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4693200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f469410d500 0x7f469419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4693200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f46941188b0 con 0x7f4694108f40 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4693200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4694118730 con 0x7f4694108580 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4693200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4694118a30 con 0x7f469410d500 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108f40 0x7f46941a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108f40 0x7f46941a4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55256/0 (socket says 172.21.15.146:55256) 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 -- 172.21.15.146:0/472734365 learned_addr learned my addr 172.21.15.146:0/472734365 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691e00640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108580 0x7f46941a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4692800640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f469410d500 0x7f469419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 -- 172.21.15.146:0/472734365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f469410d500 msgr2=0x7f469419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f469410d500 0x7f469419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 -- 172.21.15.146:0/472734365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108580 msgr2=0x7f46941a4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108580 0x7f46941a4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.226+0000 7f4691400640 1 -- 172.21.15.146:0/472734365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469419f230 con 0x7f4694108f40 2024-04-24T08:31:48.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.227+0000 7f4691400640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108f40 0x7f46941a4e40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f468000d950 tx=0x7f468000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.227+0000 7f468aa00640 1 -- 172.21.15.146:0/472734365 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680017030 con 0x7f4694108f40 2024-04-24T08:31:48.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.227+0000 7f468aa00640 1 -- 172.21.15.146:0/472734365 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4680004020 con 0x7f4694108f40 2024-04-24T08:31:48.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.227+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469419f520 con 0x7f4694108f40 2024-04-24T08:31:48.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.227+0000 7f468aa00640 1 -- 172.21.15.146:0/472734365 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46800047a0 con 0x7f4694108f40 2024-04-24T08:31:48.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.227+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46941ab7c0 con 0x7f4694108f40 2024-04-24T08:31:48.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.230+0000 7f469bdb9740 1 Processor -- start 2024-04-24T08:31:48.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f469bdb9740 1 -- start start 2024-04-24T08:31:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a466e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a467350 0x564e0a4714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a473e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f469bdb9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564e0a3cc1f0 con 0x564e0a471a20 2024-04-24T08:31:48.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f469bdb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e0a372350 con 0x564e0a466a30 2024-04-24T08:31:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f469bdb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e0a3c81b0 con 0x564e0a467350 2024-04-24T08:31:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f468aa00640 1 -- 172.21.15.146:0/472734365 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f46800041d0 con 0x7f4694108f40 2024-04-24T08:31:48.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f4691e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a467350 0x564e0a4714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f4691e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a467350 0x564e0a4714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33052/0 (socket says 172.21.15.146:33052) 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f4691e00640 1 -- 172.21.15.146:0/1931183067 learned_addr learned my addr 172.21.15.146:0/1931183067 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f4692800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a466e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.231+0000 7f4691400640 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a473e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f4691e00640 1 -- 172.21.15.146:0/1931183067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 msgr2=0x564e0a466e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f4691e00640 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a466e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f4691e00640 1 -- 172.21.15.146:0/1931183067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 msgr2=0x564e0a473e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f4691e00640 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a473e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f4691e00640 1 -- 172.21.15.146:0/1931183067 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e0a3c6f50 con 0x564e0a467350 2024-04-24T08:31:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f4691e00640 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a467350 0x564e0a4714e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4684009c20 tx=0x7f4684011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b3cda2de35ec22 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f467f400640 1 -- 172.21.15.146:0/1931183067 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684009dd0 con 0x564e0a467350 2024-04-24T08:31:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f467f400640 1 -- 172.21.15.146:0/1931183067 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4684005020 con 0x564e0a467350 2024-04-24T08:31:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.232+0000 7f467f400640 1 -- 172.21.15.146:0/1931183067 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684004ba0 con 0x564e0a467350 2024-04-24T08:31:48.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f468aa00640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f46780a3d20 0x7f46780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 -- 172.21.15.146:0/1931183067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a467350 msgr2=0x564e0a4714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a467350 0x564e0a4714e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4684009c20 tx=0x7f4684011810 comp rx=0 tx=0).stop 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f468aa00640 1 -- 172.21.15.146:0/472734365 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f46800d0240 con 0x7f4694108f40 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 -- 172.21.15.146:0/1931183067 shutdown_connections 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a467350 0x564e0a4714e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a473e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1931183067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a466e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 -- 172.21.15.146:0/1931183067 >> 172.21.15.146:0/1931183067 conn(0x564e0a462d50 msgr2=0x564e0a463130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f4692800640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f46780a3d20 0x7f46780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 -- 172.21.15.146:0/1931183067 shutdown_connections 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.233+0000 7f469bdb9740 1 -- 172.21.15.146:0/1931183067 wait complete. 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.234+0000 7f469bdb9740 1 -- start start 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.234+0000 7f4692800640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f46780a3d20 0x7f46780a61e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f468c009430 tx=0x7f468c009360 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.234+0000 7f469bdb9740 1 Processor -- start 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a46e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a467350 0x564e0a46eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a471a20 0x564e0a46f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f469bdb9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x564e0a474810 con 0x564e0a467350 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f469bdb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564e0a385690 con 0x564e0a466a30 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f469bdb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e0a474990 con 0x564e0a471a20 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f469bdb9740 10 client.0 _ll_register_callbacks cb 0x564e0a3c1650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a467350 0x564e0a46eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a467350 0x564e0a46eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55276/0 (socket says 172.21.15.146:55276) 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 -- 172.21.15.146:0/2200087354 learned_addr learned my addr 172.21.15.146:0/2200087354 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4692800640 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a471a20 0x564e0a46f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691e00640 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a46e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 -- 172.21.15.146:0/2200087354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a471a20 msgr2=0x564e0a46f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a471a20 0x564e0a46f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 -- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 msgr2=0x564e0a46e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a46e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.235+0000 7f4691400640 1 -- 172.21.15.146:0/2200087354 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e0a384920 con 0x564e0a467350 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f4691400640 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a467350 0x564e0a46eed0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f46800cd980 tx=0x7f468009c030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b974fcac7aba27 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f4673e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f4673e00640 1 -- 172.21.15.146:0/2200087354 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46800a0320 con 0x564e0a467350 2024-04-24T08:31:48.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f4673e00640 1 -- 172.21.15.146:0/2200087354 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f46800cdba0 con 0x564e0a467350 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4684005020 con 0x564e0a467350 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f4673e00640 1 -- 172.21.15.146:0/2200087354 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f46800cdda0 con 0x564e0a467350 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f469bdb9740 10 client.17325 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564e0a1a87d0 con 0x564e0a467350 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.236+0000 7f4673e00640 1 -- 172.21.15.146:0/2200087354 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4684005020 con 0x564e0a467350 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f4673e00640 1 -- 172.21.15.146:0/2200087354 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4680096ce0 con 0x564e0a467350 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f469bdb9740 10 client.17325 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:48.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f469bdb9740 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:48.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f469bdb9740 20 client.17325 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:48.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f469bdb9740 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a481e10 0x564e0a484260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f469bdb9740 4 client.17325 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:48.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e0a3d9c10 con 0x564e0a481e10 2024-04-24T08:31:48.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.237+0000 7f4691e00640 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a481e10 0x564e0a484260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.238+0000 7f4691e00640 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a481e10 0x564e0a484260 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e08a75e4c25dba02 server_cookie=286ef727788d54b1 in_seq=0 out_seq=0 2024-04-24T08:31:48.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.238+0000 7f4673e00640 10 client.17325 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:48.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f4673e00640 1 -- 172.21.15.146:0/2200087354 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e0a3d9c10 con 0x564e0a481e10 2024-04-24T08:31:48.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f4673e00640 10 client.17325 handle_command_reply: tid=0 2024-04-24T08:31:48.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 client.17325 shutdown 2024-04-24T08:31:48.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a467350 msgr2=0x564e0a46eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a467350 0x564e0a46eed0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f46800cd980 tx=0x7f468009c030 comp rx=0 tx=0).stop 2024-04-24T08:31:48.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 shutdown_connections 2024-04-24T08:31:48.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a481e10 0x564e0a484260 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a471a20 0x564e0a46f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a467350 0x564e0a46eed0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 --2- 172.21.15.146:0/2200087354 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a466a30 0x564e0a46e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 >> 172.21.15.146:0/2200087354 conn(0x564e0a462d50 msgr2=0x564e0a461fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f4673e00640 0 client.17325 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:48.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 shutdown_connections 2024-04-24T08:31:48.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.239+0000 7f469bdb9740 1 -- 172.21.15.146:0/2200087354 wait complete. 2024-04-24T08:31:48.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.240+0000 7f469bdb9740 20 client.17325 trim_cache size 0 max 16384 2024-04-24T08:31:48.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 Processor -- start 2024-04-24T08:31:48.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 -- start start 2024-04-24T08:31:48.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a466a30 0x564e0a47d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a4691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a489b30 0x564e0a489fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b6a40 con 0x564e0a471a20 2024-04-24T08:31:48.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b4310 con 0x564e0a489b30 2024-04-24T08:31:48.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f469bdb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b2180 con 0x564e0a466a30 2024-04-24T08:31:48.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.259+0000 7f4691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a4691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a466a30 0x564e0a47d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a4691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55286/0 (socket says 172.21.15.146:55286) 2024-04-24T08:31:48.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691400640 1 -- 172.21.15.146:0/1475645260 learned_addr learned my addr 172.21.15.146:0/1475645260 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4692800640 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a489b30 0x564e0a489fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691e00640 1 -- 172.21.15.146:0/1475645260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a489b30 msgr2=0x564e0a489fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691e00640 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a489b30 0x564e0a489fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691e00640 1 -- 172.21.15.146:0/1475645260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 msgr2=0x564e0a4691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691e00640 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a4691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691e00640 1 -- 172.21.15.146:0/1475645260 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e0a0de3f0 con 0x564e0a466a30 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691400640 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a4691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.260+0000 7f4691e00640 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a466a30 0x564e0a47d760 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4684000c00 tx=0x7f46840045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e13bb2dca7cc4268 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f467ea00640 1 -- 172.21.15.146:0/1475645260 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468401d040 con 0x564e0a466a30 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f467ea00640 1 -- 172.21.15.146:0/1475645260 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46840052d0 con 0x564e0a466a30 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f467ea00640 1 -- 172.21.15.146:0/1475645260 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684004d30 con 0x564e0a466a30 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 -- 172.21.15.146:0/1475645260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a466a30 msgr2=0x564e0a47d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a466a30 0x564e0a47d760 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4684000c00 tx=0x7f46840045a0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 -- 172.21.15.146:0/1475645260 shutdown_connections 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a489b30 0x564e0a489fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a471a20 0x564e0a4691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 --2- 172.21.15.146:0/1475645260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a466a30 0x564e0a47d760 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 -- 172.21.15.146:0/1475645260 >> 172.21.15.146:0/1475645260 conn(0x564e0a462d50 msgr2=0x564e0a4701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 -- 172.21.15.146:0/1475645260 shutdown_connections 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.261+0000 7f469bdb9740 1 -- 172.21.15.146:0/1475645260 wait complete. 2024-04-24T08:31:48.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 -- start start 2024-04-24T08:31:48.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 Processor -- start 2024-04-24T08:31:48.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a466a30 0x564e0a479c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a471a20 0x564e0a47a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a489b30 0x564e0a476c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b27d0 con 0x564e0a466a30 2024-04-24T08:31:48.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b3c40 con 0x564e0a471a20 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46941b2ea0 con 0x564e0a489b30 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.262+0000 7f469bdb9740 10 client.0 _ll_register_callbacks cb 0x7f4680098480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a466a30 0x564e0a479c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a466a30 0x564e0a479c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55292/0 (socket says 172.21.15.146:55292) 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 -- 172.21.15.146:0/751290648 learned_addr learned my addr 172.21.15.146:0/751290648 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691400640 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a471a20 0x564e0a47a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4692800640 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a489b30 0x564e0a476c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 -- 172.21.15.146:0/751290648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a489b30 msgr2=0x564e0a476c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a489b30 0x564e0a476c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 -- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a471a20 msgr2=0x564e0a47a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a471a20 0x564e0a47a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 -- 172.21.15.146:0/751290648 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e0a09e0f0 con 0x564e0a466a30 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4691e00640 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a466a30 0x564e0a479c70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f468400a010 tx=0x7f4684011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27955376dae06fa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.263+0000 7f4673e00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.264+0000 7f4673e00640 1 -- 172.21.15.146:0/751290648 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468401d050 con 0x564e0a466a30 2024-04-24T08:31:48.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.264+0000 7f4673e00640 1 -- 172.21.15.146:0/751290648 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f4684032040 con 0x564e0a466a30 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.264+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46840052d0 con 0x564e0a466a30 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.264+0000 7f4673e00640 1 -- 172.21.15.146:0/751290648 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4684005480 con 0x564e0a466a30 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.264+0000 7f469bdb9740 10 client.17331 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.264+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564e0a391810 con 0x564e0a466a30 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.264+0000 7f4673e00640 1 -- 172.21.15.146:0/751290648 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46840052d0 con 0x564e0a466a30 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f4673e00640 1 -- 172.21.15.146:0/751290648 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f468403f030 con 0x564e0a466a30 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f469bdb9740 10 client.17331 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:48.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f469bdb9740 10 client.17331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f469bdb9740 20 client.17331 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f469bdb9740 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a4e8f50 0x564e0a4eb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f469bdb9740 4 client.17331 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e0a3d3a80 con 0x564e0a4e8f50 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.265+0000 7f4691400640 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a4e8f50 0x564e0a4eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f4691400640 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a4e8f50 0x564e0a4eb370 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3435011636e18cdb server_cookie=909f75ed6772ff0b in_seq=0 out_seq=0 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f4673e00640 10 client.17331 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f4673e00640 1 -- 172.21.15.146:0/751290648 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564e0a3d3a80 con 0x564e0a4e8f50 2024-04-24T08:31:48.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f4673e00640 10 client.17331 handle_command_reply: tid=0 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 client.17331 shutdown 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a466a30 msgr2=0x564e0a479c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a466a30 0x564e0a479c70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f468400a010 tx=0x7f4684011e50 comp rx=0 tx=0).stop 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 shutdown_connections 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x564e0a4e8f50 0x564e0a4eb370 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e0a489b30 0x564e0a476c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x564e0a471a20 0x564e0a47a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f4673e00640 0 client.17331 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:48.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 --2- 172.21.15.146:0/751290648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x564e0a466a30 0x564e0a479c70 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 >> 172.21.15.146:0/751290648 conn(0x564e0a462d50 msgr2=0x564e0a4893b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 shutdown_connections 2024-04-24T08:31:48.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 1 -- 172.21.15.146:0/751290648 wait complete. 2024-04-24T08:31:48.275 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.266+0000 7f469bdb9740 20 client.17331 trim_cache size 0 max 16384 2024-04-24T08:31:48.275 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:31:48.276 INFO:teuthology.orchestra.run.smithi146.stdout: "status": "no active scrubs running", 2024-04-24T08:31:48.276 INFO:teuthology.orchestra.run.smithi146.stdout: "scrubs": {} 2024-04-24T08:31:48.276 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:31:48.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f46780a3d20 msgr2=0x7f46780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f46780a3d20 0x7f46780a61e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f468c009430 tx=0x7f468c009360 comp rx=0 tx=0).stop 2024-04-24T08:31:48.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108f40 msgr2=0x7f46941a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108f40 0x7f46941a4e40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f468000d950 tx=0x7f468000de10 comp rx=0 tx=0).stop 2024-04-24T08:31:48.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 shutdown_connections 2024-04-24T08:31:48.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f46780a3d20 0x7f46780a61e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.277 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f469410d500 0x7f469419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4694108f40 0x7f46941a4e40 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 --2- 172.21.15.146:0/472734365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4694108580 0x7f46941a4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 >> 172.21.15.146:0/472734365 conn(0x7f4694077e20 msgr2=0x7f46940782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 shutdown_connections 2024-04-24T08:31:48.278 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.267+0000 7f4693200640 1 -- 172.21.15.146:0/472734365 wait complete. 2024-04-24T08:31:48.286 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:31:48.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:31:48.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 -- 172.21.15.146:0/2330765894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a4104440 msgr2=0x7f01a4104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 --2- 172.21.15.146:0/2330765894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a4104440 0x7f01a4104820 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0194009860 tx=0x7f019402a340 comp rx=0 tx=0).stop 2024-04-24T08:31:48.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 -- 172.21.15.146:0/2330765894 shutdown_connections 2024-04-24T08:31:48.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 --2- 172.21.15.146:0/2330765894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4109510 0x7f01a41110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 --2- 172.21.15.146:0/2330765894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01a4104df0 0x7f01a4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 --2- 172.21.15.146:0/2330765894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a4104440 0x7f01a4104820 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 -- 172.21.15.146:0/2330765894 >> 172.21.15.146:0/2330765894 conn(0x7f01a40ffcc0 msgr2=0x7f01a41020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 -- 172.21.15.146:0/2330765894 shutdown_connections 2024-04-24T08:31:48.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.485+0000 7f01aae00640 1 -- 172.21.15.146:0/2330765894 wait complete. 2024-04-24T08:31:48.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.486+0000 7f01aae00640 1 Processor -- start 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01aae00640 1 -- start start 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01aae00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4104440 0x7f01a41a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01aae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01a4109510 0x7f01a41a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01aae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a419a970 0x7f01a419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01aae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f01a41145d0 con 0x7f01a4109510 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01aae00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01a4114450 con 0x7f01a4104440 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01aae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01a4114750 con 0x7f01a419a970 2024-04-24T08:31:48.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f019b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01a4109510 0x7f01a41a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01a3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4104440 0x7f01a41a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01a3e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4104440 0x7f01a41a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:55680/0 (socket says 172.21.15.146:55680) 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01a3e00640 1 -- 172.21.15.146:0/4215477836 learned_addr learned my addr 172.21.15.146:0/4215477836 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.487+0000 7f01a8e00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a419a970 0x7f01a419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a3e00640 1 -- 172.21.15.146:0/4215477836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a419a970 msgr2=0x7f01a419ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a3e00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a419a970 0x7f01a419ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a3e00640 1 -- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01a4109510 msgr2=0x7f01a41a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a3e00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01a4109510 0x7f01a41a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a3e00640 1 -- 172.21.15.146:0/4215477836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a419b490 con 0x7f01a4104440 2024-04-24T08:31:48.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a3e00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4104440 0x7f01a41a0770 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0194035040 tx=0x7f0194033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a1600640 1 -- 172.21.15.146:0/4215477836 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194037040 con 0x7f01a4104440 2024-04-24T08:31:48.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a419b720 con 0x7f01a4104440 2024-04-24T08:31:48.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a1600640 1 -- 172.21.15.146:0/4215477836 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0194040070 con 0x7f01a4104440 2024-04-24T08:31:48.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.488+0000 7f01a1600640 1 -- 172.21.15.146:0/4215477836 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01940039b0 con 0x7f01a4104440 2024-04-24T08:31:48.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.489+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a41a7920 con 0x7f01a4104440 2024-04-24T08:31:48.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.490+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a4069270 con 0x7f01a4104440 2024-04-24T08:31:48.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.492+0000 7f01a1600640 1 -- 172.21.15.146:0/4215477836 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f01940305e0 con 0x7f01a4104440 2024-04-24T08:31:48.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.493+0000 7f01a1600640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f01880a3da0 0x7f01880a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.493+0000 7f01a1600640 1 -- 172.21.15.146:0/4215477836 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f01940f0ca0 con 0x7f01a4104440 2024-04-24T08:31:48.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.494+0000 7f019b400640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f01880a3da0 0x7f01880a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.494+0000 7f019b400640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f01880a3da0 0x7f01880a6260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f01a4103410 tx=0x7f018c009260 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.496+0000 7f01a1600640 1 -- 172.21.15.146:0/4215477836 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01940bd3e0 con 0x7f01a4104440 2024-04-24T08:31:48.740 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.739+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01a419be50 con 0x7f01a4104440 2024-04-24T08:31:48.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.741+0000 7f01a1600640 1 -- 172.21.15.146:0/4215477836 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f01940bcce0 con 0x7f01a4104440 2024-04-24T08:31:48.742 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:31:48.742 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:31:48.743 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:31:48.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f01880a3da0 msgr2=0x7f01880a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f01880a3da0 0x7f01880a6260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f01a4103410 tx=0x7f018c009260 comp rx=0 tx=0).stop 2024-04-24T08:31:48.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4104440 msgr2=0x7f01a41a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4104440 0x7f01a41a0770 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0194035040 tx=0x7f0194033b60 comp rx=0 tx=0).stop 2024-04-24T08:31:48.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 shutdown_connections 2024-04-24T08:31:48.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f01880a3da0 0x7f01880a6260 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.745 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01a419a970 0x7f01a419ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01a4109510 0x7f01a41a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 --2- 172.21.15.146:0/4215477836 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01a4104440 0x7f01a41a0770 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 >> 172.21.15.146:0/4215477836 conn(0x7f01a40ffcc0 msgr2=0x7f01a4102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 shutdown_connections 2024-04-24T08:31:48.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.744+0000 7f01aae00640 1 -- 172.21.15.146:0/4215477836 wait complete. 2024-04-24T08:31:48.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:31:48.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:48 smithi153 ceph-mon[32746]: pgmap v329: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:48.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.965+0000 7f6c6e200640 1 -- 172.21.15.146:0/2903505307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102ab0 msgr2=0x7f6c68106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.965+0000 7f6c6e200640 1 --2- 172.21.15.146:0/2903505307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102ab0 0x7f6c68106af0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009860 tx=0x7f6c5002a340 comp rx=0 tx=0).stop 2024-04-24T08:31:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.965+0000 7f6c6e200640 1 -- 172.21.15.146:0/2903505307 shutdown_connections 2024-04-24T08:31:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.965+0000 7f6c6e200640 1 --2- 172.21.15.146:0/2903505307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102ab0 0x7f6c68106af0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.965+0000 7f6c6e200640 1 --2- 172.21.15.146:0/2903505307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681071d0 0x7f6c6810eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.965+0000 7f6c6e200640 1 --2- 172.21.15.146:0/2903505307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68102100 0x7f6c681024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.965+0000 7f6c6e200640 1 -- 172.21.15.146:0/2903505307 >> 172.21.15.146:0/2903505307 conn(0x7f6c680fd9a0 msgr2=0x7f6c680ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.966+0000 7f6c6e200640 1 -- 172.21.15.146:0/2903505307 shutdown_connections 2024-04-24T08:31:48.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.966+0000 7f6c6e200640 1 -- 172.21.15.146:0/2903505307 wait complete. 2024-04-24T08:31:48.969 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.967+0000 7f6c6e200640 1 Processor -- start 2024-04-24T08:31:48.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.967+0000 7f6c6e200640 1 -- start start 2024-04-24T08:31:48.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.967+0000 7f6c6e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102100 0x7f6c68074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6e200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681071d0 0x7f6c68074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68074d80 0x7f6c681a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6c68112240 con 0x7f6c68074d80 2024-04-24T08:31:48.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6e200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681120c0 con 0x7f6c681071d0 2024-04-24T08:31:48.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681123c0 con 0x7f6c68102100 2024-04-24T08:31:48.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68074d80 0x7f6c681a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102100 0x7f6c68074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68074d80 0x7f6c681a69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55310/0 (socket says 172.21.15.146:55310) 2024-04-24T08:31:48.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 -- 172.21.15.146:0/3605765325 learned_addr learned my addr 172.21.15.146:0/3605765325 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c67e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681071d0 0x7f6c68074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 -- 172.21.15.146:0/3605765325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102100 msgr2=0x7f6c68074300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102100 0x7f6c68074300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.973 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 -- 172.21.15.146:0/3605765325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681071d0 msgr2=0x7f6c68074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681071d0 0x7f6c68074840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c6d800640 1 -- 172.21.15.146:0/3605765325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c681a7070 con 0x7f6c68074d80 2024-04-24T08:31:48.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.968+0000 7f6c67e00640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681071d0 0x7f6c68074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:48.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.969+0000 7f6c6d800640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68074d80 0x7f6c681a69d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000eb10 tx=0x7f6c6000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.969+0000 7f6c65600640 1 -- 172.21.15.146:0/3605765325 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60004b20 con 0x7f6c68074d80 2024-04-24T08:31:48.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.969+0000 7f6c65600640 1 -- 172.21.15.146:0/3605765325 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c60004cc0 con 0x7f6c68074d80 2024-04-24T08:31:48.974 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.969+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c681a7360 con 0x7f6c68074d80 2024-04-24T08:31:48.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.969+0000 7f6c65600640 1 -- 172.21.15.146:0/3605765325 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60010760 con 0x7f6c68074d80 2024-04-24T08:31:48.975 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.969+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c681a7830 con 0x7f6c68074d80 2024-04-24T08:31:48.978 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.972+0000 7f6c65600640 1 -- 172.21.15.146:0/3605765325 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f6c60005020 con 0x7f6c68074d80 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.972+0000 7f6c7256d740 1 Processor -- start 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c7256d740 1 -- start start 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd7b40 0x561b92bd7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd8460 0x561b92be25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92be4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c7256d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561b92b3d1d0 con 0x561b92bd7b40 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c7256d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561b92ae3450 con 0x561b92be2b30 2024-04-24T08:31:48.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c7256d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561b92b39190 con 0x561b92bd8460 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c67e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd7b40 0x561b92bd7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c67e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd7b40 0x561b92bd7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55316/0 (socket says 172.21.15.146:55316) 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c67e00640 1 -- 172.21.15.146:0/4101511984 learned_addr learned my addr 172.21.15.146:0/4101511984 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd8460 0x561b92be25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.973+0000 7f6c67e00640 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92be4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c6ce00640 1 -- 172.21.15.146:0/4101511984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 msgr2=0x561b92be4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92be4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c6ce00640 1 -- 172.21.15.146:0/4101511984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd7b40 msgr2=0x561b92bd7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd7b40 0x561b92bd7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c6ce00640 1 -- 172.21.15.146:0/4101511984 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b92ba0310 con 0x561b92bd8460 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c67e00640 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92be4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd8460 0x561b92be25f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005fd0 tx=0x7f6c58011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd01fdfda78f347d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c65600640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6c4c0a3d20 0x7f6c4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c5e000640 1 -- 172.21.15.146:0/4101511984 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004020 con 0x561b92bd8460 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c5e000640 1 -- 172.21.15.146:0/4101511984 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c580041c0 con 0x561b92bd8460 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.974+0000 7f6c5e000640 1 -- 172.21.15.146:0/4101511984 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58019890 con 0x561b92bd8460 2024-04-24T08:31:48.981 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c65600640 1 -- 172.21.15.146:0/3605765325 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f6c600d0d70 con 0x7f6c68074d80 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c67e00640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6c4c0a3d20 0x7f6c4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c7256d740 1 -- 172.21.15.146:0/4101511984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd8460 msgr2=0x561b92be25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd8460 0x561b92be25f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005fd0 tx=0x7f6c58011810 comp rx=0 tx=0).stop 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c67e00640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6c4c0a3d20 0x7f6c4c0a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x561b92bd4ff0 tx=0x7f6c500058d0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c7256d740 1 -- 172.21.15.146:0/4101511984 shutdown_connections 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd8460 0x561b92be25f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92be4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4101511984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd7b40 0x561b92bd7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.975+0000 7f6c7256d740 1 -- 172.21.15.146:0/4101511984 >> 172.21.15.146:0/4101511984 conn(0x561b92bd3e60 msgr2=0x561b92bd4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.976+0000 7f6c7256d740 1 -- 172.21.15.146:0/4101511984 shutdown_connections 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.976+0000 7f6c7256d740 1 -- 172.21.15.146:0/4101511984 wait complete. 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.976+0000 7f6c7256d740 1 -- start start 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.976+0000 7f6c7256d740 1 Processor -- start 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92be25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd8460 0x561b92bdf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92be2b30 0x561b92bdffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c7256d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x561b92be59c0 con 0x561b92be2b30 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c7256d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x561b92af6810 con 0x561b92bd8460 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c7256d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561b92be5b40 con 0x561b92bd7b40 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c7256d740 10 client.0 _ll_register_callbacks cb 0x561b92ac4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92be2b30 0x561b92bdffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92be2b30 0x561b92bdffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55318/0 (socket says 172.21.15.146:55318) 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 -- 172.21.15.146:0/4159840429 learned_addr learned my addr 172.21.15.146:0/4159840429 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92be25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c67e00640 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd8460 0x561b92bdf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 -- 172.21.15.146:0/4159840429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 msgr2=0x561b92be25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92be25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 -- 172.21.15.146:0/4159840429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd8460 msgr2=0x561b92bdf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd8460 0x561b92bdf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.977+0000 7f6c6d800640 1 -- 172.21.15.146:0/4159840429 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b92b073f0 con 0x561b92be2b30 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c6d800640 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92be2b30 0x561b92bdffb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6c600a53e0 tx=0x7f6c6009c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16ea98e6ac3b1e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c56a00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c56a00640 1 -- 172.21.15.146:0/4159840429 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c600bbe60 con 0x561b92be2b30 2024-04-24T08:31:48.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c56a00640 1 -- 172.21.15.146:0/4159840429 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f6c600a5480 con 0x561b92be2b30 2024-04-24T08:31:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c580041c0 con 0x561b92be2b30 2024-04-24T08:31:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c56a00640 1 -- 172.21.15.146:0/4159840429 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6c600b6200 con 0x561b92be2b30 2024-04-24T08:31:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c7256d740 10 client.17349 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561b9284f3f0 con 0x561b92be2b30 2024-04-24T08:31:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.978+0000 7f6c56a00640 1 -- 172.21.15.146:0/4159840429 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c580041c0 con 0x561b92be2b30 2024-04-24T08:31:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c56a00640 1 -- 172.21.15.146:0/4159840429 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6c600bd070 con 0x561b92be2b30 2024-04-24T08:31:48.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c7256d740 10 client.17349 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c7256d740 10 client.17349 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c7256d740 20 client.17349 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92bf2d10 0x561b92bf5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c7256d740 4 client.17349 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b92b4abf0 con 0x561b92bf2d10 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.979+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92bf2d10 0x561b92bf5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.980+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92bf2d10 0x561b92bf5140 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f14ab43ff9d8a424 server_cookie=d9e0cba30332b618 in_seq=0 out_seq=0 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.980+0000 7f6c56a00640 10 client.17349 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:48.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.980+0000 7f6c56a00640 1 -- 172.21.15.146:0/4159840429 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b92b4abf0 con 0x561b92bf2d10 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.980+0000 7f6c56a00640 10 client.17349 handle_command_reply: tid=0 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.980+0000 7f6c7256d740 1 client.17349 shutdown 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92be2b30 msgr2=0x561b92bdffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92be2b30 0x561b92bdffb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6c600a53e0 tx=0x7f6c6009c6c0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 shutdown_connections 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92bf2d10 0x561b92bf5140 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92be2b30 0x561b92bdffb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.991 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd8460 0x561b92bdf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4159840429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92be25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:48.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c56a00640 0 client.17349 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:48.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 >> 172.21.15.146:0/4159840429 conn(0x561b92bd3e60 msgr2=0x561b92bd9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:48.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 shutdown_connections 2024-04-24T08:31:48.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 1 -- 172.21.15.146:0/4159840429 wait complete. 2024-04-24T08:31:48.992 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.981+0000 7f6c7256d740 20 client.17349 trim_cache size 0 max 16384 2024-04-24T08:31:48.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:48 smithi146 ceph-mon[27805]: pgmap v329: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.995+0000 7f6c7256d740 1 Processor -- start 2024-04-24T08:31:48.997 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c7256d740 1 -- start start 2024-04-24T08:31:48.998 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd7b40 0x561b92be4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92be2b30 0x561b92bd4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:48.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c7256d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681b26b0 con 0x561b92bd8460 2024-04-24T08:31:48.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c7256d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681aff80 con 0x561b92bd7b40 2024-04-24T08:31:48.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c7256d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681addf0 con 0x561b92be2b30 2024-04-24T08:31:48.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c6ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd7b40 0x561b92be4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:48.999 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c6d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92be2b30 0x561b92bd4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c6ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd7b40 0x561b92be4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:55718/0 (socket says 172.21.15.146:55718) 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c6ce00640 1 -- 172.21.15.146:0/3365550070 learned_addr learned my addr 172.21.15.146:0/3365550070 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.996+0000 7f6c67e00640 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c6d800640 1 -- 172.21.15.146:0/3365550070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd7b40 msgr2=0x561b92be4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c6d800640 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd7b40 0x561b92be4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c6d800640 1 -- 172.21.15.146:0/3365550070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 msgr2=0x561b92bee8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c6d800640 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bee8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c6d800640 1 -- 172.21.15.146:0/3365550070 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b929197d0 con 0x561b92be2b30 2024-04-24T08:31:49.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd7b40 0x561b92be4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:49.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c6d800640 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92be2b30 0x561b92bd4240 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c60099670 tx=0x7f6c6009c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c123f0fbb8b7b1cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c5d600640 1 -- 172.21.15.146:0/3365550070 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c600bbe60 con 0x561b92be2b30 2024-04-24T08:31:49.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c5d600640 1 -- 172.21.15.146:0/3365550070 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c600a1050 con 0x561b92be2b30 2024-04-24T08:31:49.001 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.997+0000 7f6c5d600640 1 -- 172.21.15.146:0/3365550070 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c600a5480 con 0x561b92be2b30 2024-04-24T08:31:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 -- 172.21.15.146:0/3365550070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92be2b30 msgr2=0x561b92bd4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.005 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92be2b30 0x561b92bd4240 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c60099670 tx=0x7f6c6009c730 comp rx=0 tx=0).stop 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 -- 172.21.15.146:0/3365550070 shutdown_connections 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92be2b30 0x561b92bd4240 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 --2- 172.21.15.146:0/3365550070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92bd7b40 0x561b92be4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 -- 172.21.15.146:0/3365550070 >> 172.21.15.146:0/3365550070 conn(0x561b92bd3e60 msgr2=0x561b92bdf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 -- 172.21.15.146:0/3365550070 shutdown_connections 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 -- 172.21.15.146:0/3365550070 wait complete. 2024-04-24T08:31:49.006 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.998+0000 7f6c7256d740 1 -- start start 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 1 Processor -- start 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92bdc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bdca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92bdcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681ae440 con 0x561b92bd8460 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681af8b0 con 0x561b92be2b30 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c681aeb10 con 0x561b92bd7b40 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c7256d740 10 client.0 _ll_register_callbacks cb 0x7f6c6009cae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:49.007 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c6ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92bdc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c6ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92bdc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33190/0 (socket says 172.21.15.146:33190) 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c67e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bdca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c6ce00640 1 -- 172.21.15.146:0/4211267090 learned_addr learned my addr 172.21.15.146:0/4211267090 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:48.999+0000 7f6c6d800640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92bdcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c6ce00640 1 -- 172.21.15.146:0/4211267090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 msgr2=0x561b92bdcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92bdcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c6ce00640 1 -- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 msgr2=0x561b92bdca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.008 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bdca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c6ce00640 1 -- 172.21.15.146:0/4211267090 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b92be8f00 con 0x561b92bd7b40 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c67e00640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bdca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c6d800640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92bdcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c6ce00640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92bdc520 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005fd0 tx=0x7f6c58003f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68ca777865aae611 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c56a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c56a00640 1 -- 172.21.15.146:0/4211267090 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5802bb50 con 0x561b92bd7b40 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c56a00640 1 -- 172.21.15.146:0/4211267090 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f6c5802bcf0 con 0x561b92bd7b40 2024-04-24T08:31:49.009 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.000+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c600a1050 con 0x561b92bd7b40 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c56a00640 1 -- 172.21.15.146:0/4211267090 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6c580040c0 con 0x561b92bd7b40 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 10 client.27508 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561b92c594e0 con 0x561b92bd7b40 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c56a00640 1 -- 172.21.15.146:0/4211267090 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c600a1050 con 0x561b92bd7b40 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c56a00640 1 -- 172.21.15.146:0/4211267090 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6c58005200 con 0x561b92bd7b40 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 10 client.27508 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 10 client.27508 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-04-24T08:31:49.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 20 client.27508 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92c59a40 0x561b92c5be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 4 client.27508 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 --> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b92b44a60 con 0x561b92c59a40 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.001+0000 7f6c67e00640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92c59a40 0x561b92c5be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c67e00640 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92c59a40 0x561b92c5be60 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=33355e951cd44de5 server_cookie=b21abfbcabdba5de in_seq=0 out_seq=0 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c56a00640 10 client.27508 ms_handle_connect on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c56a00640 1 -- 172.21.15.146:0/4211267090 <== mds.2 v2:172.21.15.146:6838/2207966302 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b92b44a60 con 0x561b92c59a40 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c56a00640 10 client.27508 handle_command_reply: tid=0 2024-04-24T08:31:49.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 client.27508 shutdown 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 msgr2=0x561b92bdc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92bdc520 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6c58005fd0 tx=0x7f6c58003f20 comp rx=0 tx=0).stop 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 shutdown_connections 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:6838/2207966302,v1:172.21.15.146:6839/2207966302] conn(0x561b92c59a40 0x561b92c5be60 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x561b92be2b30 0x561b92bdcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561b92bd8460 0x561b92bdca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c7256d740 1 --2- 172.21.15.146:0/4211267090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b92bd7b40 0x561b92bdc520 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.002+0000 7f6c56a00640 0 client.27508 ms_handle_reset on v2:172.21.15.146:6838/2207966302 2024-04-24T08:31:49.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 >> 172.21.15.146:0/4211267090 conn(0x561b92bd3e60 msgr2=0x561b92bd4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 shutdown_connections 2024-04-24T08:31:49.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c7256d740 1 -- 172.21.15.146:0/4211267090 wait complete. 2024-04-24T08:31:49.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c7256d740 20 client.27508 trim_cache size 0 max 16384 2024-04-24T08:31:49.013 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:31:49.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6c4c0a3d20 msgr2=0x7f6c4c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c6e200640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6c4c0a3d20 0x7f6c4c0a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x561b92bd4ff0 tx=0x7f6c500058d0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68074d80 msgr2=0x7f6c681a69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.003+0000 7f6c6e200640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68074d80 0x7f6c681a69d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000eb10 tx=0x7f6c6000c2d0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 shutdown_connections 2024-04-24T08:31:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6c4c0a3d20 0x7f6c4c0a61e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c68074d80 0x7f6c681a69d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c681071d0 0x7f6c68074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 --2- 172.21.15.146:0/3605765325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c68102100 0x7f6c68074300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 >> 172.21.15.146:0/3605765325 conn(0x7f6c680fd9a0 msgr2=0x7f6c680fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 shutdown_connections 2024-04-24T08:31:49.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.004+0000 7f6c6e200640 1 -- 172.21.15.146:0/3605765325 wait complete. 2024-04-24T08:31:49.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:31:49.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:48 smithi177 ceph-mon[33046]: pgmap v329: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:49.224 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 -- 172.21.15.146:0/1140037979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde141071e0 msgr2=0x7fde1410edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 --2- 172.21.15.146:0/1140037979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde141071e0 0x7fde1410edb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fde0c009ca0 tx=0x7fde0c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 -- 172.21.15.146:0/1140037979 shutdown_connections 2024-04-24T08:31:49.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 --2- 172.21.15.146:0/1140037979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde141071e0 0x7fde1410edb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 --2- 172.21.15.146:0/1140037979 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde14102ac0 0x7fde14106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 --2- 172.21.15.146:0/1140037979 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde14102110 0x7fde141024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.225 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 -- 172.21.15.146:0/1140037979 >> 172.21.15.146:0/1140037979 conn(0x7fde140fd9d0 msgr2=0x7fde140ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.222+0000 7fde13200640 1 -- 172.21.15.146:0/1140037979 shutdown_connections 2024-04-24T08:31:49.226 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.223+0000 7fde13200640 1 -- 172.21.15.146:0/1140037979 wait complete. 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.223+0000 7fde13200640 1 Processor -- start 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde13200640 1 -- start start 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde13200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14102110 0x7fde141a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde13200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde141071e0 0x7fde141a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde13200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde1419a740 0x7fde1419abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde13200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fde141122f0 con 0x7fde141071e0 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde13200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde14112170 con 0x7fde1419a740 2024-04-24T08:31:49.228 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde13200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fde14112470 con 0x7fde14102110 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14102110 0x7fde141a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14102110 0x7fde141a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33214/0 (socket says 172.21.15.146:33214) 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde11e00640 1 -- 172.21.15.146:0/3331279963 learned_addr learned my addr 172.21.15.146:0/3331279963 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde09400640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde141071e0 0x7fde141a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.224+0000 7fde12800640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde1419a740 0x7fde1419abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde11e00640 1 -- 172.21.15.146:0/3331279963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde1419a740 msgr2=0x7fde1419abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde11e00640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde1419a740 0x7fde1419abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.229 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde11e00640 1 -- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde141071e0 msgr2=0x7fde141a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde11e00640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde141071e0 0x7fde141a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde11e00640 1 -- 172.21.15.146:0/3331279963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1419b340 con 0x7fde14102110 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde12800640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde1419a740 0x7fde1419abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde11e00640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14102110 0x7fde141a0680 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fde0400e9f0 tx=0x7fde0400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde0b400640 1 -- 172.21.15.146:0/3331279963 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0400cd80 con 0x7fde14102110 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde0b400640 1 -- 172.21.15.146:0/3331279963 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde04010910 con 0x7fde14102110 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde141a7420 con 0x7fde14102110 2024-04-24T08:31:49.230 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde0b400640 1 -- 172.21.15.146:0/3331279963 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04018990 con 0x7fde14102110 2024-04-24T08:31:49.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.225+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde141a7820 con 0x7fde14102110 2024-04-24T08:31:49.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.228+0000 7fde1b80e740 1 Processor -- start 2024-04-24T08:31:49.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.228+0000 7fde1b80e740 1 -- start start 2024-04-24T08:31:49.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde0b400640 1 -- 172.21.15.146:0/3331279963 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fde04010ac0 con 0x7fde14102110 2024-04-24T08:31:49.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22ebf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde1b80e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5592c22511d0 con 0x5592c22f6b30 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde1b80e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5592c21f7450 con 0x5592c22ebb40 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde1b80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5592c224d190 con 0x5592c22ec460 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde12800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22ebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde09400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.236 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33224/0 (socket says 172.21.15.146:33224) 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde12800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22ebf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:55750/0 (socket says 172.21.15.146:55750) 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.229+0000 7fde11e00640 1 -- 172.21.15.146:0/3825068465 learned_addr learned my addr 172.21.15.146:0/3825068465 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde09400640 1 -- 172.21.15.146:0/3825068465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 msgr2=0x5592c22f65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde09400640 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde09400640 1 -- 172.21.15.146:0/3825068465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 msgr2=0x5592c22ebf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde09400640 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22ebf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde09400640 1 -- 172.21.15.146:0/3825068465 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592c22b4310 con 0x5592c22f6b30 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde11e00640 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:49.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde12800640 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22ebf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:49.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde09400640 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8f20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fde1406b560 tx=0x7fddfc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7336a1614261491c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde03000640 1 -- 172.21.15.146:0/3825068465 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc004050 con 0x5592c22f6b30 2024-04-24T08:31:49.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde03000640 1 -- 172.21.15.146:0/3825068465 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddfc005500 con 0x5592c22f6b30 2024-04-24T08:31:49.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.230+0000 7fde03000640 1 -- 172.21.15.146:0/3825068465 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc004d90 con 0x5592c22f6b30 2024-04-24T08:31:49.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde0b400640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fddf80a3d20 0x7fddf80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde0b400640 1 -- 172.21.15.146:0/3331279963 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fde040d0be0 con 0x7fde14102110 2024-04-24T08:31:49.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 -- 172.21.15.146:0/3825068465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 msgr2=0x5592c22f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8f20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fde1406b560 tx=0x7fddfc011810 comp rx=0 tx=0).stop 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde12800640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fddf80a3d20 0x7fddf80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 -- 172.21.15.146:0/3825068465 shutdown_connections 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8f20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3825068465 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22ebf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 -- 172.21.15.146:0/3825068465 >> 172.21.15.146:0/3825068465 conn(0x5592c22e7e60 msgr2=0x5592c22e8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 -- 172.21.15.146:0/3825068465 shutdown_connections 2024-04-24T08:31:49.239 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.231+0000 7fde1b80e740 1 -- 172.21.15.146:0/3825068465 wait complete. 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 -- start start 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde12800640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fddf80a3d20 0x7fddf80a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fde0c009430 tx=0x7fde0c004a40 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 Processor -- start 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5592c22f99c0 con 0x5592c22f6b30 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5592c220a810 con 0x5592c22ebb40 2024-04-24T08:31:49.240 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5592c22f9b40 con 0x5592c22ec460 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.232+0000 7fde1b80e740 10 client.0 _ll_register_callbacks cb 0x5592c21d8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33228/0 (socket says 172.21.15.146:33228) 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde12800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 -- 172.21.15.146:0/3918515616 learned_addr learned my addr 172.21.15.146:0/3918515616 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde11e00640 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 -- 172.21.15.146:0/3918515616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 msgr2=0x5592c22f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.241 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 -- 172.21.15.146:0/3918515616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 msgr2=0x5592c22f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 -- 172.21.15.146:0/3918515616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592c221b3f0 con 0x5592c22ec460 2024-04-24T08:31:49.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fde09400640 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f3870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009380 tx=0x7fddfc009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19af5ce9a93cf5a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.233+0000 7fddf7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:31:49.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fddf7400640 1 -- 172.21.15.146:0/3918515616 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc029070 con 0x5592c22ec460 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fddf7400640 1 -- 172.21.15.146:0/3918515616 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fddfc004f20 con 0x5592c22ec460 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddfc005500 con 0x5592c22ec460 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fddf7400640 1 -- 172.21.15.146:0/3918515616 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fddfc035050 con 0x5592c22ec460 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fde1b80e740 10 client.27520 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5592c1f633f0 con 0x5592c22ec460 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fddf7400640 1 -- 172.21.15.146:0/3918515616 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddfc005500 con 0x5592c22ec460 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.234+0000 7fddf7400640 1 -- 172.21.15.146:0/3918515616 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fddfc003c90 con 0x5592c22ec460 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde1b80e740 10 client.27520 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:49.245 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde1b80e740 10 client.27520 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:31:49.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde1b80e740 20 client.27520 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:49.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c2306c20 0x5592c2309010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde1b80e740 4 client.27520 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:49.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592c225ebf0 con 0x5592c2306c20 2024-04-24T08:31:49.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde11e00640 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c2306c20 0x5592c2309010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fddf7400640 10 client.27520 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:49.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.235+0000 7fde11e00640 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c2306c20 0x5592c2309010 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0c6b4b1e7ea6609 server_cookie=3ef290a8d2250fb8 in_seq=0 out_seq=0 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fddf7400640 1 -- 172.21.15.146:0/3918515616 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592c225ebf0 con 0x5592c2306c20 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fddf7400640 10 client.27520 handle_command_reply: tid=0 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 client.27520 shutdown 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 msgr2=0x5592c22f3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f3870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009380 tx=0x7fddfc009f80 comp rx=0 tx=0).stop 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 shutdown_connections 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c2306c20 0x5592c2309010 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f3870 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 --2- 172.21.15.146:0/3918515616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 >> 172.21.15.146:0/3918515616 conn(0x5592c22e7e60 msgr2=0x5592c22edc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fddf7400640 0 client.27520 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:49.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.236+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 shutdown_connections 2024-04-24T08:31:49.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.237+0000 7fde1b80e740 1 -- 172.21.15.146:0/3918515616 wait complete. 2024-04-24T08:31:49.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.237+0000 7fde1b80e740 20 client.27520 trim_cache size 0 max 16384 2024-04-24T08:31:49.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.250+0000 7fde1b80e740 1 Processor -- start 2024-04-24T08:31:49.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde1b80e740 1 -- start start 2024-04-24T08:31:49.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22ebb40 0x5592c23028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22f6b30 0x5592c22f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde1b80e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fde141b2670 con 0x5592c22ebb40 2024-04-24T08:31:49.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde1b80e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde141aff40 con 0x5592c22f6b30 2024-04-24T08:31:49.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde1b80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fde141addb0 con 0x5592c22ec460 2024-04-24T08:31:49.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde11e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22ebb40 0x5592c23028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde11e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22ebb40 0x5592c23028b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55374/0 (socket says 172.21.15.146:55374) 2024-04-24T08:31:49.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde11e00640 1 -- 172.21.15.146:0/520315005 learned_addr learned my addr 172.21.15.146:0/520315005 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde09400640 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.251+0000 7fde12800640 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22f6b30 0x5592c22f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde11e00640 1 -- 172.21.15.146:0/520315005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 msgr2=0x5592c22f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde11e00640 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde11e00640 1 -- 172.21.15.146:0/520315005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22f6b30 msgr2=0x5592c22f65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde11e00640 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22f6b30 0x5592c22f65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde11e00640 1 -- 172.21.15.146:0/520315005 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592c202d7d0 con 0x5592c22ebb40 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde11e00640 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22ebb40 0x5592c23028b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fde04099540 tx=0x7fde040d30c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9d61e5389a4110 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde02600640 1 -- 172.21.15.146:0/520315005 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0409d270 con 0x5592c22ebb40 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde02600640 1 -- 172.21.15.146:0/520315005 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0409cb70 con 0x5592c22ebb40 2024-04-24T08:31:49.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.252+0000 7fde02600640 1 -- 172.21.15.146:0/520315005 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040991e0 con 0x5592c22ebb40 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 -- 172.21.15.146:0/520315005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22ebb40 msgr2=0x5592c23028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22ebb40 0x5592c23028b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fde04099540 tx=0x7fde040d30c0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 -- 172.21.15.146:0/520315005 shutdown_connections 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22f6b30 0x5592c22f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 --2- 172.21.15.146:0/520315005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22ebb40 0x5592c23028b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 -- 172.21.15.146:0/520315005 >> 172.21.15.146:0/520315005 conn(0x5592c22e7e60 msgr2=0x5592c22e8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 -- 172.21.15.146:0/520315005 shutdown_connections 2024-04-24T08:31:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 -- 172.21.15.146:0/520315005 wait complete. 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 -- start start 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.253+0000 7fde1b80e740 1 Processor -- start 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde1b80e740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde1b80e740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fde141ae400 con 0x5592c22f6b30 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde1b80e740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fde141af870 con 0x5592c22ebb40 2024-04-24T08:31:49.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde1b80e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fde141aead0 con 0x5592c22ec460 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde1b80e740 10 client.0 _ll_register_callbacks cb 0x7fde04005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde12800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33240/0 (socket says 172.21.15.146:33240) 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 -- 172.21.15.146:0/1040938334 learned_addr learned my addr 172.21.15.146:0/1040938334 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde11e00640 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 -- 172.21.15.146:0/1040938334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 msgr2=0x5592c22f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 -- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 msgr2=0x5592c22f8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde09400640 1 -- 172.21.15.146:0/1040938334 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592c22fd200 con 0x5592c22ec460 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.254+0000 7fde12800640 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fde09400640 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f7f00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fddfc011850 tx=0x7fddfc009690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5fe7dadd5e3262b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fddf7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fddf7400640 1 -- 172.21.15.146:0/1040938334 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc029040 con 0x5592c22ec460 2024-04-24T08:31:49.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde0409cb70 con 0x5592c22ec460 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fddf7400640 1 -- 172.21.15.146:0/1040938334 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fddfc004410 con 0x5592c22ec460 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fddf7400640 1 -- 172.21.15.146:0/1040938334 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fddfc041050 con 0x5592c22ec460 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fde1b80e740 10 client.27526 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.255+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5592c236d450 con 0x5592c22ec460 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fddf7400640 1 -- 172.21.15.146:0/1040938334 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0409cb70 con 0x5592c22ec460 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fddf7400640 1 -- 172.21.15.146:0/1040938334 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fddfc003ae0 con 0x5592c22ec460 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fde1b80e740 10 client.27526 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:49.268 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fde1b80e740 10 client.27526 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fde1b80e740 20 client.27526 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fde1b80e740 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c236db70 0x5592c236ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fde1b80e740 4 client.27526 mds_command: new command op to 24568 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 --> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5592c2258a60 con 0x5592c236db70 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.256+0000 7fde11e00640 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c236db70 0x5592c236ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde11e00640 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c236db70 0x5592c236ff90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c7b0e82d3ff58154 server_cookie=6de51861ffb4eb4c in_seq=0 out_seq=0 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fddf7400640 10 client.27526 ms_handle_connect on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fddf7400640 1 -- 172.21.15.146:0/1040938334 <== mds.1 v2:172.21.15.146:6840/1590374458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5592c2258a60 con 0x5592c236db70 2024-04-24T08:31:49.269 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fddf7400640 10 client.27526 handle_command_reply: tid=0 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 client.27526 shutdown 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 msgr2=0x5592c22f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f7f00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fddfc011850 tx=0x7fddfc009690 comp rx=0 tx=0).stop 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 shutdown_connections 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:6840/1590374458,v1:172.21.15.146:6841/1590374458] conn(0x5592c236db70 0x5592c236ff90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5592c22ec460 0x5592c22f7f00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592c22f6b30 0x5592c22f8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.270 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 --2- 172.21.15.146:0/1040938334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5592c22ebb40 0x5592c22f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 >> 172.21.15.146:0/1040938334 conn(0x5592c22e7e60 msgr2=0x5592c22e66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fddf7400640 0 client.27526 ms_handle_reset on v2:172.21.15.146:6840/1590374458 2024-04-24T08:31:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 shutdown_connections 2024-04-24T08:31:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 1 -- 172.21.15.146:0/1040938334 wait complete. 2024-04-24T08:31:49.271 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.257+0000 7fde1b80e740 20 client.27526 trim_cache size 0 max 16384 2024-04-24T08:31:49.271 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:31:49.272 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fddf80a3d20 msgr2=0x7fddf80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fddf80a3d20 0x7fddf80a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fde0c009430 tx=0x7fde0c004a40 comp rx=0 tx=0).stop 2024-04-24T08:31:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14102110 msgr2=0x7fde141a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14102110 0x7fde141a0680 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fde0400e9f0 tx=0x7fde0400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 shutdown_connections 2024-04-24T08:31:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fddf80a3d20 0x7fddf80a61e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fde1419a740 0x7fde1419abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.273 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde141071e0 0x7fde141a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 --2- 172.21.15.146:0/3331279963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde14102110 0x7fde141a0680 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 >> 172.21.15.146:0/3331279963 conn(0x7fde140fd9d0 msgr2=0x7fde140fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 shutdown_connections 2024-04-24T08:31:49.274 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.258+0000 7fde13200640 1 -- 172.21.15.146:0/3331279963 wait complete. 2024-04-24T08:31:49.277 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:31:49.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 -- 172.21.15.146:0/1844441205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e8108610 msgr2=0x7f81e8108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 --2- 172.21.15.146:0/1844441205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e8108610 0x7f81e8108a90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009ca0 tx=0x7f81d002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.470 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 -- 172.21.15.146:0/1844441205 shutdown_connections 2024-04-24T08:31:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 --2- 172.21.15.146:0/1844441205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81e8108fd0 0x7f81e8115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 --2- 172.21.15.146:0/1844441205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e8108610 0x7f81e8108a90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 --2- 172.21.15.146:0/1844441205 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e810e660 0x7f81e810ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 -- 172.21.15.146:0/1844441205 >> 172.21.15.146:0/1844441205 conn(0x7f81e8076540 msgr2=0x7f81e8078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 -- 172.21.15.146:0/1844441205 shutdown_connections 2024-04-24T08:31:49.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.468+0000 7f81ed600640 1 -- 172.21.15.146:0/1844441205 wait complete. 2024-04-24T08:31:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.469+0000 7f81ed600640 1 Processor -- start 2024-04-24T08:31:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.469+0000 7f81ed600640 1 -- start start 2024-04-24T08:31:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81ed600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e8108610 0x7f81e819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81ed600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81e8108fd0 0x7f81e819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81ed600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e810e660 0x7f81e81a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81ed600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81189a0 con 0x7f81e8108fd0 2024-04-24T08:31:49.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81ed600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8118820 con 0x7f81e810e660 2024-04-24T08:31:49.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81ed600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81e8118b20 con 0x7f81e8108610 2024-04-24T08:31:49.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e8108610 0x7f81e819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81e6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81e8108fd0 0x7f81e819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81e6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81e8108fd0 0x7f81e819f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55396/0 (socket says 172.21.15.146:55396) 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e8108610 0x7f81e819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33264/0 (socket says 172.21.15.146:33264) 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81e6000640 1 -- 172.21.15.146:0/945297390 learned_addr learned my addr 172.21.15.146:0/945297390 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.470+0000 7f81e7400640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e810e660 0x7f81e81a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81e6a00640 1 -- 172.21.15.146:0/945297390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e810e660 msgr2=0x7f81e81a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81e6a00640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e810e660 0x7f81e81a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81e6a00640 1 -- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81e8108fd0 msgr2=0x7f81e819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81e6a00640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81e8108fd0 0x7f81e819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81e6a00640 1 -- 172.21.15.146:0/945297390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e81a3ee0 con 0x7f81e8108610 2024-04-24T08:31:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81e6a00640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e8108610 0x7f81e819ef40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f81d800e9c0 tx=0x7f81d800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81df400640 1 -- 172.21.15.146:0/945297390 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d800ccc0 con 0x7f81e8108610 2024-04-24T08:31:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81df400640 1 -- 172.21.15.146:0/945297390 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d80035f0 con 0x7f81e8108610 2024-04-24T08:31:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81df400640 1 -- 172.21.15.146:0/945297390 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d800ce60 con 0x7f81e8108610 2024-04-24T08:31:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e81a41d0 con 0x7f81e8108610 2024-04-24T08:31:49.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.471+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e81aba60 con 0x7f81e8108610 2024-04-24T08:31:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 Processor -- start 2024-04-24T08:31:49.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 -- start start 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81df400640 1 -- 172.21.15.146:0/945297390 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f81d80037a0 con 0x7f81e8108610 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130bad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5643130201d0 con 0x5643130ba950 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564312fc6450 con 0x5643130c5940 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81f0406740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56431301c190 con 0x5643130bb270 2024-04-24T08:31:49.480 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81e7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130bad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:33278/0 (socket says 172.21.15.146:33278) 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81e6a00640 1 -- 172.21.15.146:0/1917315264 learned_addr learned my addr 172.21.15.146:0/1917315264 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81e7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130bad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55412/0 (socket says 172.21.15.146:55412) 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.475+0000 7f81e6000640 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81e6a00640 1 -- 172.21.15.146:0/1917315264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 msgr2=0x5643130c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81e6a00640 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81e6a00640 1 -- 172.21.15.146:0/1917315264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 msgr2=0x5643130bad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.481 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81e6a00640 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130bad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81e6a00640 1 -- 172.21.15.146:0/1917315264 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564313083310 con 0x5643130bb270 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81e6a00640 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c5400 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f81d80ce170 tx=0x7f81d80d4bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4a69afed97bbf25 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81d7400640 1 -- 172.21.15.146:0/1917315264 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d80d9c30 con 0x5643130bb270 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81d7400640 1 -- 172.21.15.146:0/1917315264 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d80e2070 con 0x5643130bb270 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.476+0000 7f81d7400640 1 -- 172.21.15.146:0/1917315264 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d80dc3e0 con 0x5643130bb270 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81df400640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f81cc0a3d20 0x7f81cc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81df400640 1 -- 172.21.15.146:0/945297390 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f81d80d0b50 con 0x7f81e8108610 2024-04-24T08:31:49.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81e7400640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f81cc0a3d20 0x7f81cc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 -- 172.21.15.146:0/1917315264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 msgr2=0x5643130c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c5400 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f81d80ce170 tx=0x7f81d80d4bf0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 -- 172.21.15.146:0/1917315264 shutdown_connections 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c5400 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 --2- 172.21.15.146:0/1917315264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130bad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 -- 172.21.15.146:0/1917315264 >> 172.21.15.146:0/1917315264 conn(0x5643130b6c70 msgr2=0x5643130b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 -- 172.21.15.146:0/1917315264 shutdown_connections 2024-04-24T08:31:49.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.477+0000 7f81f0406740 1 -- 172.21.15.146:0/1917315264 wait complete. 2024-04-24T08:31:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81e7400640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f81cc0a3d20 0x7f81cc0a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009ce0 tx=0x7f81e0005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 -- start start 2024-04-24T08:31:49.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 Processor -- start 2024-04-24T08:31:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130bb270 0x5643130c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130c5940 0x5643130c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x5643130c8770 con 0x5643130ba950 2024-04-24T08:31:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x564312fd9810 con 0x5643130bb270 2024-04-24T08:31:49.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.478+0000 7f81f0406740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5643130c88f0 con 0x5643130c5940 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81f0406740 10 client.0 _ll_register_callbacks cb 0x564312fa7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130bb270 0x5643130c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130bb270 0x5643130c26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:55804/0 (socket says 172.21.15.146:55804) 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 -- 172.21.15.146:0/1423710611 learned_addr learned my addr 172.21.15.146:0/1423710611 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e7400640 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130c5940 0x5643130c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 -- 172.21.15.146:0/1423710611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130c5940 msgr2=0x5643130c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130c5940 0x5643130c2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 -- 172.21.15.146:0/1423710611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 msgr2=0x5643130c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 -- 172.21.15.146:0/1423710611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564312fea3f0 con 0x5643130bb270 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81e6000640 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130bb270 0x5643130c26e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f81d002ade0 tx=0x7f81d0003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc40b76c4894fd47 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.479+0000 7f81cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.480+0000 7f81cbe00640 1 -- 172.21.15.146:0/1423710611 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0003bf0 con 0x5643130bb270 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.480+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81d80e2070 con 0x5643130bb270 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.480+0000 7f81cbe00640 1 -- 172.21.15.146:0/1423710611 <== mon.2 v2:172.21.15.153:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f81d0003d90 con 0x5643130bb270 2024-04-24T08:31:49.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.480+0000 7f81cbe00640 1 -- 172.21.15.146:0/1423710611 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f81d0004b40 con 0x5643130bb270 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.480+0000 7f81f0406740 10 client.27533 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.480+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564312d323f0 con 0x5643130bb270 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.480+0000 7f81cbe00640 1 -- 172.21.15.146:0/1423710611 <== mon.2 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d80e2070 con 0x5643130bb270 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81cbe00640 1 -- 172.21.15.146:0/1423710611 <== mon.2 v2:172.21.15.153:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f81d0004520 con 0x5643130bb270 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81f0406740 10 client.27533 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81f0406740 10 client.27533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81f0406740 20 client.27533 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:49.492 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81f0406740 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5643130d5ab0 0x5643130d7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81f0406740 4 client.27533 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56431302dbf0 con 0x5643130d5ab0 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.481+0000 7f81e6a00640 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5643130d5ab0 0x5643130d7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.482+0000 7f81e6a00640 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5643130d5ab0 0x5643130d7ee0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e457e14af9d4ce7 server_cookie=16f705040af8d85d in_seq=0 out_seq=0 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.482+0000 7f81cbe00640 10 client.27533 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.482+0000 7f81cbe00640 1 -- 172.21.15.146:0/1423710611 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56431302dbf0 con 0x5643130d5ab0 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.482+0000 7f81cbe00640 10 client.27533 handle_command_reply: tid=0 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.482+0000 7f81f0406740 1 client.27533 shutdown 2024-04-24T08:31:49.493 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130bb270 msgr2=0x5643130c26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130bb270 0x5643130c26e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f81d002ade0 tx=0x7f81d0003970 comp rx=0 tx=0).stop 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 shutdown_connections 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x5643130d5ab0 0x5643130d7ee0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130c5940 0x5643130c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130bb270 0x5643130c26e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 --2- 172.21.15.146:0/1423710611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 >> 172.21.15.146:0/1423710611 conn(0x5643130b6c70 msgr2=0x5643130bca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.494 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81cbe00640 0 client.27533 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:49.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 shutdown_connections 2024-04-24T08:31:49.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 1 -- 172.21.15.146:0/1423710611 wait complete. 2024-04-24T08:31:49.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.483+0000 7f81f0406740 20 client.27533 trim_cache size 0 max 16384 2024-04-24T08:31:49.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.497+0000 7f81f0406740 1 Processor -- start 2024-04-24T08:31:49.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.497+0000 7f81f0406740 1 -- start start 2024-04-24T08:31:49.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81f0406740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81f0406740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81f0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130dd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81f0406740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81b6c50 con 0x5643130ba950 2024-04-24T08:31:49.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81f0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81b4520 con 0x5643130c5940 2024-04-24T08:31:49.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81f0406740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81b2390 con 0x5643130bb270 2024-04-24T08:31:49.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130dd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81e6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55434/0 (socket says 172.21.15.146:55434) 2024-04-24T08:31:49.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.498+0000 7f81e6a00640 1 -- 172.21.15.146:0/1439515522 learned_addr learned my addr 172.21.15.146:0/1439515522 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81e6000640 1 -- 172.21.15.146:0/1439515522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 msgr2=0x5643130dd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81e6000640 1 --2- 172.21.15.146:0/1439515522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130dd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81e6000640 1 -- 172.21.15.146:0/1439515522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 msgr2=0x5643130d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81e6000640 1 --2- 172.21.15.146:0/1439515522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81e6000640 1 -- 172.21.15.146:0/1439515522 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643130c3190 con 0x5643130bb270 2024-04-24T08:31:49.509 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81e6000640 1 --2- 172.21.15.146:0/1439515522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c4f20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009bf0 tx=0x7f81d0033000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55a8eb49333bfe36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81d6a00640 1 -- 172.21.15.146:0/1439515522 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0030b10 con 0x5643130bb270 2024-04-24T08:31:49.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81d6a00640 1 -- 172.21.15.146:0/1439515522 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d0036050 con 0x5643130bb270 2024-04-24T08:31:49.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.499+0000 7f81d6a00640 1 -- 172.21.15.146:0/1439515522 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d002f900 con 0x5643130bb270 2024-04-24T08:31:49.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 -- 172.21.15.146:0/1439515522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 msgr2=0x5643130c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 --2- 172.21.15.146:0/1439515522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c4f20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009bf0 tx=0x7f81d0033000 comp rx=0 tx=0).stop 2024-04-24T08:31:49.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 -- 172.21.15.146:0/1439515522 shutdown_connections 2024-04-24T08:31:49.510 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 --2- 172.21.15.146:0/1439515522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130c4f20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 --2- 172.21.15.146:0/1439515522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130dd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 --2- 172.21.15.146:0/1439515522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 -- 172.21.15.146:0/1439515522 >> 172.21.15.146:0/1439515522 conn(0x5643130b6c70 msgr2=0x5643130b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 -- 172.21.15.146:0/1439515522 shutdown_connections 2024-04-24T08:31:49.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 -- 172.21.15.146:0/1439515522 wait complete. 2024-04-24T08:31:49.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.500+0000 7f81f0406740 1 -- start start 2024-04-24T08:31:49.511 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 1 Processor -- start 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81b29e0 con 0x5643130ba950 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81b3e50 con 0x5643130c5940 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81e81b30b0 con 0x5643130bb270 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81f0406740 10 client.0 _ll_register_callbacks cb 0x7f81d0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:31:49.512 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81e6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55440/0 (socket says 172.21.15.146:55440) 2024-04-24T08:31:49.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81e6a00640 1 -- 172.21.15.146:0/2779597177 learned_addr learned my addr 172.21.15.146:0/2779597177 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:49.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.501+0000 7f81e7400640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6000640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6a00640 1 -- 172.21.15.146:0/2779597177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 msgr2=0x5643130ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6a00640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.513 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6a00640 1 -- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 msgr2=0x5643130c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:31:49.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6a00640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6a00640 1 -- 172.21.15.146:0/2779597177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643130bb0a0 con 0x5643130ba950 2024-04-24T08:31:49.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6000640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:49.514 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81e6a00640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130cdb50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d8003e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c0d60fa5497a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:49.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.146:3300/0 2024-04-24T08:31:49.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81cbe00640 1 -- 172.21.15.146:0/2779597177 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8098d20 con 0x5643130ba950 2024-04-24T08:31:49.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81cbe00640 1 -- 172.21.15.146:0/2779597177 <== mon.0 v2:172.21.15.146:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f81d80d4670 con 0x5643130ba950 2024-04-24T08:31:49.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81d0036050 con 0x5643130ba950 2024-04-24T08:31:49.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81cbe00640 1 -- 172.21.15.146:0/2779597177 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f81d809c3e0 con 0x5643130ba950 2024-04-24T08:31:49.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.502+0000 7f81f0406740 10 client.17391 fetch_fsmap learned FSMap version 26 2024-04-24T08:31:49.515 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564312dfc7d0 con 0x5643130ba950 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81cbe00640 1 -- 172.21.15.146:0/2779597177 <== mon.0 v2:172.21.15.146:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d0036050 con 0x5643130ba950 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81cbe00640 1 -- 172.21.15.146:0/2779597177 <== mon.0 v2:172.21.15.146:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f81d80990f0 con 0x5643130ba950 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81f0406740 10 client.17391 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81f0406740 10 client.17391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81f0406740 20 client.17391 populate_metadata read hostname 'smithi146' 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81f0406740 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56431313cbb0 0x56431313efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81f0406740 4 client.17391 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:31:49.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 --> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564313027a60 con 0x56431313cbb0 2024-04-24T08:31:49.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.503+0000 7f81e6000640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56431313cbb0 0x56431313efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:49.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.504+0000 7f81e6000640 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56431313cbb0 0x56431313efa0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9df848a036f18d7a server_cookie=4653febdcfa3ea03 in_seq=0 out_seq=0 2024-04-24T08:31:49.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.504+0000 7f81cbe00640 10 client.17391 ms_handle_connect on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:49.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.504+0000 7f81cbe00640 1 -- 172.21.15.146:0/2779597177 <== mds.0 v2:172.21.15.153:6838/2023799727 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564313027a60 con 0x56431313cbb0 2024-04-24T08:31:49.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.504+0000 7f81cbe00640 10 client.17391 handle_command_reply: tid=0 2024-04-24T08:31:49.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.504+0000 7f81f0406740 1 client.17391 shutdown 2024-04-24T08:31:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.504+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 msgr2=0x5643130cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.504+0000 7f81f0406740 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130cdb50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f81d8000c00 tx=0x7f81d8003e10 comp rx=0 tx=0).stop 2024-04-24T08:31:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 shutdown_connections 2024-04-24T08:31:49.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:6838/2023799727,v1:172.21.15.153:6839/2023799727] conn(0x56431313cbb0 0x56431313efa0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5643130bb270 0x5643130ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5643130c5940 0x5643130c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81cbe00640 0 client.17391 ms_handle_reset on v2:172.21.15.153:6838/2023799727 2024-04-24T08:31:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 --2- 172.21.15.146:0/2779597177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5643130ba950 0x5643130cdb50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 >> 172.21.15.146:0/2779597177 conn(0x5643130b6c70 msgr2=0x5643130b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 shutdown_connections 2024-04-24T08:31:49.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 1 -- 172.21.15.146:0/2779597177 wait complete. 2024-04-24T08:31:49.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81f0406740 20 client.17391 trim_cache size 0 max 16384 2024-04-24T08:31:49.520 INFO:teuthology.orchestra.run.smithi146.stdout:[] 2024-04-24T08:31:49.521 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f81cc0a3d20 msgr2=0x7f81cc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81ed600640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f81cc0a3d20 0x7f81cc0a61e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009ce0 tx=0x7f81e0005f90 comp rx=0 tx=0).stop 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e8108610 msgr2=0x7f81e819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.505+0000 7f81ed600640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e8108610 0x7f81e819ef40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f81d800e9c0 tx=0x7f81d800ee80 comp rx=0 tx=0).stop 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 shutdown_connections 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f81cc0a3d20 0x7f81cc0a61e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81e810e660 0x7f81e81a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81e8108fd0 0x7f81e819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.522 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 --2- 172.21.15.146:0/945297390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81e8108610 0x7f81e819ef40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 >> 172.21.15.146:0/945297390 conn(0x7f81e8076540 msgr2=0x7f81e8077d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 shutdown_connections 2024-04-24T08:31:49.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:49.506+0000 7f81ed600640 1 -- 172.21.15.146:0/945297390 wait complete. 2024-04-24T08:31:49.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:49 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/4215477836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:31:49.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:49 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/4215477836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:31:50.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:49 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/4215477836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:31:50.525 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-24T08:31:50.525 INFO:tasks.fwd_scrub:done joining 2024-04-24T08:31:50.526 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-24T08:31:50.538 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:31:50.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 -- 172.21.15.146:0/218659180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7241084a0 msgr2=0x7ff724108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:50.750 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 --2- 172.21.15.146:0/218659180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7241084a0 0x7ff724108900 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009900 tx=0x7ff70c02a480 comp rx=0 tx=0).stop 2024-04-24T08:31:50.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 -- 172.21.15.146:0/218659180 shutdown_connections 2024-04-24T08:31:50.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 --2- 172.21.15.146:0/218659180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff724108e40 0x7ff7241152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 --2- 172.21.15.146:0/218659180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff7241084a0 0x7ff724108900 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 --2- 172.21.15.146:0/218659180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff72410d3c0 0x7ff72410d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 -- 172.21.15.146:0/218659180 >> 172.21.15.146:0/218659180 conn(0x7ff724077b80 msgr2=0x7ff724079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:50.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 -- 172.21.15.146:0/218659180 shutdown_connections 2024-04-24T08:31:50.751 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.748+0000 7ff72a600640 1 -- 172.21.15.146:0/218659180 wait complete. 2024-04-24T08:31:50.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.749+0000 7ff72a600640 1 Processor -- start 2024-04-24T08:31:50.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.749+0000 7ff72a600640 1 -- start start 2024-04-24T08:31:50.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff72a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7241084a0 0x7ff7241a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:50.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff72a600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff724108e40 0x7ff7241a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:50.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff72a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff72410d3c0 0x7ff72419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:50.754 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff72a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ff724118830 con 0x7ff72410d3c0 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff72a600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241186b0 con 0x7ff724108e40 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff72a600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff7241189b0 con 0x7ff7241084a0 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff729c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff72410d3c0 0x7ff72419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff729c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff72410d3c0 0x7ff72419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55456/0 (socket says 172.21.15.146:55456) 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff729200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7241084a0 0x7ff7241a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff729c00640 1 -- 172.21.15.146:0/3481435706 learned_addr learned my addr 172.21.15.146:0/3481435706 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff71be00640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff724108e40 0x7ff7241a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff729200640 1 -- 172.21.15.146:0/3481435706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff724108e40 msgr2=0x7ff7241a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:50.755 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.750+0000 7ff729200640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff724108e40 0x7ff7241a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff729200640 1 -- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff72410d3c0 msgr2=0x7ff72419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff729200640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff72410d3c0 0x7ff72419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff729200640 1 -- 172.21.15.146:0/3481435706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72419f250 con 0x7ff7241084a0 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff729200640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7241084a0 0x7ff7241a4920 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff71400e9f0 tx=0x7ff71400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff722000640 1 -- 172.21.15.146:0/3481435706 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71400cd80 con 0x7ff7241084a0 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff722000640 1 -- 172.21.15.146:0/3481435706 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7140035f0 con 0x7ff7241084a0 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff72419f540 con 0x7ff7241084a0 2024-04-24T08:31:50.756 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff722000640 1 -- 172.21.15.146:0/3481435706 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714010690 con 0x7ff7241084a0 2024-04-24T08:31:50.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.751+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7241ab7e0 con 0x7ff7241084a0 2024-04-24T08:31:50.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.753+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff724069190 con 0x7ff7241084a0 2024-04-24T08:31:50.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.754+0000 7ff722000640 1 -- 172.21.15.146:0/3481435706 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7ff714003b30 con 0x7ff7241084a0 2024-04-24T08:31:50.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.756+0000 7ff722000640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7080a3da0 0x7ff7080a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:50.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.756+0000 7ff722000640 1 -- 172.21.15.146:0/3481435706 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7ff7140d0a10 con 0x7ff7241084a0 2024-04-24T08:31:50.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.756+0000 7ff71be00640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7080a3da0 0x7ff7080a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:50.759 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.757+0000 7ff71be00640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7080a3da0 0x7ff7080a6260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff70c000c00 tx=0x7ff70c005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:50.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.760+0000 7ff722000640 1 -- 172.21.15.146:0/3481435706 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff71409d040 con 0x7ff7241084a0 2024-04-24T08:31:50.894 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:50 smithi146 ceph-mon[27805]: pgmap v330: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:50.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:50 smithi153 ceph-mon[32746]: pgmap v330: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:50.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.963+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff72419ffc0 con 0x7ff7241084a0 2024-04-24T08:31:50.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.965+0000 7ff722000640 1 -- 172.21.15.146:0/3481435706 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+23158 (secure 0 0 0) 0x7ff71409c940 con 0x7ff7241084a0 2024-04-24T08:31:50.968 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:31:50.968 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":99,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:31:15.910980+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:0/4151754658":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2355630587":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6800/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/393585939":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/4023745007":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:31:50.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7080a3da0 msgr2=0x7ff7080a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:50.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7080a3da0 0x7ff7080a6260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff70c000c00 tx=0x7ff70c005b50 comp rx=0 tx=0).stop 2024-04-24T08:31:50.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7241084a0 msgr2=0x7ff7241a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:50.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7241084a0 0x7ff7241a4920 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff71400e9f0 tx=0x7ff71400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.970 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 shutdown_connections 2024-04-24T08:31:50.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ff7080a3da0 0x7ff7080a6260 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff72410d3c0 0x7ff72419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff724108e40 0x7ff7241a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 --2- 172.21.15.146:0/3481435706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7241084a0 0x7ff7241a4920 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:50.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 >> 172.21.15.146:0/3481435706 conn(0x7ff724077b80 msgr2=0x7ff7240781a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:50.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 shutdown_connections 2024-04-24T08:31:50.971 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:50.968+0000 7ff72a600640 1 -- 172.21.15.146:0/3481435706 wait complete. 2024-04-24T08:31:50.990 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:31:50.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:31:51.065 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:50 smithi177 ceph-mon[33046]: pgmap v330: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:51.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 -- 172.21.15.146:0/3853268752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b0104e60 msgr2=0x7fa0b0108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 --2- 172.21.15.146:0/3853268752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b0104e60 0x7fa0b0108ea0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa09c009900 tx=0x7fa09c02a410 comp rx=0 tx=0).stop 2024-04-24T08:31:51.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 -- 172.21.15.146:0/3853268752 shutdown_connections 2024-04-24T08:31:51.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 --2- 172.21.15.146:0/3853268752 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b0109580 0x7fa0b0111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.205 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 --2- 172.21.15.146:0/3853268752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b0104e60 0x7fa0b0108ea0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 --2- 172.21.15.146:0/3853268752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0b01044b0 0x7fa0b0104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 -- 172.21.15.146:0/3853268752 >> 172.21.15.146:0/3853268752 conn(0x7fa0b00ffd90 msgr2=0x7fa0b01021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:51.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 -- 172.21.15.146:0/3853268752 shutdown_connections 2024-04-24T08:31:51.206 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.203+0000 7fa0b8000640 1 -- 172.21.15.146:0/3853268752 wait complete. 2024-04-24T08:31:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 Processor -- start 2024-04-24T08:31:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 -- start start 2024-04-24T08:31:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b01044b0 0x7fa0b01a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.209 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b0109580 0x7fa0b01a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0b019a9a0 0x7fa0b019ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b0114640 con 0x7fa0b01044b0 2024-04-24T08:31:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b01144c0 con 0x7fa0b0109580 2024-04-24T08:31:51.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.204+0000 7fa0b8000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b01147c0 con 0x7fa0b019a9a0 2024-04-24T08:31:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b01044b0 0x7fa0b01a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b01044b0 0x7fa0b01a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41572/0 (socket says 172.21.15.146:41572) 2024-04-24T08:31:51.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 -- 172.21.15.146:0/650602437 learned_addr learned my addr 172.21.15.146:0/650602437 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b6000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0b019a9a0 0x7fa0b019ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b4c00640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b0109580 0x7fa0b01a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 -- 172.21.15.146:0/650602437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0b019a9a0 msgr2=0x7fa0b019ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0b019a9a0 0x7fa0b019ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 -- 172.21.15.146:0/650602437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b0109580 msgr2=0x7fa0b01a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b0109580 0x7fa0b01a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 -- 172.21.15.146:0/650602437 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b019b4c0 con 0x7fa0b01044b0 2024-04-24T08:31:51.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b6000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0b019a9a0 0x7fa0b019ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:31:51.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.205+0000 7fa0b5600640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b01044b0 0x7fa0b01a07f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400d950 tx=0x7fa0a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:51.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.206+0000 7fa0ae000640 1 -- 172.21.15.146:0/650602437 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4017030 con 0x7fa0b01044b0 2024-04-24T08:31:51.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.206+0000 7fa0ae000640 1 -- 172.21.15.146:0/650602437 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0a4004020 con 0x7fa0b01044b0 2024-04-24T08:31:51.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.206+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b019b7b0 con 0x7fa0b01044b0 2024-04-24T08:31:51.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.206+0000 7fa0ae000640 1 -- 172.21.15.146:0/650602437 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40047a0 con 0x7fa0b01044b0 2024-04-24T08:31:51.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.206+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b01a7a00 con 0x7fa0b01044b0 2024-04-24T08:31:51.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.207+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b0069270 con 0x7fa0b01044b0 2024-04-24T08:31:51.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.209+0000 7fa0ae000640 1 -- 172.21.15.146:0/650602437 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fa0a40041d0 con 0x7fa0b01044b0 2024-04-24T08:31:51.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.210+0000 7fa0ae000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa0980a3d20 0x7fa0980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.210+0000 7fa0ae000640 1 -- 172.21.15.146:0/650602437 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fa0a40d0340 con 0x7fa0b01044b0 2024-04-24T08:31:51.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.211+0000 7fa0b4c00640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa0980a3d20 0x7fa0980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.216 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.211+0000 7fa0b4c00640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa0980a3d20 0x7fa0980a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa09c006010 tx=0x7fa09c005a20 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:51.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.215+0000 7fa0ae000640 1 -- 172.21.15.146:0/650602437 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0a409c970 con 0x7fa0b01044b0 2024-04-24T08:31:51.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.426+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa0b010c7e0 con 0x7fa0b01044b0 2024-04-24T08:31:51.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.427+0000 7fa0ae000640 1 -- 172.21.15.146:0/650602437 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7fa0a409c270 con 0x7fa0b01044b0 2024-04-24T08:31:51.429 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:31:51.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa0980a3d20 msgr2=0x7fa0980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa0980a3d20 0x7fa0980a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa09c006010 tx=0x7fa09c005a20 comp rx=0 tx=0).stop 2024-04-24T08:31:51.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b01044b0 msgr2=0x7fa0b01a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b01044b0 0x7fa0b01a07f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400d950 tx=0x7fa0a400de10 comp rx=0 tx=0).stop 2024-04-24T08:31:51.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 shutdown_connections 2024-04-24T08:31:51.431 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fa0980a3d20 0x7fa0980a61e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0b019a9a0 0x7fa0b019ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0b0109580 0x7fa0b01a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 --2- 172.21.15.146:0/650602437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0b01044b0 0x7fa0b01a07f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 >> 172.21.15.146:0/650602437 conn(0x7fa0b00ffd90 msgr2=0x7fa0b0101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:51.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 shutdown_connections 2024-04-24T08:31:51.432 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.429+0000 7fa0b8000640 1 -- 172.21.15.146:0/650602437 wait complete. 2024-04-24T08:31:51.450 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:31:51.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 -- 172.21.15.146:0/4166383730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 msgr2=0x7f3f480746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.653 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 --2- 172.21.15.146:0/4166383730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 0x7f3f480746b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009900 tx=0x7f3f3802a480 comp rx=0 tx=0).stop 2024-04-24T08:31:51.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 -- 172.21.15.146:0/4166383730 shutdown_connections 2024-04-24T08:31:51.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 --2- 172.21.15.146:0/4166383730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f48074bf0 0x7f3f4810e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 --2- 172.21.15.146:0/4166383730 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f4810e8e0 0x7f3f48110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 --2- 172.21.15.146:0/4166383730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 0x7f3f480746b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 -- 172.21.15.146:0/4166383730 >> 172.21.15.146:0/4166383730 conn(0x7f3f48079450 msgr2=0x7f3f48079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:51.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 -- 172.21.15.146:0/4166383730 shutdown_connections 2024-04-24T08:31:51.655 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.651+0000 7f3f4f600640 1 -- 172.21.15.146:0/4166383730 wait complete. 2024-04-24T08:31:51.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.652+0000 7f3f4f600640 1 Processor -- start 2024-04-24T08:31:51.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.652+0000 7f3f4f600640 1 -- start start 2024-04-24T08:31:51.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.652+0000 7f3f4f600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 0x7f3f481a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.657 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.652+0000 7f3f4f600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48074bf0 0x7f3f481a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4f600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f4810e8e0 0x7f3f4819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4f600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3f48114620 con 0x7f3f480742d0 2024-04-24T08:31:51.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4f600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481144a0 con 0x7f3f48074bf0 2024-04-24T08:31:51.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4f600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3f481147a0 con 0x7f3f4810e8e0 2024-04-24T08:31:51.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 0x7f3f481a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.658 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f3fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48074bf0 0x7f3f481a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f4810e8e0 0x7f3f4819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 0x7f3f481a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41584/0 (socket says 172.21.15.146:41584) 2024-04-24T08:31:51.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4cc00640 1 -- 172.21.15.146:0/3446945708 learned_addr learned my addr 172.21.15.146:0/3446945708 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:51.659 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f3fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48074bf0 0x7f3f481a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:55278/0 (socket says 172.21.15.146:55278) 2024-04-24T08:31:51.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f3fe00640 1 -- 172.21.15.146:0/3446945708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f4810e8e0 msgr2=0x7f3f4819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f3fe00640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f4810e8e0 0x7f3f4819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f3fe00640 1 -- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 msgr2=0x7f3f481a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f3fe00640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 0x7f3f481a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.660 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f3fe00640 1 -- 172.21.15.146:0/3446945708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4819b0c0 con 0x7f3f48074bf0 2024-04-24T08:31:51.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.653+0000 7f3f4d600640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f4810e8e0 0x7f3f4819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:51.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.654+0000 7f3f3fe00640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48074bf0 0x7f3f481a0c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000e9f0 tx=0x7f3f3000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:51.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.654+0000 7f3f46000640 1 -- 172.21.15.146:0/3446945708 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3000cd80 con 0x7f3f48074bf0 2024-04-24T08:31:51.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.654+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4819b3b0 con 0x7f3f48074bf0 2024-04-24T08:31:51.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.654+0000 7f3f46000640 1 -- 172.21.15.146:0/3446945708 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f300035f0 con 0x7f3f48074bf0 2024-04-24T08:31:51.661 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.654+0000 7f3f46000640 1 -- 172.21.15.146:0/3446945708 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30010670 con 0x7f3f48074bf0 2024-04-24T08:31:51.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.654+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f481a7600 con 0x7f3f48074bf0 2024-04-24T08:31:51.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.655+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f4819be20 con 0x7f3f48074bf0 2024-04-24T08:31:51.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.657+0000 7f3f46000640 1 -- 172.21.15.146:0/3446945708 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3f300037a0 con 0x7f3f48074bf0 2024-04-24T08:31:51.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.659+0000 7f3f46000640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3f2c0a3d20 0x7f3f2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:51.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.659+0000 7f3f46000640 1 -- 172.21.15.146:0/3446945708 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f3f300d0bd0 con 0x7f3f48074bf0 2024-04-24T08:31:51.663 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.659+0000 7f3f4cc00640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3f2c0a3d20 0x7f3f2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:51.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.660+0000 7f3f4cc00640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3f2c0a3d20 0x7f3f2c0a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f380096c0 tx=0x7f3f380046c0 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:51.664 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.661+0000 7f3f46000640 1 -- 172.21.15.146:0/3446945708 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f3009d290 con 0x7f3f48074bf0 2024-04-24T08:31:51.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.864+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3f4802d620 con 0x7f3f48074bf0 2024-04-24T08:31:51.866 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.865+0000 7f3f46000640 1 -- 172.21.15.146:0/3446945708 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) ==== 104+0+11 (secure 0 0 0) 0x7f3f3009cb90 con 0x7f3f48074bf0 2024-04-24T08:31:51.867 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:31:51.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3f2c0a3d20 msgr2=0x7f3f2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3f2c0a3d20 0x7f3f2c0a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f380096c0 tx=0x7f3f380046c0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48074bf0 msgr2=0x7f3f481a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48074bf0 0x7f3f481a0c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000e9f0 tx=0x7f3f3000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 shutdown_connections 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3f2c0a3d20 0x7f3f2c0a61e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f4810e8e0 0x7f3f4819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f48074bf0 0x7f3f481a0c80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 --2- 172.21.15.146:0/3446945708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f480742d0 0x7f3f481a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:51.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 >> 172.21.15.146:0/3446945708 conn(0x7f3f48079450 msgr2=0x7f3f480fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:51.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 shutdown_connections 2024-04-24T08:31:51.871 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:51.867+0000 7f3f4f600640 1 -- 172.21.15.146:0/3446945708 wait complete. 2024-04-24T08:31:51.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:31:52.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 -- 172.21.15.146:0/3743011957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c481044b0 msgr2=0x7f5c48104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/3743011957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c481044b0 0x7f5c48104890 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009ca0 tx=0x7f5c4002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 -- 172.21.15.146:0/3743011957 shutdown_connections 2024-04-24T08:31:52.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/3743011957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c48109580 0x7f5c48111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/3743011957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c48104e60 0x7f5c48108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/3743011957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c481044b0 0x7f5c48104890 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 -- 172.21.15.146:0/3743011957 >> 172.21.15.146:0/3743011957 conn(0x7f5c480ffd90 msgr2=0x7f5c481021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:52.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 -- 172.21.15.146:0/3743011957 shutdown_connections 2024-04-24T08:31:52.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.096+0000 7f5c4ce00640 1 -- 172.21.15.146:0/3743011957 wait complete. 2024-04-24T08:31:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.097+0000 7f5c4ce00640 1 Processor -- start 2024-04-24T08:31:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c4ce00640 1 -- start start 2024-04-24T08:31:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c4ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c481044b0 0x7f5c481a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c4ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c48109580 0x7f5c481a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.102 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c4ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4819aa30 0x7f5c4819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c4ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48114690 con 0x7f5c48109580 2024-04-24T08:31:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c4ce00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48114510 con 0x7f5c481044b0 2024-04-24T08:31:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c4ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48114810 con 0x7f5c4819aa30 2024-04-24T08:31:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c45400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c48109580 0x7f5c481a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c45e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c481044b0 0x7f5c481a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c46800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4819aa30 0x7f5c4819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.103 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c45400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c48109580 0x7f5c481a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41592/0 (socket says 172.21.15.146:41592) 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c45e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c481044b0 0x7f5c481a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:55296/0 (socket says 172.21.15.146:55296) 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.098+0000 7f5c45400640 1 -- 172.21.15.146:0/1851482662 learned_addr learned my addr 172.21.15.146:0/1851482662 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c45400640 1 -- 172.21.15.146:0/1851482662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4819aa30 msgr2=0x7f5c4819aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c45400640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4819aa30 0x7f5c4819aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c45400640 1 -- 172.21.15.146:0/1851482662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c481044b0 msgr2=0x7f5c481a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c45400640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c481044b0 0x7f5c481a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c45400640 1 -- 172.21.15.146:0/1851482662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4819b550 con 0x7f5c48109580 2024-04-24T08:31:52.104 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c45e00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c481044b0 0x7f5c481a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:31:52.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c45400640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c48109580 0x7f5c481a0dc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000e9a0 tx=0x7f5c3000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:52.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.099+0000 7f5c3ea00640 1 -- 172.21.15.146:0/1851482662 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000cd80 con 0x7f5c48109580 2024-04-24T08:31:52.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.100+0000 7f5c3ea00640 1 -- 172.21.15.146:0/1851482662 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c300035f0 con 0x7f5c48109580 2024-04-24T08:31:52.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.100+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4819b840 con 0x7f5c48109580 2024-04-24T08:31:52.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.100+0000 7f5c3ea00640 1 -- 172.21.15.146:0/1851482662 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30010670 con 0x7f5c48109580 2024-04-24T08:31:52.105 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.100+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c481a7a90 con 0x7f5c48109580 2024-04-24T08:31:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.101+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c48069270 con 0x7f5c48109580 2024-04-24T08:31:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.103+0000 7f5c3ea00640 1 -- 172.21.15.146:0/1851482662 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f5c300037a0 con 0x7f5c48109580 2024-04-24T08:31:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.104+0000 7f5c3ea00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5c2c0a3d20 0x7f5c2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.105+0000 7f5c3ea00640 1 -- 172.21.15.146:0/1851482662 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7f5c300d0aa0 con 0x7f5c48109580 2024-04-24T08:31:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.105+0000 7f5c45e00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5c2c0a3d20 0x7f5c2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.107 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.105+0000 7f5c45e00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5c2c0a3d20 0x7f5c2c0a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c4819bb70 tx=0x7f5c40009800 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:52.110 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.108+0000 7f5c3ea00640 1 -- 172.21.15.146:0/1851482662 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c3009d130 con 0x7f5c48109580 2024-04-24T08:31:52.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:51 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3481435706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:31:52.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:51 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/650602437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:31:52.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:51 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3446945708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:31:52.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:51 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3481435706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:31:52.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:51 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/650602437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:31:52.243 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:51 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3446945708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:31:52.314 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.313+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5c4810c810 con 0x7f5c48109580 2024-04-24T08:31:52.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:51 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3481435706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:31:52.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:51 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/650602437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:31:52.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:51 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3446945708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:31:52.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.314+0000 7f5c3ea00640 1 -- 172.21.15.146:0/1851482662 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) ==== 100+0+11 (secure 0 0 0) 0x7f5c3009ca30 con 0x7f5c48109580 2024-04-24T08:31:52.316 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:31:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5c2c0a3d20 msgr2=0x7f5c2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5c2c0a3d20 0x7f5c2c0a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c4819bb70 tx=0x7f5c40009800 comp rx=0 tx=0).stop 2024-04-24T08:31:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c48109580 msgr2=0x7f5c481a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c48109580 0x7f5c481a0dc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000e9a0 tx=0x7f5c3000ee60 comp rx=0 tx=0).stop 2024-04-24T08:31:52.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 shutdown_connections 2024-04-24T08:31:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f5c2c0a3d20 0x7f5c2c0a61e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c4819aa30 0x7f5c4819aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c48109580 0x7f5c481a0dc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 --2- 172.21.15.146:0/1851482662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c481044b0 0x7f5c481a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 >> 172.21.15.146:0/1851482662 conn(0x7f5c480ffd90 msgr2=0x7f5c481020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 shutdown_connections 2024-04-24T08:31:52.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.316+0000 7f5c4ce00640 1 -- 172.21.15.146:0/1851482662 wait complete. 2024-04-24T08:31:52.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-24T08:31:52.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 -- 172.21.15.146:0/3112794983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 msgr2=0x7fb51010e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 --2- 172.21.15.146:0/3112794983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 0x7fb51010e180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009900 tx=0x7fb4f802a480 comp rx=0 tx=0).stop 2024-04-24T08:31:52.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 -- 172.21.15.146:0/3112794983 shutdown_connections 2024-04-24T08:31:52.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 --2- 172.21.15.146:0/3112794983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 0x7fb51010e180 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 --2- 172.21.15.146:0/3112794983 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb51010e6c0 0x7fb510110ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 --2- 172.21.15.146:0/3112794983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 0x7fb5100747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 -- 172.21.15.146:0/3112794983 >> 172.21.15.146:0/3112794983 conn(0x7fb5100fd930 msgr2=0x7fb5100ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:52.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 -- 172.21.15.146:0/3112794983 shutdown_connections 2024-04-24T08:31:52.541 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.538+0000 7fb50fe00640 1 -- 172.21.15.146:0/3112794983 wait complete. 2024-04-24T08:31:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.539+0000 7fb50fe00640 1 Processor -- start 2024-04-24T08:31:52.543 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.539+0000 7fb50fe00640 1 -- start start 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.539+0000 7fb50fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 0x7fb5101a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.539+0000 7fb50fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 0x7fb5101a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb51010e6c0 0x7fb51019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fb510114490 con 0x7fb5100743d0 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50fe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb510114310 con 0x7fb51010e6c0 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb510114610 con 0x7fb510074cf0 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 0x7fb5101a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.544 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 0x7fb5101a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41612/0 (socket says 172.21.15.146:41612) 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 -- 172.21.15.146:0/1696473464 learned_addr learned my addr 172.21.15.146:0/1696473464 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50f400640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb51010e6c0 0x7fb51019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb506000640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 0x7fb5101a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 -- 172.21.15.146:0/1696473464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 msgr2=0x7fb5101a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 0x7fb5101a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 -- 172.21.15.146:0/1696473464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb51010e6c0 msgr2=0x7fb51019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb51010e6c0 0x7fb51019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.540+0000 7fb50ea00640 1 -- 172.21.15.146:0/1696473464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51019afa0 con 0x7fb5100743d0 2024-04-24T08:31:52.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.541+0000 7fb50ea00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 0x7fb5101a0660 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb50000e9f0 tx=0x7fb50000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.541+0000 7fb507e00640 1 -- 172.21.15.146:0/1696473464 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50000cd80 con 0x7fb5100743d0 2024-04-24T08:31:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.541+0000 7fb507e00640 1 -- 172.21.15.146:0/1696473464 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb500010910 con 0x7fb5100743d0 2024-04-24T08:31:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.541+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51019b290 con 0x7fb5100743d0 2024-04-24T08:31:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.541+0000 7fb507e00640 1 -- 172.21.15.146:0/1696473464 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5000188f0 con 0x7fb5100743d0 2024-04-24T08:31:52.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.541+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5101a74b0 con 0x7fb5100743d0 2024-04-24T08:31:52.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.542+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb510075a80 con 0x7fb5100743d0 2024-04-24T08:31:52.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.544+0000 7fb507e00640 1 -- 172.21.15.146:0/1696473464 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fb500010ac0 con 0x7fb5100743d0 2024-04-24T08:31:52.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.545+0000 7fb507e00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb4f40a3d20 0x7fb4f40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:31:52.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.546+0000 7fb507e00640 1 -- 172.21.15.146:0/1696473464 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8542+0+0 (secure 0 0 0) 0x7fb5000d0c10 con 0x7fb5100743d0 2024-04-24T08:31:52.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.546+0000 7fb506000640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb4f40a3d20 0x7fb4f40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:31:52.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.546+0000 7fb506000640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb4f40a3d20 0x7fb4f40a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8000c00 tx=0x7fb4f8005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:31:52.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.550+0000 7fb507e00640 1 -- 172.21.15.146:0/1696473464 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb50009cbf0 con 0x7fb5100743d0 2024-04-24T08:31:52.807 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.805+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fb510068d40 con 0x7fb5100743d0 2024-04-24T08:31:52.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.817+0000 7fb507e00640 1 -- 172.21.15.146:0/1696473464 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+746068 (secure 0 0 0) 0x7fb5000a0ef0 con 0x7fb5100743d0 2024-04-24T08:31:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:31:52.833 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:31:52.833 INFO:teuthology.orchestra.run.smithi146.stdout: "epoch": 27, 2024-04-24T08:31:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "active_gid": 15087, 2024-04-24T08:31:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "active_name": "x", 2024-04-24T08:31:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "active_addrs": { 2024-04-24T08:31:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-04-24T08:31:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:52.834 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-04-24T08:31:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6800", 2024-04-24T08:31:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 827025587 2024-04-24T08:31:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-04-24T08:31:52.835 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:6801", 2024-04-24T08:31:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 827025587 2024-04-24T08:31:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "active_addr": "172.21.15.146:6801/827025587", 2024-04-24T08:31:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "active_change": "2024-04-24T08:21:11.077824+0000", 2024-04-24T08:31:52.836 INFO:teuthology.orchestra.run.smithi146.stdout: "active_mgr_features": 4540701547738038271, 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "available": true, 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "standbys": [ 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "gid": 25193, 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "y", 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_features": 4540701547738038271, 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: "available_modules": [ 2024-04-24T08:31:52.837 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "alerts", 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.838 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.839 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T08:31:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.840 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:52.841 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.842 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.843 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:52.844 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:52.845 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:52.846 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.847 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.848 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.849 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_destination": { 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_destination", 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.850 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Email address to send alerts to", 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_from_name": { 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_from_name", 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.851 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "Ceph", 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Email From: name", 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.852 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_host": { 2024-04-24T08:31:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_host", 2024-04-24T08:31:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.853 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP server", 2024-04-24T08:31:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.854 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_password": { 2024-04-24T08:31:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_password", 2024-04-24T08:31:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.855 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Password to authenticate with", 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_port": { 2024-04-24T08:31:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_port", 2024-04-24T08:31:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "465", 2024-04-24T08:31:52.857 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP port", 2024-04-24T08:31:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.858 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_sender": { 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_sender", 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.859 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP envelope sender", 2024-04-24T08:31:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_ssl": { 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_ssl", 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.861 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_user": { 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_user", 2024-04-24T08:31:52.862 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "User to authenticate as", 2024-04-24T08:31:52.863 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:52.864 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "balancer", 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "active": { 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "active", 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.865 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.866 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "begin_time": { 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "begin_time", 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0000", 2024-04-24T08:31:52.867 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "begin_weekday": { 2024-04-24T08:31:52.868 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "begin_weekday", 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0", 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "6", 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.869 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T08:31:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:31:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_max_iterations": { 2024-04-24T08:31:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_max_iterations", 2024-04-24T08:31:52.870 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "25", 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "250", 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.871 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_metrics": { 2024-04-24T08:31:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_metrics", 2024-04-24T08:31:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.872 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T08:31:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.873 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_step": { 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_step", 2024-04-24T08:31:52.874 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0.5", 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0.001", 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "0.999", 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "aggressiveness of optimization", 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T08:31:52.875 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "end_time": { 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "end_time", 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "2359", 2024-04-24T08:31:52.876 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.877 INFO:teuthology.orchestra.run.smithi146.stdout: "end_weekday": { 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "end_weekday", 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0", 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "6", 2024-04-24T08:31:52.878 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.879 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T08:31:52.879 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:31:52.879 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.879 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.880 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:52.880 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:52.880 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.880 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.880 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.881 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.881 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.881 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.881 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.882 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:52.882 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:52.882 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:52.882 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:52.883 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:52.883 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:52.883 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.883 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.884 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.884 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.886 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.886 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.886 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.887 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.887 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.887 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.887 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.888 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.888 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.888 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:52.889 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:52.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.889 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.890 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:52.890 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.890 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.890 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.891 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:52.891 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:52.891 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:52.891 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:52.892 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:52.892 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:52.892 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.893 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.893 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.893 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.893 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.894 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.894 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:52.894 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:52.895 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.895 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.895 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.895 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.896 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.896 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.896 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.896 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.897 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.897 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.897 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.898 INFO:teuthology.orchestra.run.smithi146.stdout: "min_score": { 2024-04-24T08:31:52.898 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "min_score", 2024-04-24T08:31:52.898 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:52.899 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.899 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.899 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:52.899 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.900 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.900 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.900 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T08:31:52.900 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.901 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.901 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": { 2024-04-24T08:31:52.902 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mode", 2024-04-24T08:31:52.902 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.902 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.903 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.903 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "upmap", 2024-04-24T08:31:52.903 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.903 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.904 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.904 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-compat", 2024-04-24T08:31:52.904 INFO:teuthology.orchestra.run.smithi146.stdout: "none", 2024-04-24T08:31:52.904 INFO:teuthology.orchestra.run.smithi146.stdout: "read", 2024-04-24T08:31:52.905 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap", 2024-04-24T08:31:52.905 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap-read" 2024-04-24T08:31:52.905 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.906 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Balancer mode", 2024-04-24T08:31:52.906 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.906 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.906 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.907 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_ids": { 2024-04-24T08:31:52.907 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pool_ids", 2024-04-24T08:31:52.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.908 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.908 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.908 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.908 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.909 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.909 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.909 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T08:31:52.910 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.910 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.910 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.911 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:52.911 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:52.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.911 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.912 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:52.912 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.912 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.912 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.912 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T08:31:52.913 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.913 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.913 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.913 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.914 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap_max_deviation": { 2024-04-24T08:31:52.914 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "upmap_max_deviation", 2024-04-24T08:31:52.914 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:52.914 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.915 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.915 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:52.915 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:52.915 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.915 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.915 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap_max_optimizations": { 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "upmap_max_optimizations", 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:52.916 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.917 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.917 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:52.917 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.918 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.918 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.918 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T08:31:52.918 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.918 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cephadm", 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:52.919 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:52.920 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:52.920 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_down_multiplier": { 2024-04-24T08:31:52.920 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_down_multiplier", 2024-04-24T08:31:52.920 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:52.921 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.921 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.921 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3.0", 2024-04-24T08:31:52.922 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.922 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.922 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.922 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T08:31:52.923 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.923 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.923 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.923 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_refresh_rate": { 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_refresh_rate", 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "20", 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.924 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_starting_port": { 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_starting_port", 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:52.925 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.926 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "4721", 2024-04-24T08:31:52.926 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.926 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.926 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.926 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_ptrace": { 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_ptrace", 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.927 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T08:31:52.928 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: "autotune_interval": { 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "autotune_interval", 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.929 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "autotune_memory_target_ratio": { 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0.7", 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.931 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm_log_destination": { 2024-04-24T08:31:52.932 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cephadm_log_destination", 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.933 INFO:teuthology.orchestra.run.smithi146.stdout: "file", 2024-04-24T08:31:52.934 INFO:teuthology.orchestra.run.smithi146.stdout: "file,syslog", 2024-04-24T08:31:52.934 INFO:teuthology.orchestra.run.smithi146.stdout: "syslog" 2024-04-24T08:31:52.934 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.934 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T08:31:52.934 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.934 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.937 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.938 INFO:teuthology.orchestra.run.smithi146.stdout: "cgroups_split": { 2024-04-24T08:31:52.938 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cgroups_split", 2024-04-24T08:31:52.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.938 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.938 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:52.938 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.939 INFO:teuthology.orchestra.run.smithi146.stdout: "config_checks_enabled": { 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_checks_enabled", 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.940 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: "config_dashboard": { 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_dashboard", 2024-04-24T08:31:52.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.942 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.942 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.942 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:52.942 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.942 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.942 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.942 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_alertmanager": { 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_alertmanager", 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.943 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.944 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_base": { 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_base", 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.945 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T08:31:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Container image name, without the tag", 2024-04-24T08:31:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.946 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.947 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.947 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_elasticsearch": { 2024-04-24T08:31:52.947 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_elasticsearch", 2024-04-24T08:31:52.947 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.947 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.947 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "elasticsearch container image", 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_grafana": { 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_grafana", 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.949 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_haproxy": { 2024-04-24T08:31:52.950 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_haproxy", 2024-04-24T08:31:52.951 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.951 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.951 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.951 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T08:31:52.951 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.951 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.951 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.952 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "HAproxy container image", 2024-04-24T08:31:52.952 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.952 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.952 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_agent": { 2024-04-24T08:31:52.952 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_agent", 2024-04-24T08:31:52.953 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.953 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.953 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.953 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T08:31:52.953 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.953 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.953 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger agent container image", 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_collector": { 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_collector", 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.954 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger collector container image", 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.955 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_query": { 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_query", 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.956 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger query container image", 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.957 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_keepalived": { 2024-04-24T08:31:52.958 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_keepalived", 2024-04-24T08:31:52.958 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.958 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.958 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.958 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T08:31:52.958 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.958 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Keepalived container image", 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_loki": { 2024-04-24T08:31:52.959 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_loki", 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Loki container image", 2024-04-24T08:31:52.960 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_node_exporter": { 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_node_exporter", 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_nvmeof": { 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_nvmeof", 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.963 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Nvme-of container image", 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_prometheus": { 2024-04-24T08:31:52.964 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_prometheus", 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:52.965 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_promtail": { 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_promtail", 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.966 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Promtail container image", 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.968 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_samba": { 2024-04-24T08:31:52.968 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_samba", 2024-04-24T08:31:52.968 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.968 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.968 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.968 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T08:31:52.968 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Samba/SMB container image", 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_snmp_gateway": { 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_snmp_gateway", 2024-04-24T08:31:52.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SNMP Gateway container image", 2024-04-24T08:31:52.970 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: "container_init": { 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_init", 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.971 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.972 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:52.972 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.972 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.972 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.972 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T08:31:52.972 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "daemon_cache_timeout": { 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "daemon_cache_timeout", 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:52.973 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.974 INFO:teuthology.orchestra.run.smithi146.stdout: "default_cephadm_command_timeout": { 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "900", 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.975 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "default_registry": { 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default_registry", 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.976 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io", 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.977 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: "device_cache_timeout": { 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_cache_timeout", 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1800", 2024-04-24T08:31:52.978 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache device inventory", 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.979 INFO:teuthology.orchestra.run.smithi146.stdout: "device_enhanced_scan": { 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_enhanced_scan", 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.980 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: "facts_cache_timeout": { 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "facts_cache_timeout", 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.981 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache host facts data", 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.982 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: "grafana_dashboards_path": { 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "grafana_dashboards_path", 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T08:31:52.983 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "host_check_interval": { 2024-04-24T08:31:52.984 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "host_check_interval", 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.985 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to perform a host check", 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: "hw_monitoring": { 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "hw_monitoring", 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.986 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.987 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: "inventory_list_all": { 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "inventory_list_all", 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.988 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:52.989 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.990 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.991 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: "log_refresh_metadata": { 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_refresh_metadata", 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.993 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:52.994 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.995 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.996 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.997 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.998 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.998 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:52.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:52.998 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:52.998 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:52.998 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:52.998 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:52.999 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.000 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T08:31:53.001 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "*", 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T08:31:53.002 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "max_count_per_host": { 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_count_per_host", 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.003 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "max number of daemons per service per host", 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd_draining_count": { 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_osd_draining_count", 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.005 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "migration_current": { 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "migration_current", 2024-04-24T08:31:53.006 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "internal - do not modify", 2024-04-24T08:31:53.007 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": { 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mode", 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.008 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "root", 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm-package", 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "root" 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T08:31:53.009 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: "oob_default_addr": { 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "oob_default_addr", 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.010 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "169.254.1.1", 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.011 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: "prometheus_alerts_path": { 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prometheus_alerts_path", 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T08:31:53.012 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.013 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.013 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.013 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T08:31:53.013 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.013 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.013 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.014 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_insecure": { 2024-04-24T08:31:53.014 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_insecure", 2024-04-24T08:31:53.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.014 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.014 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.014 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.014 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_password": { 2024-04-24T08:31:53.015 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_password", 2024-04-24T08:31:53.016 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.016 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.016 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.016 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.016 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.016 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.016 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_url": { 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_url", 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.017 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.018 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_username": { 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_username", 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.019 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.020 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.020 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.020 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.020 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T08:31:53.020 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.020 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.020 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "secure_monitoring_stack": { 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "secure_monitoring_stack", 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.021 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: "service_discovery_port": { 2024-04-24T08:31:53.022 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "service_discovery_port", 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8765", 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.023 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "cephadm service discovery port", 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "ssh_config_file": { 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssh_config_file", 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.024 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.025 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "use_agent": { 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "use_agent", 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.026 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: "use_repo_digest": { 2024-04-24T08:31:53.027 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "use_repo_digest", 2024-04-24T08:31:53.028 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.028 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.028 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.028 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.028 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.028 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.028 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_failed_host_check": { 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_failed_host_check", 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.029 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.030 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.030 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.030 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.030 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.030 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.030 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T08:31:53.030 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_stray_daemons": { 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_stray_daemons", 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.031 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.032 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_stray_hosts": { 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_stray_hosts", 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.033 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.034 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crash", 2024-04-24T08:31:53.035 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.035 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.035 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.035 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.035 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.035 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.035 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.036 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.037 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.038 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.039 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.040 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.041 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "retain_interval": { 2024-04-24T08:31:53.043 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "retain_interval", 2024-04-24T08:31:53.044 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.044 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.044 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.044 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "31536000", 2024-04-24T08:31:53.044 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.044 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.044 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_recent_interval": { 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_recent_interval", 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.045 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1209600", 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.046 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dashboard", 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.047 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.048 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T08:31:53.049 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.050 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.051 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.052 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: "AUDIT_API_ENABLED": { 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.053 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.054 INFO:teuthology.orchestra.run.smithi146.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.055 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.056 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.057 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.057 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.057 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.057 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.057 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.057 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.057 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.058 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.059 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.060 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.061 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.062 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T08:31:53.063 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.064 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.065 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.066 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.066 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.066 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.066 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.066 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.066 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.066 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.068 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.069 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T08:31:53.069 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T08:31:53.069 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.069 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.069 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.069 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.069 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.070 INFO:teuthology.orchestra.run.smithi146.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.071 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.072 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.073 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.074 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.075 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_URL": { 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_URL", 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.076 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.077 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.077 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.077 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.077 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.077 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T08:31:53.077 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T08:31:53.077 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.078 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.079 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.080 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.080 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.080 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.080 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.080 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.080 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.081 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.082 INFO:teuthology.orchestra.run.smithi146.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.083 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.084 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.085 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "[]", 2024-04-24T08:31:53.086 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "MULTICLUSTER_CONFIG": { 2024-04-24T08:31:53.087 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "{}", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.088 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.089 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T08:31:53.090 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.091 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.092 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.093 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.094 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T08:31:53.095 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.096 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.097 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T08:31:53.100 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T08:31:53.101 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.101 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.101 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.101 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.101 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.101 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.101 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.102 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.102 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.102 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.102 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T08:31:53.102 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T08:31:53.103 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.103 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.103 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.103 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.103 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.103 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.104 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.105 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.107 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.108 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T08:31:53.109 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8", 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.110 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.111 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "45", 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.113 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T08:31:53.114 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.116 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.117 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.118 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: "UNSAFE_TLS_v1_2": { 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-24T08:31:53.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.120 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.121 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.123 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.124 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.124 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.124 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.124 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T08:31:53.124 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.125 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: "cross_origin_url": { 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cross_origin_url", 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.126 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.127 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "crt_file": { 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crt_file", 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.128 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: "debug": { 2024-04-24T08:31:53.129 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "debug", 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.130 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable/disable debug options", 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "jwt_token_ttl": { 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "jwt_token_ttl", 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.131 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "28800", 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.132 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "key_file": { 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "key_file", 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.133 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.134 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.135 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.136 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.137 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.139 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.140 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.140 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.140 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.140 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.140 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.140 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.140 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.141 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.141 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.141 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.141 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.141 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.141 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.141 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.142 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: "motd": { 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "motd", 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.143 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "The message of the day", 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.144 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_resolve_ip_addr": { 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.145 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:53.146 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "::", 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.147 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.148 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8080", 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.149 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl": { 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl", 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl_server_port": { 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl_server_port", 2024-04-24T08:31:53.151 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8443", 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.152 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_behaviour": { 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_behaviour", 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "redirect", 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect" 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.154 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_error_status_code": { 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_error_status_code", 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.155 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "500", 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "400", 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "599", 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.156 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: "url_prefix": { 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "url_prefix", 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.157 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "devicehealth", 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.159 INFO:teuthology.orchestra.run.smithi146.stdout: "enable_monitoring": { 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enable_monitoring", 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.160 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "monitor device health metrics", 2024-04-24T08:31:53.161 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.161 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.161 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.161 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.161 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.161 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.161 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.162 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.162 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.162 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.162 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.162 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.162 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.162 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.163 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.164 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.165 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.166 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.167 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.168 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.169 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.170 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.170 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.170 INFO:teuthology.orchestra.run.smithi146.stdout: "mark_out_threshold": { 2024-04-24T08:31:53.170 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mark_out_threshold", 2024-04-24T08:31:53.170 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.170 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.170 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "2419200", 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.171 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": { 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pool_name", 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "device_health_metrics", 2024-04-24T08:31:53.172 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.173 INFO:teuthology.orchestra.run.smithi146.stdout: "retention_period": { 2024-04-24T08:31:53.174 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "retention_period", 2024-04-24T08:31:53.174 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.174 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.174 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.174 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15552000", 2024-04-24T08:31:53.174 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.174 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long to retain device health metrics", 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: "scrape_frequency": { 2024-04-24T08:31:53.175 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "scrape_frequency", 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "86400", 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.176 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: "self_heal": { 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "self_heal", 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.177 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:53.179 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_threshold": { 2024-04-24T08:31:53.180 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_threshold", 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "7257600", 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.181 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T08:31:53.182 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.182 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.182 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.182 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.182 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.182 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "diskprediction_local", 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.183 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.184 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.185 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.186 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.187 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.188 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.190 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.191 INFO:teuthology.orchestra.run.smithi146.stdout: "predict_interval": { 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "predict_interval", 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "86400", 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.192 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "predictor_model": { 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "predictor_model", 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.193 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "prophetstor", 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.194 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.195 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.196 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "influx", 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": false, 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "influxdb python module not found", 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: "batch_size": { 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "batch_size", 2024-04-24T08:31:53.197 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.198 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.198 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.198 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5000", 2024-04-24T08:31:53.198 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.198 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.198 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.198 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "database": { 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "database", 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.199 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "ceph", 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": { 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "hostname", 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.201 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.202 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB server hostname", 2024-04-24T08:31:53.202 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.202 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.202 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.202 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.202 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:53.202 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "30", 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "5", 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T08:31:53.203 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.204 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.204 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.204 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.204 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.204 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.204 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.205 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.206 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.207 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.208 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.209 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.210 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.211 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "password": { 2024-04-24T08:31:53.212 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "password", 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.213 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "password of InfluxDB server user", 2024-04-24T08:31:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "port": { 2024-04-24T08:31:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "port", 2024-04-24T08:31:53.214 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8086", 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB server port", 2024-04-24T08:31:53.215 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl": { 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl", 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.216 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "false", 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.217 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "threads": { 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "threads", 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:53.218 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "32", 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "username": { 2024-04-24T08:31:53.219 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "username", 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.220 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "username of InfluxDB server user", 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "verify_ssl": { 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "verify_ssl", 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.221 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "true", 2024-04-24T08:31:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:31:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.222 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "insights", 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.223 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.224 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.225 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.226 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.227 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.228 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.229 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.230 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.231 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "iostat", 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.232 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.233 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.234 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.235 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.236 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.237 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.240 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.241 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "k8sevents", 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_event_retention_days": { 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ceph_event_retention_days", 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.242 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "7", 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.243 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "config_check_secs": { 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_check_secs", 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.244 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "10", 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.245 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.246 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.247 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.248 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.249 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.250 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.251 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.252 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.253 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.254 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "localpool", 2024-04-24T08:31:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "failure_domain": { 2024-04-24T08:31:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "failure_domain", 2024-04-24T08:31:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.255 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "host", 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "failure domain for any created local pool", 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T08:31:53.256 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.257 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.258 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.259 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.260 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.261 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.262 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.265 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": { 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "min_size", 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.266 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default min_size for any created local pool", 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "num_rep": { 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "num_rep", 2024-04-24T08:31:53.267 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3", 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default replica count for any created local pool", 2024-04-24T08:31:53.268 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": { 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pg_num", 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.269 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "128", 2024-04-24T08:31:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T08:31:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.270 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "prefix": { 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prefix", 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.271 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "name prefix for any created local pool", 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.272 INFO:teuthology.orchestra.run.smithi146.stdout: "subtree": { 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "subtree", 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "rack", 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.273 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.274 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mds_autoscaler", 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.275 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.276 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.277 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.278 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.279 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.282 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mirroring", 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.284 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.285 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.287 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.288 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.289 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.291 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.292 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "nfs", 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.293 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.294 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.295 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.296 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.298 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.300 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.301 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "orchestrator", 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "fail_fs": { 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "fail_fs", 2024-04-24T08:31:53.302 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T08:31:53.303 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.304 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.305 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.306 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.307 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.308 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.309 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.310 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.311 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.312 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator": { 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "orchestrator", 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.313 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm", 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "rook", 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "test_orchestrator" 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Orchestrator backend", 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.314 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "osd_perf_query", 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.315 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.316 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.317 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.318 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.319 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.320 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.321 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.322 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.324 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "osd_support", 2024-04-24T08:31:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.326 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.327 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.328 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.329 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.330 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.331 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.332 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.333 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pg_autoscaler", 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.334 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.335 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.336 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.337 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.338 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.339 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.340 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.341 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.342 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:53.343 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.345 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.345 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.345 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:53.345 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.345 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.345 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: "threshold": { 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "threshold", 2024-04-24T08:31:53.346 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3.0", 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1.0", 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "scaling threshold", 2024-04-24T08:31:53.347 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "progress", 2024-04-24T08:31:53.348 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_pg_recovery_event": { 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_pg_recovery_event", 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.349 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": { 2024-04-24T08:31:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enabled", 2024-04-24T08:31:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.351 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.352 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.353 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.353 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.353 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.353 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.353 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.353 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.353 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.354 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.355 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.355 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.355 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.355 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.355 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.355 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.356 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.356 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.356 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.356 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.356 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.356 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.356 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.357 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.358 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.358 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.358 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.358 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.358 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.358 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.358 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.359 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.360 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.360 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.360 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.360 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.360 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.360 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.360 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.361 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.361 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.361 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.361 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.361 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.361 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.361 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: "max_completed_events": { 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_completed_events", 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.362 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "50", 2024-04-24T08:31:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "number of past completed events to remember", 2024-04-24T08:31:53.363 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.364 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long the module is going to sleep", 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prometheus", 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: "cache": { 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cache", 2024-04-24T08:31:53.366 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.367 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "exclude_perf_counters": { 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "exclude_perf_counters", 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.368 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.369 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.370 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.371 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.372 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.373 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.373 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.373 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.373 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.373 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.373 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.373 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.374 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.375 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.376 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.377 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_stats_pools": { 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_stats_pools", 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.378 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "300", 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.380 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "scrape_interval": { 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "scrape_interval", 2024-04-24T08:31:53.381 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:53.382 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.382 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.382 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15.0", 2024-04-24T08:31:53.382 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.382 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.382 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.382 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.383 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "::", 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.384 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "9283", 2024-04-24T08:31:53.385 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "stale_cache_strategy": { 2024-04-24T08:31:53.386 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "stale_cache_strategy", 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "log", 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.387 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_behaviour": { 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_behaviour", 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "default", 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "default", 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "error" 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_error_status_code": { 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_error_status_code", 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.390 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "500", 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "400", 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "599", 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.391 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_support", 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.392 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.393 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.394 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.395 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.396 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.397 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.398 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.399 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.399 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.399 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.399 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.399 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.399 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.400 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.400 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.400 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.400 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.400 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.400 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.401 INFO:teuthology.orchestra.run.smithi146.stdout: "max_concurrent_snap_create": { 2024-04-24T08:31:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_concurrent_snap_create", 2024-04-24T08:31:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.402 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "mirror_snapshot_schedule": { 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T08:31:53.403 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.404 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.405 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.405 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.405 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.405 INFO:teuthology.orchestra.run.smithi146.stdout: "trash_purge_schedule": { 2024-04-24T08:31:53.405 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "trash_purge_schedule", 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.406 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.407 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "restful", 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "enable_auth": { 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enable_auth", 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.408 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.409 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.409 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.409 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.409 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.409 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.409 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "key_file": { 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "key_file", 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.411 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.411 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.411 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.411 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.411 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.411 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.411 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.412 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.413 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.414 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.415 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.416 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.417 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.418 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.419 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.420 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:53.421 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:53.422 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.422 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.422 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.422 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.422 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.422 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.422 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.423 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rgw", 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.424 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.425 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.426 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.426 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.426 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.426 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.426 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.426 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.426 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.427 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.428 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.429 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.430 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.431 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rook", 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.433 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.434 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.435 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.436 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.437 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.438 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.439 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.440 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.441 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.441 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.441 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.441 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.441 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_class": { 2024-04-24T08:31:53.441 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "storage_class", 2024-04-24T08:31:53.442 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.442 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.442 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.442 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "local", 2024-04-24T08:31:53.442 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.442 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.442 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.443 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.444 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "selftest", 2024-04-24T08:31:53.444 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.444 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.444 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.444 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.444 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.444 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.445 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.446 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.447 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.448 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.449 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.450 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.451 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "roption1": { 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "roption1", 2024-04-24T08:31:53.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.453 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: "roption2": { 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "roption2", 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.454 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "xyz", 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption1": { 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption1", 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.456 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption2": { 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption2", 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.457 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.458 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.458 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.458 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.458 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.458 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.458 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.458 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption3": { 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption3", 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.459 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption4": { 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption4", 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.461 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption5": { 2024-04-24T08:31:53.462 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption5", 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.463 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption6": { 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption6", 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.464 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.465 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption7": { 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption7", 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.466 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "42", 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.467 INFO:teuthology.orchestra.run.smithi146.stdout: "testkey": { 2024-04-24T08:31:53.468 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testkey", 2024-04-24T08:31:53.468 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.468 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.468 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.468 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.468 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.468 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: "testlkey": { 2024-04-24T08:31:53.469 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testlkey", 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.470 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: "testnewline": { 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testnewline", 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.471 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.472 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snap_schedule", 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.473 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_m_granularity": { 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_m_granularity", 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.474 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "dump_on_update": { 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dump_on_update", 2024-04-24T08:31:53.475 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.476 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.476 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.476 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.476 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.476 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.476 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.476 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "dump database to debug log on update", 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.477 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.478 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.480 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.481 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.482 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.483 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.484 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "stats", 2024-04-24T08:31:53.485 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.486 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.487 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.488 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.489 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.490 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.490 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.490 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.490 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.490 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.490 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.490 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.491 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.491 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.491 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.491 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.491 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.491 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.492 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.493 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.494 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "status", 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.495 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.496 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.497 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.498 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.499 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.500 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.501 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.502 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.502 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.502 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.502 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.502 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.502 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.503 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "telegraf", 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: "address": { 2024-04-24T08:31:53.504 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "address", 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.505 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15", 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.507 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.508 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.509 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.510 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.511 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.512 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.513 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.514 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.515 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.516 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "telemetry", 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_basic": { 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_basic", 2024-04-24T08:31:53.517 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T08:31:53.518 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_crash": { 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_crash", 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.519 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_device": { 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_device", 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.521 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_ident": { 2024-04-24T08:31:53.522 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_ident", 2024-04-24T08:31:53.523 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.523 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.523 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.523 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.523 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.523 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.523 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_perf": { 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_perf", 2024-04-24T08:31:53.524 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T08:31:53.525 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: "contact": { 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "contact", 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.527 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.528 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.528 INFO:teuthology.orchestra.run.smithi146.stdout: "description": { 2024-04-24T08:31:53.528 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "description", 2024-04-24T08:31:53.528 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.528 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.528 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.528 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "device_url": { 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_url", 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.530 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": { 2024-04-24T08:31:53.531 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enabled", 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.532 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.533 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "24", 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "8", 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.534 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: "last_opt_revision": { 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "last_opt_revision", 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1", 2024-04-24T08:31:53.535 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "leaderboard": { 2024-04-24T08:31:53.536 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "leaderboard", 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.537 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: "leaderboard_description": { 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "leaderboard_description", 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.538 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.540 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.541 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.542 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.543 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.544 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.545 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.546 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.547 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.547 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.547 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.547 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.547 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.547 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.547 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "organization": { 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "organization", 2024-04-24T08:31:53.548 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.549 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.549 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.549 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.549 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.549 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.549 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: "proxy": { 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "proxy", 2024-04-24T08:31:53.550 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.551 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.551 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.551 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.551 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.551 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.551 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "url": { 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "url", 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.552 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.553 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.553 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T08:31:53.553 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.553 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.553 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.553 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.553 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.554 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.554 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.554 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.554 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.554 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test_orchestrator", 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.555 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.556 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.558 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.558 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.558 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.558 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.558 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.558 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.558 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.559 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.560 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.561 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.562 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.563 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "volumes", 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.564 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.565 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.566 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.567 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.569 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.570 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.570 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.570 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.570 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.570 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.570 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.570 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.571 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.571 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.571 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.571 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.571 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.571 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.571 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.572 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.573 INFO:teuthology.orchestra.run.smithi146.stdout: "max_concurrent_clones": { 2024-04-24T08:31:53.574 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_concurrent_clones", 2024-04-24T08:31:53.574 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.574 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.574 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.574 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "4", 2024-04-24T08:31:53.574 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.574 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: "periodic_async_work": { 2024-04-24T08:31:53.575 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "periodic_async_work", 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Periodically check for async work", 2024-04-24T08:31:53.576 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: "snapshot_clone_delay": { 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snapshot_clone_delay", 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.577 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.578 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: "snapshot_clone_no_wait": { 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.579 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.580 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix", 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: "discovery_interval": { 2024-04-24T08:31:53.581 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "discovery_interval", 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "100", 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.582 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "identifier": { 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "identifier", 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.583 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.584 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.585 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.586 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.587 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.588 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.589 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.591 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.592 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.593 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.594 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.594 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.594 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.594 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.594 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.594 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.594 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_host": { 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_host", 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.595 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.596 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_port": { 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_port", 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10051", 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.597 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_sender": { 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_sender", 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.598 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.599 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.599 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T08:31:53.599 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.599 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.599 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.599 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.599 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.600 INFO:teuthology.orchestra.run.smithi146.stdout: "gid": 25198, 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "z", 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: "mgr_features": 4540701547738038271, 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: "available_modules": [ 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "alerts", 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.601 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.602 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.603 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.610 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.611 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_destination": { 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_destination", 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.612 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Email address to send alerts to", 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.613 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_from_name": { 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_from_name", 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "Ceph", 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.614 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Email From: name", 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_host": { 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_host", 2024-04-24T08:31:53.615 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP server", 2024-04-24T08:31:53.616 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_password": { 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_password", 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.617 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.618 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.618 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.618 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.618 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.618 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Password to authenticate with", 2024-04-24T08:31:53.618 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.618 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_port": { 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_port", 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "465", 2024-04-24T08:31:53.619 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.620 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.620 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.620 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP port", 2024-04-24T08:31:53.620 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.620 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.620 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_sender": { 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_sender", 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.621 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP envelope sender", 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_ssl": { 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_ssl", 2024-04-24T08:31:53.622 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.623 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.623 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.623 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.623 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.623 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.623 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.623 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_user": { 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_user", 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.624 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "User to authenticate as", 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.625 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "balancer", 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.626 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "active": { 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "active", 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.627 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: "begin_time": { 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "begin_time", 2024-04-24T08:31:53.628 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0000", 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T08:31:53.629 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: "begin_weekday": { 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "begin_weekday", 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.630 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0", 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "6", 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.631 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_max_iterations": { 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_max_iterations", 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "25", 2024-04-24T08:31:53.632 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "250", 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.633 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_metrics": { 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_metrics", 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.634 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_step": { 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_step", 2024-04-24T08:31:53.635 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0.5", 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0.001", 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "0.999", 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "aggressiveness of optimization", 2024-04-24T08:31:53.636 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "end_time": { 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "end_time", 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.637 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "2359", 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "end_weekday": { 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "end_weekday", 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0", 2024-04-24T08:31:53.639 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "6", 2024-04-24T08:31:53.640 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.640 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T08:31:53.640 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:31:53.640 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.640 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.640 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.641 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.642 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.642 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.642 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.642 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.642 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.642 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.642 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.643 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.644 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.645 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.646 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.647 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: "min_score": { 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "min_score", 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:53.649 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.650 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": { 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mode", 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "upmap", 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.651 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-compat", 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: "none", 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: "read", 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap", 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap-read" 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Balancer mode", 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.652 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_ids": { 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pool_ids", 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.653 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.655 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:53.655 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:53.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.655 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.655 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:53.655 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap_max_deviation": { 2024-04-24T08:31:53.656 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "upmap_max_deviation", 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.657 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap_max_optimizations": { 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "upmap_max_optimizations", 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:53.658 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.659 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cephadm", 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.660 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_down_multiplier": { 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_down_multiplier", 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3.0", 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.661 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T08:31:53.662 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.662 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.662 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.662 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_refresh_rate": { 2024-04-24T08:31:53.662 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_refresh_rate", 2024-04-24T08:31:53.662 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "20", 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T08:31:53.663 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_starting_port": { 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_starting_port", 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.664 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "4721", 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.665 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.666 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_ptrace": { 2024-04-24T08:31:53.666 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_ptrace", 2024-04-24T08:31:53.666 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.666 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.666 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.666 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.666 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.667 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.667 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.667 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T08:31:53.667 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T08:31:53.667 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.667 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "autotune_interval": { 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "autotune_interval", 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.668 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "autotune_memory_target_ratio": { 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T08:31:53.669 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:53.670 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.670 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.670 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0.7", 2024-04-24T08:31:53.670 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.670 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.670 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.670 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm_log_destination": { 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cephadm_log_destination", 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.671 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: "file", 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: "file,syslog", 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: "syslog" 2024-04-24T08:31:53.672 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: "cgroups_split": { 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cgroups_split", 2024-04-24T08:31:53.673 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T08:31:53.674 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: "config_checks_enabled": { 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_checks_enabled", 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.675 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.676 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: "config_dashboard": { 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_dashboard", 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.677 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.678 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_alertmanager": { 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_alertmanager", 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.679 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_base": { 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_base", 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.680 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Container image name, without the tag", 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.681 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_elasticsearch": { 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_elasticsearch", 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T08:31:53.682 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "elasticsearch container image", 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.683 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_grafana": { 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_grafana", 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.684 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:53.685 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.685 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.685 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.685 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_haproxy": { 2024-04-24T08:31:53.685 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_haproxy", 2024-04-24T08:31:53.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "HAproxy container image", 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_agent": { 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_agent", 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.687 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger agent container image", 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_collector": { 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_collector", 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.689 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger collector container image", 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_query": { 2024-04-24T08:31:53.690 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_query", 2024-04-24T08:31:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T08:31:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.691 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger query container image", 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_keepalived": { 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_keepalived", 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Keepalived container image", 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.693 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_loki": { 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_loki", 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T08:31:53.694 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Loki container image", 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.695 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_node_exporter": { 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_node_exporter", 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.696 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_nvmeof": { 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_nvmeof", 2024-04-24T08:31:53.697 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Nvme-of container image", 2024-04-24T08:31:53.698 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_prometheus": { 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_prometheus", 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.699 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_promtail": { 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_promtail", 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.701 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Promtail container image", 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_samba": { 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_samba", 2024-04-24T08:31:53.702 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Samba/SMB container image", 2024-04-24T08:31:53.703 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_snmp_gateway": { 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_snmp_gateway", 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.704 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SNMP Gateway container image", 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.705 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: "container_init": { 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_init", 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.706 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.707 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.707 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.707 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T08:31:53.707 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.707 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.707 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "daemon_cache_timeout": { 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "daemon_cache_timeout", 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.708 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: "default_cephadm_command_timeout": { 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T08:31:53.709 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "900", 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T08:31:53.710 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: "default_registry": { 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default_registry", 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.711 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io", 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.712 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "device_cache_timeout": { 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_cache_timeout", 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1800", 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.713 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache device inventory", 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: "device_enhanced_scan": { 2024-04-24T08:31:53.714 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_enhanced_scan", 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.715 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: "facts_cache_timeout": { 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "facts_cache_timeout", 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.716 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache host facts data", 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.717 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: "grafana_dashboards_path": { 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "grafana_dashboards_path", 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T08:31:53.718 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.719 INFO:teuthology.orchestra.run.smithi146.stdout: "host_check_interval": { 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "host_check_interval", 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.720 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to perform a host check", 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "hw_monitoring": { 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "hw_monitoring", 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.721 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: "inventory_list_all": { 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "inventory_list_all", 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.723 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.724 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.724 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.724 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.724 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T08:31:53.724 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.724 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.724 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.725 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.726 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: "log_refresh_metadata": { 2024-04-24T08:31:53.727 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_refresh_metadata", 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T08:31:53.728 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.729 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.730 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.731 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.732 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.733 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.734 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.736 INFO:teuthology.orchestra.run.smithi146.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "*", 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.737 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "max_count_per_host": { 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_count_per_host", 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.738 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "max number of daemons per service per host", 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.739 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd_draining_count": { 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_osd_draining_count", 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.740 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.741 INFO:teuthology.orchestra.run.smithi146.stdout: "migration_current": { 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "migration_current", 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.742 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "internal - do not modify", 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": { 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mode", 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.743 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "root", 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm-package", 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "root" 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.744 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: "oob_default_addr": { 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "oob_default_addr", 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.745 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "169.254.1.1", 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.746 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: "prometheus_alerts_path": { 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prometheus_alerts_path", 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.747 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.748 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_insecure": { 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_insecure", 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.749 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_password": { 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_password", 2024-04-24T08:31:53.750 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.751 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.751 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.751 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.751 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.751 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.751 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.751 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T08:31:53.753 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.753 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.753 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.753 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.753 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_url": { 2024-04-24T08:31:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_url", 2024-04-24T08:31:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.754 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_username": { 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_username", 2024-04-24T08:31:53.755 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T08:31:53.756 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: "secure_monitoring_stack": { 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "secure_monitoring_stack", 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.757 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.758 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: "service_discovery_port": { 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "service_discovery_port", 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8765", 2024-04-24T08:31:53.759 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "cephadm service discovery port", 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.760 INFO:teuthology.orchestra.run.smithi146.stdout: "ssh_config_file": { 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssh_config_file", 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.761 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: "use_agent": { 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "use_agent", 2024-04-24T08:31:53.762 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T08:31:53.763 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "use_repo_digest": { 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "use_repo_digest", 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.764 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.765 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_failed_host_check": { 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_failed_host_check", 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.766 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_stray_daemons": { 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_stray_daemons", 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.767 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.768 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.769 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_stray_hosts": { 2024-04-24T08:31:53.769 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_stray_hosts", 2024-04-24T08:31:53.769 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.769 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.769 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.769 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.770 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crash", 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.771 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.772 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.773 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.774 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.775 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.776 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.777 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.778 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.779 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: "retain_interval": { 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "retain_interval", 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.780 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "31536000", 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.781 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_recent_interval": { 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_recent_interval", 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1209600", 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.782 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.783 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dashboard", 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.786 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T08:31:53.787 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.788 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "AUDIT_API_ENABLED": { 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.789 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.790 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.793 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.794 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.796 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T08:31:53.797 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.798 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.798 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.798 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.798 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.798 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.798 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.798 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.799 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.800 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.800 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.800 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.800 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.800 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.800 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.800 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T08:31:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T08:31:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.801 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.802 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T08:31:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T08:31:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.803 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.805 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T08:31:53.806 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.807 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.808 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.810 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_URL": { 2024-04-24T08:31:53.811 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_URL", 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.812 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.813 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.813 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.813 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.813 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T08:31:53.813 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T08:31:53.813 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.814 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.815 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.816 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.817 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T08:31:53.818 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.819 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.820 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "[]", 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.822 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "MULTICLUSTER_CONFIG": { 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.823 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "{}", 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.824 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.826 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.827 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.828 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.829 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.830 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.830 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T08:31:53.830 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T08:31:53.830 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.830 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.830 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.830 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.832 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.833 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.835 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.835 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.835 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T08:31:53.835 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T08:31:53.835 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.835 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.835 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.836 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.837 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T08:31:53.838 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T08:31:53.839 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.839 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.839 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.839 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.839 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.839 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.840 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.841 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T08:31:53.842 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.843 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.844 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.845 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8", 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.846 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "45", 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.847 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T08:31:53.848 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.849 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.850 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.851 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:53.851 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.851 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.851 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.851 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.851 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.851 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.852 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.854 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "UNSAFE_TLS_v1_2": { 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-24T08:31:53.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.856 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.857 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.859 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T08:31:53.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "cross_origin_url": { 2024-04-24T08:31:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cross_origin_url", 2024-04-24T08:31:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.862 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.863 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "crt_file": { 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crt_file", 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.864 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "debug": { 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "debug", 2024-04-24T08:31:53.865 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable/disable debug options", 2024-04-24T08:31:53.866 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "jwt_token_ttl": { 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "jwt_token_ttl", 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.867 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "28800", 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.868 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "key_file": { 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "key_file", 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.869 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.870 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.871 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.875 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.876 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.878 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "motd": { 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "motd", 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.879 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "The message of the day", 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_resolve_ip_addr": { 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.881 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:53.882 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "::", 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.883 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.884 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8080", 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.885 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl": { 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl", 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.886 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.887 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl_server_port": { 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl_server_port", 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8443", 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.888 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_behaviour": { 2024-04-24T08:31:53.889 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_behaviour", 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "redirect", 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.890 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect" 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_error_status_code": { 2024-04-24T08:31:53.891 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_error_status_code", 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "500", 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "400", 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "599", 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.892 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "url_prefix": { 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "url_prefix", 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.893 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.894 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "devicehealth", 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "enable_monitoring": { 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enable_monitoring", 2024-04-24T08:31:53.895 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "monitor device health metrics", 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.896 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.897 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.898 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.899 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.900 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.901 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.902 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.903 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.904 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "mark_out_threshold": { 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mark_out_threshold", 2024-04-24T08:31:53.905 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "2419200", 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T08:31:53.906 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": { 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pool_name", 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "device_health_metrics", 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.908 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "retention_period": { 2024-04-24T08:31:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "retention_period", 2024-04-24T08:31:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15552000", 2024-04-24T08:31:53.909 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long to retain device health metrics", 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.910 INFO:teuthology.orchestra.run.smithi146.stdout: "scrape_frequency": { 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "scrape_frequency", 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "86400", 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.911 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "self_heal": { 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "self_heal", 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.912 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T08:31:53.913 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.914 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.915 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_threshold": { 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_threshold", 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "7257600", 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.916 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.917 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "diskprediction_local", 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.918 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.919 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.920 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.921 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.922 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.923 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.924 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.925 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.926 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: "predict_interval": { 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "predict_interval", 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "86400", 2024-04-24T08:31:53.927 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.928 INFO:teuthology.orchestra.run.smithi146.stdout: "predictor_model": { 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "predictor_model", 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "prophetstor", 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.929 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.930 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.931 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "influx", 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": false, 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "influxdb python module not found", 2024-04-24T08:31:53.932 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "batch_size": { 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "batch_size", 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5000", 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.933 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "database": { 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "database", 2024-04-24T08:31:53.934 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "ceph", 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.935 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": { 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "hostname", 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.936 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB server hostname", 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.937 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "30", 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "5", 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.938 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T08:31:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.939 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.940 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.941 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.942 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.944 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.945 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.946 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "password": { 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "password", 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.948 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "password of InfluxDB server user", 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "port": { 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "port", 2024-04-24T08:31:53.949 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8086", 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB server port", 2024-04-24T08:31:53.950 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl": { 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl", 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.951 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "false", 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.952 INFO:teuthology.orchestra.run.smithi146.stdout: "threads": { 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "threads", 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "32", 2024-04-24T08:31:53.953 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "username": { 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "username", 2024-04-24T08:31:53.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "username of InfluxDB server user", 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.955 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "verify_ssl": { 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "verify_ssl", 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "true", 2024-04-24T08:31:53.956 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "insights", 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.958 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.959 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.960 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.962 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.963 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.964 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.965 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.966 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.967 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "iostat", 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.968 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.971 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.972 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.973 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.974 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.975 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.976 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "k8sevents", 2024-04-24T08:31:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.977 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_event_retention_days": { 2024-04-24T08:31:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ceph_event_retention_days", 2024-04-24T08:31:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "7", 2024-04-24T08:31:53.978 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T08:31:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "config_check_secs": { 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_check_secs", 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "10", 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.980 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.981 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.982 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.983 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.984 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.987 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:53.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.989 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:53.990 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "localpool", 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "failure_domain": { 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "failure_domain", 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.991 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "host", 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "failure domain for any created local pool", 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.992 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.993 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.994 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.995 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:53.996 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:53.997 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:53.998 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:53.999 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.000 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": { 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "min_size", 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.001 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default min_size for any created local pool", 2024-04-24T08:31:54.002 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "num_rep": { 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "num_rep", 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.003 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3", 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default replica count for any created local pool", 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.004 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": { 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pg_num", 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "128", 2024-04-24T08:31:54.005 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.006 INFO:teuthology.orchestra.run.smithi146.stdout: "prefix": { 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prefix", 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.007 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "name prefix for any created local pool", 2024-04-24T08:31:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "subtree": { 2024-04-24T08:31:54.008 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "subtree", 2024-04-24T08:31:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "rack", 2024-04-24T08:31:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.009 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.010 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mds_autoscaler", 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.011 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.012 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.013 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.014 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.015 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.016 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.017 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.018 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mirroring", 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.021 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.022 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.023 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.025 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.026 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.026 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.026 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.026 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.026 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.026 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.026 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.027 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.028 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "nfs", 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.029 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.030 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.031 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.033 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.034 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.035 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.037 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.038 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "orchestrator", 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "fail_fs": { 2024-04-24T08:31:54.039 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "fail_fs", 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.040 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.041 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.042 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.043 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.044 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.045 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.046 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.047 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.048 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.049 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator": { 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "orchestrator", 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.050 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm", 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "rook", 2024-04-24T08:31:54.051 INFO:teuthology.orchestra.run.smithi146.stdout: "test_orchestrator" 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Orchestrator backend", 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.052 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "osd_perf_query", 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.053 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.054 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.055 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.056 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.057 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.058 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.059 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.060 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "osd_support", 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.062 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.063 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.064 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.065 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.066 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.067 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.068 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.069 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pg_autoscaler", 2024-04-24T08:31:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.071 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.072 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.073 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.074 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.075 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.076 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.077 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.078 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.079 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.080 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.081 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "threshold": { 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "threshold", 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3.0", 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1.0", 2024-04-24T08:31:54.082 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "scaling threshold", 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "progress", 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_pg_recovery_event": { 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_pg_recovery_event", 2024-04-24T08:31:54.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T08:31:54.085 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": { 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enabled", 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.086 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.087 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.088 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.089 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.090 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.090 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.090 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.090 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.090 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.090 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.090 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.092 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.093 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.094 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.095 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "max_completed_events": { 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_completed_events", 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.096 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "50", 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "number of past completed events to remember", 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.097 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.098 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long the module is going to sleep", 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.099 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prometheus", 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.100 INFO:teuthology.orchestra.run.smithi146.stdout: "cache": { 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cache", 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.101 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: "exclude_perf_counters": { 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "exclude_perf_counters", 2024-04-24T08:31:54.102 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T08:31:54.103 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.104 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.105 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.106 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.106 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.106 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.106 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.106 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.106 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.106 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.107 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.108 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.109 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.110 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.111 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.112 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_stats_pools": { 2024-04-24T08:31:54.113 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_stats_pools", 2024-04-24T08:31:54.113 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.113 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.113 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.113 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.113 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.113 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T08:31:54.114 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "300", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.115 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "scrape_interval": { 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "scrape_interval", 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15.0", 2024-04-24T08:31:54.116 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:54.117 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "::", 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.118 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.119 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.120 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "9283", 2024-04-24T08:31:54.120 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.120 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.120 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.120 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T08:31:54.120 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.120 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: "stale_cache_strategy": { 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "stale_cache_strategy", 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.121 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "log", 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.122 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_behaviour": { 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_behaviour", 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "default", 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.123 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: "default", 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: "error" 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_error_status_code": { 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_error_status_code", 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "500", 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "400", 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "599", 2024-04-24T08:31:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.126 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_support", 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.127 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.128 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.129 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.130 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.130 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.130 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.130 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.130 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.130 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.130 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.131 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.132 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.133 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.134 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: "max_concurrent_snap_create": { 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_concurrent_snap_create", 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.135 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.136 INFO:teuthology.orchestra.run.smithi146.stdout: "mirror_snapshot_schedule": { 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.137 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: "trash_purge_schedule": { 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "trash_purge_schedule", 2024-04-24T08:31:54.138 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.139 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "restful", 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.140 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "enable_auth": { 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enable_auth", 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.141 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.142 INFO:teuthology.orchestra.run.smithi146.stdout: "key_file": { 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "key_file", 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.143 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.144 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.145 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.146 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.147 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.148 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.149 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:54.152 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.153 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.154 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.155 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rgw", 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.156 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.157 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.158 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.158 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.158 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.158 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.158 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.158 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.158 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.159 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.160 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.161 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.162 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.163 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.164 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.165 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rook", 2024-04-24T08:31:54.165 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.165 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.165 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.165 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.165 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.165 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.166 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.166 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.166 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.166 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.166 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.166 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.166 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.167 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.168 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.169 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.170 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.171 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.173 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_class": { 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "storage_class", 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "local", 2024-04-24T08:31:54.174 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.175 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.175 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.175 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T08:31:54.175 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.175 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.175 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.175 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "selftest", 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.176 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.177 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.178 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.179 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.180 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.181 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.182 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.182 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.182 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.182 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.182 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.182 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.182 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.183 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "roption1": { 2024-04-24T08:31:54.184 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "roption1", 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.185 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: "roption2": { 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "roption2", 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.186 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "xyz", 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.187 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption1": { 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption1", 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.188 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.189 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.189 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.189 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.189 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.189 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.189 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.189 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption2": { 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption2", 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.190 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.191 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.191 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.191 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.191 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.191 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.191 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption3": { 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption3", 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.192 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption4": { 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption4", 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.193 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.194 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption5": { 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption5", 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.195 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.196 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption6": { 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption6", 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.197 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption7": { 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption7", 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.198 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "42", 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.199 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "testkey": { 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testkey", 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.201 INFO:teuthology.orchestra.run.smithi146.stdout: "testlkey": { 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testlkey", 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: "testnewline": { 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testnewline", 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.203 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snap_schedule", 2024-04-24T08:31:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_m_granularity": { 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_m_granularity", 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T08:31:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: "dump_on_update": { 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dump_on_update", 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "dump database to debug log on update", 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.218 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "stats", 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.219 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.220 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.221 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.222 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.223 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.224 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.225 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.226 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.226 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.226 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.226 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.226 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.226 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.226 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.227 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.228 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "status", 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.229 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.230 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.230 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.230 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.230 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.230 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.230 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.230 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.231 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.232 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.233 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.234 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.235 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.236 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.237 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.237 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.237 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.237 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.237 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.237 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.237 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "telegraf", 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.238 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "address": { 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "address", 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.239 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:54.240 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15", 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.241 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.242 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.243 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.244 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.245 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.246 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.246 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.246 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.246 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.246 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.246 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.249 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.250 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.250 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.250 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.250 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.250 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.250 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.250 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "telemetry", 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.251 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_basic": { 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_basic", 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.252 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_crash": { 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_crash", 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.253 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.254 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_device": { 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_device", 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.255 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.256 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_ident": { 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_ident", 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.257 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_perf": { 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_perf", 2024-04-24T08:31:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T08:31:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "contact": { 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "contact", 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.260 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.261 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "description": { 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "description", 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.262 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: "device_url": { 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_url", 2024-04-24T08:31:54.263 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.264 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": { 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enabled", 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.265 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "24", 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "8", 2024-04-24T08:31:54.267 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "last_opt_revision": { 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "last_opt_revision", 2024-04-24T08:31:54.268 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1", 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.269 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.270 INFO:teuthology.orchestra.run.smithi146.stdout: "leaderboard": { 2024-04-24T08:31:54.270 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "leaderboard", 2024-04-24T08:31:54.270 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.270 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.270 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.270 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.271 INFO:teuthology.orchestra.run.smithi146.stdout: "leaderboard_description": { 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "leaderboard_description", 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.272 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.273 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.274 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.275 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.275 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.275 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.275 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.275 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.275 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.275 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.276 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.277 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.278 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.279 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.280 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "organization": { 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "organization", 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.282 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "proxy": { 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "proxy", 2024-04-24T08:31:54.283 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.284 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "url": { 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "url", 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T08:31:54.285 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.286 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test_orchestrator", 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.287 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.288 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.289 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.290 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.291 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.292 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.293 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.294 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.294 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.294 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.294 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.294 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.294 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.294 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.295 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "volumes", 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.296 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.297 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.299 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.300 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.301 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.302 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.303 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "max_concurrent_clones": { 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_concurrent_clones", 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "4", 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.305 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: "periodic_async_work": { 2024-04-24T08:31:54.306 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "periodic_async_work", 2024-04-24T08:31:54.307 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.307 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.307 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.307 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.307 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.307 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.307 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Periodically check for async work", 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: "snapshot_clone_delay": { 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snapshot_clone_delay", 2024-04-24T08:31:54.308 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.309 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: "snapshot_clone_no_wait": { 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.310 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.311 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix", 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: "discovery_interval": { 2024-04-24T08:31:54.312 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "discovery_interval", 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "100", 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.313 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: "identifier": { 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "identifier", 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.314 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.315 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:54.316 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.317 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.318 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.319 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.320 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_host": { 2024-04-24T08:31:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_host", 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_port": { 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_port", 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10051", 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_sender": { 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_sender", 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: "modules": [ 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm", 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: "dashboard", 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: "iostat", 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: "nfs", 2024-04-24T08:31:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: "restful", 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: "snap_schedule" 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: "available_modules": [ 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "alerts", 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.337 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.338 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.339 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.340 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.341 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.342 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.343 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_destination": { 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_destination", 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.344 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Email address to send alerts to", 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_from_name": { 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_from_name", 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.345 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "Ceph", 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Email From: name", 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.346 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_host": { 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_host", 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.347 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP server", 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_password": { 2024-04-24T08:31:54.348 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_password", 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.349 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Password to authenticate with", 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_port": { 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_port", 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.350 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "465", 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP port", 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.351 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_sender": { 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_sender", 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SMTP envelope sender", 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.353 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_ssl": { 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_ssl", 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.354 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "smtp_user": { 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "smtp_user", 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.355 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "User to authenticate as", 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.356 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "balancer", 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.357 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.358 INFO:teuthology.orchestra.run.smithi146.stdout: "active": { 2024-04-24T08:31:54.358 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "active", 2024-04-24T08:31:54.358 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.358 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.358 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.358 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.358 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.359 INFO:teuthology.orchestra.run.smithi146.stdout: "begin_time": { 2024-04-24T08:31:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "begin_time", 2024-04-24T08:31:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0000", 2024-04-24T08:31:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.360 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.361 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.361 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T08:31:54.361 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:31:54.361 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.361 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.361 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.361 INFO:teuthology.orchestra.run.smithi146.stdout: "begin_weekday": { 2024-04-24T08:31:54.362 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "begin_weekday", 2024-04-24T08:31:54.362 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:54.362 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.362 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.362 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:54.362 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0", 2024-04-24T08:31:54.362 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "6", 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_max_iterations": { 2024-04-24T08:31:54.363 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_max_iterations", 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "25", 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "250", 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.364 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_metrics": { 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_metrics", 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.365 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T08:31:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T08:31:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T08:31:54.366 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: "crush_compat_step": { 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crush_compat_step", 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.367 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0.5", 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0.001", 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "0.999", 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "aggressiveness of optimization", 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.368 INFO:teuthology.orchestra.run.smithi146.stdout: "end_time": { 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "end_time", 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "2359", 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.369 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: "end_weekday": { 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "end_weekday", 2024-04-24T08:31:54.370 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "0", 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "6", 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T08:31:54.371 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.372 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.373 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.373 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.373 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.373 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.373 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.373 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.374 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.375 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.376 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.377 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.377 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.377 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.377 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.377 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.377 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.377 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.378 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.379 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.380 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: "min_score": { 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "min_score", 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.381 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.382 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.383 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.383 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.383 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": { 2024-04-24T08:31:54.383 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mode", 2024-04-24T08:31:54.383 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.383 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.383 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "upmap", 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "crush-compat", 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "none", 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "read", 2024-04-24T08:31:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap", 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap-read" 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Balancer mode", 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.385 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_ids": { 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pool_ids", 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.386 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:54.387 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T08:31:54.388 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap_max_deviation": { 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "upmap_max_deviation", 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.389 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.390 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: "upmap_max_optimizations": { 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "upmap_max_optimizations", 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.391 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.392 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cephadm", 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_down_multiplier": { 2024-04-24T08:31:54.393 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_down_multiplier", 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3.0", 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.394 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T08:31:54.395 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.395 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.395 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.395 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.395 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_refresh_rate": { 2024-04-24T08:31:54.395 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_refresh_rate", 2024-04-24T08:31:54.395 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.396 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.396 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.396 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "20", 2024-04-24T08:31:54.396 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.396 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.396 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.396 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T08:31:54.397 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.397 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.397 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.397 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.397 INFO:teuthology.orchestra.run.smithi146.stdout: "agent_starting_port": { 2024-04-24T08:31:54.397 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "agent_starting_port", 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "4721", 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.398 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_ptrace": { 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_ptrace", 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.399 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T08:31:54.400 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: "autotune_interval": { 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "autotune_interval", 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.401 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.402 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.403 INFO:teuthology.orchestra.run.smithi146.stdout: "autotune_memory_target_ratio": { 2024-04-24T08:31:54.403 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T08:31:54.403 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.403 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.403 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.403 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0.7", 2024-04-24T08:31:54.403 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.404 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm_log_destination": { 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cephadm_log_destination", 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.405 INFO:teuthology.orchestra.run.smithi146.stdout: "file", 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: "file,syslog", 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: "syslog" 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.406 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "cgroups_split": { 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cgroups_split", 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.407 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: "config_checks_enabled": { 2024-04-24T08:31:54.408 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_checks_enabled", 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T08:31:54.409 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: "config_dashboard": { 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_dashboard", 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.410 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.411 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_alertmanager": { 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_alertmanager", 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T08:31:54.412 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.413 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_base": { 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_base", 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.414 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Container image name, without the tag", 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_elasticsearch": { 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_elasticsearch", 2024-04-24T08:31:54.415 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "elasticsearch container image", 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.416 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_grafana": { 2024-04-24T08:31:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_grafana", 2024-04-24T08:31:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.417 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.418 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_haproxy": { 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_haproxy", 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T08:31:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "HAproxy container image", 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.420 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_agent": { 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_agent", 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.421 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger agent container image", 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_collector": { 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_collector", 2024-04-24T08:31:54.422 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger collector container image", 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.423 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_jaeger_query": { 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_jaeger_query", 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.424 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Jaeger query container image", 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.425 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.426 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_keepalived": { 2024-04-24T08:31:54.426 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_keepalived", 2024-04-24T08:31:54.426 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.426 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.426 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.426 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T08:31:54.426 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Keepalived container image", 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_loki": { 2024-04-24T08:31:54.427 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_loki", 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.428 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Loki container image", 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_node_exporter": { 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_node_exporter", 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.429 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.430 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_nvmeof": { 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_nvmeof", 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T08:31:54.431 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Nvme-of container image", 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_prometheus": { 2024-04-24T08:31:54.432 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_prometheus", 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Prometheus container image", 2024-04-24T08:31:54.433 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.434 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.434 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.434 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.434 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_promtail": { 2024-04-24T08:31:54.434 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_promtail", 2024-04-24T08:31:54.434 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.434 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Promtail container image", 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.435 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_samba": { 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_samba", 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.436 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T08:31:54.437 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.437 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.437 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.437 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Samba/SMB container image", 2024-04-24T08:31:54.437 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.437 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.437 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: "container_image_snmp_gateway": { 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_image_snmp_gateway", 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T08:31:54.438 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "SNMP Gateway container image", 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: "container_init": { 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "container_init", 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: "daemon_cache_timeout": { 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "daemon_cache_timeout", 2024-04-24T08:31:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "default_cephadm_command_timeout": { 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "900", 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: "default_registry": { 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "default_registry", 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "docker.io", 2024-04-24T08:31:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "device_cache_timeout": { 2024-04-24T08:31:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_cache_timeout", 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1800", 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache device inventory", 2024-04-24T08:31:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: "device_enhanced_scan": { 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_enhanced_scan", 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "facts_cache_timeout": { 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "facts_cache_timeout", 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "seconds to cache host facts data", 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: "grafana_dashboards_path": { 2024-04-24T08:31:54.451 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "grafana_dashboards_path", 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T08:31:54.452 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: "host_check_interval": { 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "host_check_interval", 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.453 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to perform a host check", 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.454 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: "hw_monitoring": { 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "hw_monitoring", 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.455 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.456 INFO:teuthology.orchestra.run.smithi146.stdout: "inventory_list_all": { 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "inventory_list_all", 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.457 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.458 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.459 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.460 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "log_refresh_metadata": { 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_refresh_metadata", 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.461 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.463 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.464 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.465 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.466 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.467 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.468 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.469 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.470 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "*", 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.471 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "max_count_per_host": { 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_count_per_host", 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.472 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "max number of daemons per service per host", 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "max_osd_draining_count": { 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_osd_draining_count", 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.473 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.474 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: "migration_current": { 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "migration_current", 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.475 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "internal - do not modify", 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.476 INFO:teuthology.orchestra.run.smithi146.stdout: "mode": { 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mode", 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "root", 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.477 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.478 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm-package", 2024-04-24T08:31:54.478 INFO:teuthology.orchestra.run.smithi146.stdout: "root" 2024-04-24T08:31:54.478 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.478 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T08:31:54.478 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.478 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.478 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.479 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.479 INFO:teuthology.orchestra.run.smithi146.stdout: "oob_default_addr": { 2024-04-24T08:31:54.479 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "oob_default_addr", 2024-04-24T08:31:54.479 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.479 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.479 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.479 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "169.254.1.1", 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.480 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "prometheus_alerts_path": { 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prometheus_alerts_path", 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.481 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.482 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.482 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T08:31:54.482 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.482 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.482 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.482 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.482 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_insecure": { 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_insecure", 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.483 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_password": { 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_password", 2024-04-24T08:31:54.484 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.485 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_url": { 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_url", 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.486 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "registry_username": { 2024-04-24T08:31:54.487 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "registry_username", 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.488 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: "secure_monitoring_stack": { 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "secure_monitoring_stack", 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.489 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.490 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "service_discovery_port": { 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "service_discovery_port", 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8765", 2024-04-24T08:31:54.491 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "cephadm service discovery port", 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "ssh_config_file": { 2024-04-24T08:31:54.492 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssh_config_file", 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T08:31:54.493 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: "use_agent": { 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "use_agent", 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.494 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.495 INFO:teuthology.orchestra.run.smithi146.stdout: "use_repo_digest": { 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "use_repo_digest", 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.496 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_failed_host_check": { 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_failed_host_check", 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.497 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.498 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_stray_daemons": { 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_stray_daemons", 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.499 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_on_stray_hosts": { 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_on_stray_hosts", 2024-04-24T08:31:54.500 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T08:31:54.501 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crash", 2024-04-24T08:31:54.502 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.503 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.504 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.505 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.506 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.507 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.508 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.509 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.510 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "retain_interval": { 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "retain_interval", 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.511 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "31536000", 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.512 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_recent_interval": { 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_recent_interval", 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.513 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1209600", 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.514 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dashboard", 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T08:31:54.515 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.516 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.517 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.518 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.519 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: "AUDIT_API_ENABLED": { 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T08:31:54.520 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.521 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.522 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.523 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.524 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T08:31:54.525 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.526 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.527 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.528 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.529 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T08:31:54.530 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.531 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.532 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.533 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T08:31:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T08:31:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.534 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.535 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.536 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T08:31:54.537 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.538 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.539 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.540 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T08:31:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T08:31:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.541 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.542 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_URL": { 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_URL", 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.543 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.544 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.545 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.546 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T08:31:54.547 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.548 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.549 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.550 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.551 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-24T08:31:54.552 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "[]", 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.553 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "MULTICLUSTER_CONFIG": { 2024-04-24T08:31:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-24T08:31:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.554 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "{}", 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.555 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.556 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T08:31:54.557 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T08:31:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.558 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T08:31:54.559 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.560 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.561 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.562 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.563 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.564 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.565 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T08:31:54.566 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T08:31:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.567 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T08:31:54.568 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.569 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.570 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.571 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.572 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T08:31:54.573 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.574 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.575 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.576 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T08:31:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T08:31:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.577 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8", 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.578 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "45", 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.579 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T08:31:54.580 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.581 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.582 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "admin", 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.583 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.584 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T08:31:54.585 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.586 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "UNSAFE_TLS_v1_2": { 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.587 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.588 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.589 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T08:31:54.590 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.591 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.592 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.593 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "cross_origin_url": { 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cross_origin_url", 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.594 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.595 INFO:teuthology.orchestra.run.smithi146.stdout: "crt_file": { 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "crt_file", 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.596 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "debug": { 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "debug", 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.597 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Enable/disable debug options", 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.598 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "jwt_token_ttl": { 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "jwt_token_ttl", 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.599 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "28800", 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.600 INFO:teuthology.orchestra.run.smithi146.stdout: "key_file": { 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "key_file", 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.601 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.602 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.603 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.604 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.605 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.606 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.607 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.608 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.609 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.610 INFO:teuthology.orchestra.run.smithi146.stdout: "motd": { 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "motd", 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.611 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "The message of the day", 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect_resolve_ip_addr": { 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.612 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.613 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.614 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "::", 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8080", 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.616 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl": { 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl", 2024-04-24T08:31:54.617 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.618 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl_server_port": { 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl_server_port", 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.619 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8443", 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.620 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_behaviour": { 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_behaviour", 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "redirect", 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.621 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "redirect" 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.622 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_error_status_code": { 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_error_status_code", 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "500", 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "400", 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "599", 2024-04-24T08:31:54.623 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "url_prefix": { 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "url_prefix", 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.624 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.625 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "devicehealth", 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.626 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "enable_monitoring": { 2024-04-24T08:31:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enable_monitoring", 2024-04-24T08:31:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.627 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "monitor device health metrics", 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.628 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.629 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.630 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.631 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.632 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.633 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.634 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.635 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.636 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "mark_out_threshold": { 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mark_out_threshold", 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.637 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "2419200", 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.638 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "pool_name": { 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pool_name", 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "device_health_metrics", 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.639 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: "retention_period": { 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "retention_period", 2024-04-24T08:31:54.640 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.641 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.641 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.641 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15552000", 2024-04-24T08:31:54.641 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.641 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.641 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.641 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long to retain device health metrics", 2024-04-24T08:31:54.642 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.642 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.642 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.642 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.642 INFO:teuthology.orchestra.run.smithi146.stdout: "scrape_frequency": { 2024-04-24T08:31:54.642 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "scrape_frequency", 2024-04-24T08:31:54.642 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "86400", 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.643 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "self_heal": { 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "self_heal", 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.644 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.645 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.646 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: "warn_threshold": { 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "warn_threshold", 2024-04-24T08:31:54.647 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "7257600", 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T08:31:54.648 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "diskprediction_local", 2024-04-24T08:31:54.649 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.650 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.650 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.650 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.650 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.650 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.650 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.650 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.651 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.652 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.652 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.652 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.652 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.652 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.652 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.653 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.654 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.655 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.656 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.657 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.658 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "predict_interval": { 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "predict_interval", 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "86400", 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.659 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.660 INFO:teuthology.orchestra.run.smithi146.stdout: "predictor_model": { 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "predictor_model", 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "prophetstor", 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.661 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.662 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "600", 2024-04-24T08:31:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.663 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "influx", 2024-04-24T08:31:54.664 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": false, 2024-04-24T08:31:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "influxdb python module not found", 2024-04-24T08:31:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "batch_size": { 2024-04-24T08:31:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "batch_size", 2024-04-24T08:31:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.665 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5000", 2024-04-24T08:31:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T08:31:54.666 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "database": { 2024-04-24T08:31:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "database", 2024-04-24T08:31:54.667 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "ceph", 2024-04-24T08:31:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.668 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T08:31:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": { 2024-04-24T08:31:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "hostname", 2024-04-24T08:31:54.669 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB server hostname", 2024-04-24T08:31:54.670 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.671 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "30", 2024-04-24T08:31:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "5", 2024-04-24T08:31:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T08:31:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.672 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.673 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.674 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.675 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.676 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.677 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.678 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.679 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.680 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.681 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "password": { 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "password", 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.682 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "password of InfluxDB server user", 2024-04-24T08:31:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "port": { 2024-04-24T08:31:54.683 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "port", 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "8086", 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "InfluxDB server port", 2024-04-24T08:31:54.684 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "ssl": { 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ssl", 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.685 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "false", 2024-04-24T08:31:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:31:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.686 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "threads": { 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "threads", 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:54.687 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "32", 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.688 INFO:teuthology.orchestra.run.smithi146.stdout: "username": { 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "username", 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.689 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "username of InfluxDB server user", 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "verify_ssl": { 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "verify_ssl", 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.690 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "true", 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.691 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "insights", 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.692 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.693 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.694 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.695 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.696 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.697 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.698 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.699 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.700 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.701 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "iostat", 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.702 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.703 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.704 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.705 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.706 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.707 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.708 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.709 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.710 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.711 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "k8sevents", 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_event_retention_days": { 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "ceph_event_retention_days", 2024-04-24T08:31:54.712 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "7", 2024-04-24T08:31:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.713 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "config_check_secs": { 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "config_check_secs", 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.714 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "10", 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.715 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.716 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.717 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.718 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.719 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.720 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.721 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.722 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.723 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.724 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "localpool", 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "failure_domain": { 2024-04-24T08:31:54.725 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "failure_domain", 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "host", 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.726 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "failure domain for any created local pool", 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.727 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.728 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.729 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.730 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.731 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.732 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.733 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.734 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "min_size": { 2024-04-24T08:31:54.735 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "min_size", 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.736 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default min_size for any created local pool", 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "num_rep": { 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "num_rep", 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.737 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3", 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default replica count for any created local pool", 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.738 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_num": { 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pg_num", 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "128", 2024-04-24T08:31:54.739 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T08:31:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.740 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "prefix": { 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prefix", 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.741 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "name prefix for any created local pool", 2024-04-24T08:31:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.742 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "subtree": { 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "subtree", 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "rack", 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.743 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.744 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mds_autoscaler", 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.745 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.746 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.747 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.748 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.749 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.750 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.751 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.752 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.753 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mirroring", 2024-04-24T08:31:54.754 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.755 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.756 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.757 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.758 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.759 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.760 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.761 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.762 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.763 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.764 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "nfs", 2024-04-24T08:31:54.764 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.764 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.764 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.764 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.764 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.764 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.765 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.765 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.765 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.765 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.765 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.765 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.765 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.766 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.766 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.766 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.766 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.766 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.766 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.766 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.767 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.768 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.769 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.770 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.771 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.772 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.773 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "orchestrator", 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "fail_fs": { 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "fail_fs", 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.774 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.775 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.776 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.777 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.778 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.779 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.780 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.781 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.782 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.783 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.784 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator": { 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "orchestrator", 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.785 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: "cephadm", 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: "rook", 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: "test_orchestrator" 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Orchestrator backend", 2024-04-24T08:31:54.786 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.787 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.787 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.787 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.787 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.787 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "osd_perf_query", 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.788 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.789 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.790 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.791 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.792 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.793 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.794 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.795 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.795 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.795 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.795 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.795 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.795 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.795 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.796 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "osd_support", 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.798 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.798 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.798 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.798 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.798 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.798 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.798 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.799 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.800 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.801 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.802 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.803 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.804 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.805 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.806 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "pg_autoscaler", 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.807 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.808 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.809 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.810 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.811 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.812 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.813 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.814 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:54.815 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.816 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "threshold": { 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "threshold", 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.817 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "3.0", 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1.0", 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "scaling threshold", 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.818 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "progress", 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.819 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_pg_recovery_event": { 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_pg_recovery_event", 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.820 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": { 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enabled", 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.821 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.822 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.823 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.824 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.825 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.826 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.827 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.828 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.829 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.830 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: "max_completed_events": { 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_completed_events", 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.831 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "50", 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "number of past completed events to remember", 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.832 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "sleep_interval": { 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "sleep_interval", 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "5", 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.833 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "how long the module is going to sleep", 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.834 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "prometheus", 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: "cache": { 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "cache", 2024-04-24T08:31:54.835 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.836 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "exclude_perf_counters": { 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "exclude_perf_counters", 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.837 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.838 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.839 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.839 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.839 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.839 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.839 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.839 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.839 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.840 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.841 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.842 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.843 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.845 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.846 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.847 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_stats_pools": { 2024-04-24T08:31:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_stats_pools", 2024-04-24T08:31:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.848 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T08:31:54.849 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "300", 2024-04-24T08:31:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.850 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: "scrape_interval": { 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "scrape_interval", 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.851 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15.0", 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.852 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "::", 2024-04-24T08:31:54.853 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:54.854 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "9283", 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.855 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "stale_cache_strategy": { 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "stale_cache_strategy", 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.856 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "log", 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.857 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_behaviour": { 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_behaviour", 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.858 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "default", 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "default", 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "error" 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.859 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "standby_error_status_code": { 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "standby_error_status_code", 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.860 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "500", 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "400", 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "599", 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.861 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_support", 2024-04-24T08:31:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.862 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.863 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.864 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.865 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.866 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.867 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.868 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.869 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.870 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "max_concurrent_snap_create": { 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_concurrent_snap_create", 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.871 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10", 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.872 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "mirror_snapshot_schedule": { 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.873 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "trash_purge_schedule": { 2024-04-24T08:31:54.874 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "trash_purge_schedule", 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.875 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.876 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "restful", 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "enable_auth": { 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enable_auth", 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.877 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.878 INFO:teuthology.orchestra.run.smithi146.stdout: "key_file": { 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "key_file", 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.879 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.880 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.881 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.882 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.883 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.884 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.885 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.886 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.887 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.888 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "server_addr": { 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_addr", 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.889 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.890 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "server_port": { 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "server_port", 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.891 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.892 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rgw", 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.893 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.894 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.895 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.896 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.897 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.898 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.899 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.900 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.901 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rook", 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.902 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.903 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.904 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.905 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.906 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.907 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.908 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.909 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.910 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "storage_class": { 2024-04-24T08:31:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "storage_class", 2024-04-24T08:31:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "local", 2024-04-24T08:31:54.911 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.912 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "selftest", 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.913 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.914 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.915 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.916 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.917 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.918 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.919 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.920 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.921 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "roption1": { 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "roption1", 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.922 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.923 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "roption2": { 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "roption2", 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "xyz", 2024-04-24T08:31:54.924 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption1": { 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption1", 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.926 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption2": { 2024-04-24T08:31:54.927 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption2", 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.928 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption3": { 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption3", 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "float", 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.929 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.930 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption4": { 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption4", 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.931 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.932 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption5": { 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption5", 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.933 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption6": { 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption6", 2024-04-24T08:31:54.934 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.935 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "rwoption7": { 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rwoption7", 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.936 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "1", 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "42", 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.937 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "testkey": { 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testkey", 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.938 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "testlkey": { 2024-04-24T08:31:54.939 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testlkey", 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.940 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "testnewline": { 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "testnewline", 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.941 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.942 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snap_schedule", 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.943 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "allow_m_granularity": { 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "allow_m_granularity", 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.944 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "dump_on_update": { 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "dump_on_update", 2024-04-24T08:31:54.945 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "dump database to debug log on update", 2024-04-24T08:31:54.946 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.947 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.948 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.950 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.951 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.952 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.953 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.954 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.955 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "stats", 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.956 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.957 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.958 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.959 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.961 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.962 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.963 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.964 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "status", 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.965 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.966 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.966 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.966 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.966 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.966 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.966 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.966 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.967 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.968 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.968 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.968 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.968 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.968 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.968 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.969 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.970 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.971 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.972 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.973 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.974 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.974 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.974 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.974 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.974 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.974 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.974 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "telegraf", 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.975 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "address": { 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "address", 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.976 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:54.977 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "15", 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.979 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.980 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:54.981 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:54.982 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.982 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.982 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.982 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.982 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.982 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.982 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.983 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:54.984 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:54.985 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.986 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "telemetry", 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:54.987 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_basic": { 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_basic", 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.988 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_crash": { 2024-04-24T08:31:54.989 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_crash", 2024-04-24T08:31:54.990 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.990 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.990 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.990 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.990 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.990 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.990 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_device": { 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_device", 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.991 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.992 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_ident": { 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_ident", 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.993 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "channel_perf": { 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "channel_perf", 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:54.994 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.995 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "contact": { 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "contact", 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.996 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: "description": { 2024-04-24T08:31:54.997 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "description", 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:54.998 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: "device_url": { 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "device_url", 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:54.999 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.000 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: "enabled": { 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "enabled", 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.001 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.002 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.002 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.002 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.002 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.002 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.002 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.002 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "24", 2024-04-24T08:31:55.003 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "8", 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "last_opt_revision": { 2024-04-24T08:31:55.004 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "last_opt_revision", 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "1", 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.005 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: "leaderboard": { 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "leaderboard", 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.006 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.007 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: "leaderboard_description": { 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "leaderboard_description", 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.008 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.009 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.010 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.011 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:55.012 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.013 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.013 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.013 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.013 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.013 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.013 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.013 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.014 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.014 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.014 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.014 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.014 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:55.014 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:55.014 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.015 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.016 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.017 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.018 INFO:teuthology.orchestra.run.smithi146.stdout: "organization": { 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "organization", 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.019 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: "proxy": { 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "proxy", 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.020 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.021 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.022 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.022 INFO:teuthology.orchestra.run.smithi146.stdout: "url": { 2024-04-24T08:31:55.022 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "url", 2024-04-24T08:31:55.022 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.022 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.022 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.022 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.023 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.024 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.024 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.024 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.024 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "test_orchestrator", 2024-04-24T08:31:55.024 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:55.024 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:55.024 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.025 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.026 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:55.027 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.028 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:55.029 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.030 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.031 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.031 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.031 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.031 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.031 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.031 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.031 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.032 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.033 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.033 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.033 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.033 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.033 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "volumes", 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.034 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.035 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.036 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.037 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.038 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.038 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.038 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.038 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.038 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.038 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:55.038 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.039 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.040 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.041 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.042 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "max_concurrent_clones": { 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "max_concurrent_clones", 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "4", 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.043 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "periodic_async_work": { 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "periodic_async_work", 2024-04-24T08:31:55.044 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Periodically check for async work", 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.045 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: "snapshot_clone_delay": { 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snapshot_clone_delay", 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.046 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "0", 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.047 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "snapshot_clone_no_wait": { 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "True", 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.048 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.049 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T08:31:55.049 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.049 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.049 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.049 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.049 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix", 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: "can_run": true, 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: "error_string": "", 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: "module_options": { 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: "discovery_interval": { 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "discovery_interval", 2024-04-24T08:31:55.050 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "uint", 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "100", 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.051 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.052 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.052 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.052 INFO:teuthology.orchestra.run.smithi146.stdout: "identifier": { 2024-04-24T08:31:55.052 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "identifier", 2024-04-24T08:31:55.052 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.052 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.052 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.053 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.054 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.054 INFO:teuthology.orchestra.run.smithi146.stdout: "interval": { 2024-04-24T08:31:55.054 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "interval", 2024-04-24T08:31:55.054 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "secs", 2024-04-24T08:31:55.054 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.054 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "60", 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.055 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: "log_level": { 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_level", 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.056 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.057 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster": { 2024-04-24T08:31:55.058 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster", 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.059 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_cluster_level": { 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_cluster_level", 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.060 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "info", 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [ 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "", 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "critical", 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "debug", 2024-04-24T08:31:55.061 INFO:teuthology.orchestra.run.smithi146.stdout: "error", 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: "info", 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: "warning" 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.062 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "log_to_file": { 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "log_to_file", 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "bool", 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 1, 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "False", 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.063 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_host": { 2024-04-24T08:31:55.064 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_host", 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "", 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.065 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_port": { 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_port", 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "int", 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.066 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "10051", 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.067 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.068 INFO:teuthology.orchestra.run.smithi146.stdout: "zabbix_sender": { 2024-04-24T08:31:55.068 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "zabbix_sender", 2024-04-24T08:31:55.068 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "str", 2024-04-24T08:31:55.068 INFO:teuthology.orchestra.run.smithi146.stdout: "level": "advanced", 2024-04-24T08:31:55.068 INFO:teuthology.orchestra.run.smithi146.stdout: "flags": 0, 2024-04-24T08:31:55.068 INFO:teuthology.orchestra.run.smithi146.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T08:31:55.068 INFO:teuthology.orchestra.run.smithi146.stdout: "min": "", 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: "max": "", 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: "enum_allowed": [], 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: "desc": "", 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: "long_desc": "", 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: "tags": [], 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: "see_also": [] 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.069 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: "services": { 2024-04-24T08:31:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: "dashboard": "https://172.21.15.146:8443/" 2024-04-24T08:31:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: "always_on_modules": { 2024-04-24T08:31:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: "octopus": [ 2024-04-24T08:31:55.071 INFO:teuthology.orchestra.run.smithi146.stdout: "balancer", 2024-04-24T08:31:55.071 INFO:teuthology.orchestra.run.smithi146.stdout: "crash", 2024-04-24T08:31:55.071 INFO:teuthology.orchestra.run.smithi146.stdout: "devicehealth", 2024-04-24T08:31:55.071 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator", 2024-04-24T08:31:55.071 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscaler", 2024-04-24T08:31:55.071 INFO:teuthology.orchestra.run.smithi146.stdout: "progress", 2024-04-24T08:31:55.071 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_support", 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "status", 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "telemetry", 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "volumes" 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "pacific": [ 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "balancer", 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "crash", 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "devicehealth", 2024-04-24T08:31:55.072 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator", 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscaler", 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: "progress", 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_support", 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: "status", 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: "telemetry", 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: "volumes" 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.073 INFO:teuthology.orchestra.run.smithi146.stdout: "quincy": [ 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "balancer", 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "crash", 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "devicehealth", 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator", 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscaler", 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "progress", 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_support", 2024-04-24T08:31:55.074 INFO:teuthology.orchestra.run.smithi146.stdout: "status", 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: "telemetry", 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: "volumes" 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: "reef": [ 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: "balancer", 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: "crash", 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: "devicehealth", 2024-04-24T08:31:55.075 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator", 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscaler", 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "progress", 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_support", 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "status", 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "telemetry", 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "volumes" 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: ], 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "squid": [ 2024-04-24T08:31:55.076 INFO:teuthology.orchestra.run.smithi146.stdout: "balancer", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "crash", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "devicehealth", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "orchestrator", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "pg_autoscaler", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "progress", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "rbd_support", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "status", 2024-04-24T08:31:55.077 INFO:teuthology.orchestra.run.smithi146.stdout: "telemetry", 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: "volumes" 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: "last_failure_osd_epoch": 91, 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: "active_clients": [ 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "devicehealth", 2024-04-24T08:31:55.078 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:0", 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 963440230 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.079 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "libcephsqlite", 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:0", 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3033190584 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.080 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "rbd_support", 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:0", 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 111151854 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.081 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snap_schedule", 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:0", 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2837055320 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.082 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "snap_schedule", 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:0", 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1901132380 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.083 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: "name": "volumes", 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: "addrvec": [ 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: { 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v2", 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "172.21.15.146:0", 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1200750981 2024-04-24T08:31:55.084 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.085 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:55.085 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-04-24T08:31:55.085 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:31:55.086 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:52 smithi177 ceph-mon[33046]: pgmap v331: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:55.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:52 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1851482662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:31:55.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:52 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1696473464' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T08:31:55.087 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:53 smithi177 ceph-mon[33046]: pgmap v332: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:55.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:52 smithi153 ceph-mon[32746]: pgmap v331: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:55.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:52 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1851482662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:31:55.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:52 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1696473464' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T08:31:55.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:53 smithi153 ceph-mon[32746]: pgmap v332: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:55.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb4f40a3d20 msgr2=0x7fb4f40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:55.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb4f40a3d20 0x7fb4f40a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8000c00 tx=0x7fb4f8005b50 comp rx=0 tx=0).stop 2024-04-24T08:31:55.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 msgr2=0x7fb5101a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:31:55.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 0x7fb5101a0660 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb50000e9f0 tx=0x7fb50000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:31:55.088 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 shutdown_connections 2024-04-24T08:31:55.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fb4f40a3d20 0x7fb4f40a61e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:55.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb510074cf0 0x7fb5101a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:55.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb51010e6c0 0x7fb51019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:55.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 --2- 172.21.15.146:0/1696473464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5100743d0 0x7fb5101a0660 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:31:55.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 >> 172.21.15.146:0/1696473464 conn(0x7fb5100fd930 msgr2=0x7fb5100fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:31:55.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 shutdown_connections 2024-04-24T08:31:55.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:31:52.831+0000 7fb50fe00640 1 -- 172.21.15.146:0/1696473464 wait complete. 2024-04-24T08:31:55.097 DEBUG:tasks.check_counter:Getting stats from a 2024-04-24T08:31:55.098 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-24T08:31:55.099 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:52 smithi146 ceph-mon[27805]: pgmap v331: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:55.099 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:52 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1851482662' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:31:55.099 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:52 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1696473464' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T08:31:55.099 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:53 smithi146 ceph-mon[27805]: pgmap v332: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:55.554 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:31:56.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:56 smithi153 ceph-mon[32746]: pgmap v333: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:56.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:56 smithi146 ceph-mon[27805]: pgmap v333: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:56.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:56 smithi177 ceph-mon[33046]: pgmap v333: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:31:57.918 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 360, 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 372, 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 1793959, 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 607383, 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 48, 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 6, 2024-04-24T08:31:57.919 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.174378569, 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.057098956, 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.125421501, 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.010323081, 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 371, 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.063336165, 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000170717 2024-04-24T08:31:57.920 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.921 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:31:57.921 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 98, 2024-04-24T08:31:57.921 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000034200, 2024-04-24T08:31:57.921 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000348 2024-04-24T08:31:57.921 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.921 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 1779504, 2024-04-24T08:31:57.922 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 141872 2024-04-24T08:31:57.922 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.922 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:31:57.922 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 32, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 223, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 7821, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 496319, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 12, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 5, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.050387375, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.031515666, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.012422895, 2024-04-24T08:31:57.923 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.000381186, 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 220, 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.046812200, 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000212782 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 64, 2024-04-24T08:31:57.924 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000514352, 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000008036 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 4736, 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 467856 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 565, 2024-04-24T08:31:57.925 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 632, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 160569, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 385411, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 9, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 5, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.140434981, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.060835237, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.077553286, 2024-04-24T08:31:57.926 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.007317537, 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 631, 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.068530354, 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000108605 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 1141, 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000457566, 2024-04-24T08:31:57.927 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000401 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: "cct": { 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: "total_workers": 1, 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: "unhealthy_workers": 0 2024-04-24T08:31:57.928 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: "finisher-MDSRank": { 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: "queue_len": 0, 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: "complete_latency": { 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 49, 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.003600231, 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000073474 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.929 INFO:teuthology.orchestra.run.smithi146.stdout: "finisher-PurgeQueue": { 2024-04-24T08:31:57.930 INFO:teuthology.orchestra.run.smithi146.stdout: "queue_len": 0, 2024-04-24T08:31:57.930 INFO:teuthology.orchestra.run.smithi146.stdout: "complete_latency": { 2024-04-24T08:31:57.930 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 2, 2024-04-24T08:31:57.930 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000011868, 2024-04-24T08:31:57.930 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000005934 2024-04-24T08:31:57.930 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:57.930 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": { 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: "request": 11, 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: "reply": 5, 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: "reply_latency": { 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 5, 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.002275120, 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000455024 2024-04-24T08:31:57.931 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "slow_reply": 0, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "forward": 6, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_fetch_complete": 0, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_fetch_keys": 0, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_commit": 11, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_split": 0, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_merge": 0, 2024-04-24T08:31:57.932 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes": 17, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_top": 0, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_bottom": 17, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_pin_tail": 0, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_pinned": 16, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_expired": 0, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_with_caps": 2, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": 2, 2024-04-24T08:31:57.933 INFO:teuthology.orchestra.run.smithi146.stdout: "subtrees": 2, 2024-04-24T08:31:57.934 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse": 17, 2024-04-24T08:31:57.934 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_hit": 10, 2024-04-24T08:31:57.934 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_forward": 0, 2024-04-24T08:31:57.934 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_discover": 3, 2024-04-24T08:31:57.934 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:31:57.934 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_remote_ino": 0, 2024-04-24T08:31:57.934 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_lock": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "load_cent": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "q": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "exported": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "exported_inodes": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "imported": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "imported_inodes": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "openino_dir_fetch": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:31:57.935 INFO:teuthology.orchestra.run.smithi146.stdout: "openino_peer_discover": 0, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "root_rfiles": 3454, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "root_rbytes": 26625078, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "root_rsnaps": 6, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_set_tag": 0, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:31:57.936 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:31:57.937 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:31:57.937 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:31:57.937 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_file_inodes": 0, 2024-04-24T08:31:57.937 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:31:57.937 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_revoke": 1, 2024-04-24T08:31:57.937 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_grant": 2, 2024-04-24T08:31:57.937 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:31:57.938 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:31:57.938 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:31:57.938 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_caps": 1, 2024-04-24T08:31:57.938 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:31:57.938 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_cap_release": 0, 2024-04-24T08:31:57.938 INFO:teuthology.orchestra.run.smithi146.stdout: "process_request_cap_release": 3 2024-04-24T08:31:57.938 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache": { 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_update": 0, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_update_receipt": 3, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_try_discover": 2, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_send_discover": 5, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_handle_discover": 0, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "num_strays": 0, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "num_strays_delayed": 0, 2024-04-24T08:31:57.939 INFO:teuthology.orchestra.run.smithi146.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_created": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_enqueued": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_reintegrated": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_migrated": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "num_recovering_processing": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:31:57.940 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_started": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_completed": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_exportdir": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_flush": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_fragstats": 0, 2024-04-24T08:31:57.941 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_inodestats": 0 2024-04-24T08:31:57.942 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.942 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log": { 2024-04-24T08:31:57.942 INFO:teuthology.orchestra.run.smithi146.stdout: "evlrg": 0, 2024-04-24T08:31:57.942 INFO:teuthology.orchestra.run.smithi146.stdout: "evadd": 10, 2024-04-24T08:31:57.942 INFO:teuthology.orchestra.run.smithi146.stdout: "evex": 0, 2024-04-24T08:31:57.943 INFO:teuthology.orchestra.run.smithi146.stdout: "evtrm": 0, 2024-04-24T08:31:57.943 INFO:teuthology.orchestra.run.smithi146.stdout: "ev": 10, 2024-04-24T08:31:57.943 INFO:teuthology.orchestra.run.smithi146.stdout: "evexg": 0, 2024-04-24T08:31:57.943 INFO:teuthology.orchestra.run.smithi146.stdout: "evexd": 0, 2024-04-24T08:31:57.943 INFO:teuthology.orchestra.run.smithi146.stdout: "segadd": 2, 2024-04-24T08:31:57.943 INFO:teuthology.orchestra.run.smithi146.stdout: "segex": 0, 2024-04-24T08:31:57.943 INFO:teuthology.orchestra.run.smithi146.stdout: "segtrm": 0, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "seg": 2, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "segmjr": 2, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "segexg": 0, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "segexd": 0, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "expos": 4194304, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "wrpos": 4197189, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "rdpos": 0, 2024-04-24T08:31:57.944 INFO:teuthology.orchestra.run.smithi146.stdout: "jlat": { 2024-04-24T08:31:57.945 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 9, 2024-04-24T08:31:57.945 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.019211072, 2024-04-24T08:31:57.945 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.002134563 2024-04-24T08:31:57.945 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.946 INFO:teuthology.orchestra.run.smithi146.stdout: "replayed": 0 2024-04-24T08:31:57.946 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.946 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_mem": { 2024-04-24T08:31:57.946 INFO:teuthology.orchestra.run.smithi146.stdout: "ino": 20, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "ino+": 20, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "ino-": 0, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "dir": 19, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "dir+": 20, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "dir-": 1, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "dn": 17, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "dn+": 17, 2024-04-24T08:31:57.947 INFO:teuthology.orchestra.run.smithi146.stdout: "dn-": 0, 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: "cap": 2, 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: "cap+": 3, 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: "cap-": 1, 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: "rss": 51672, 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: "heap": 223608 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_server": { 2024-04-24T08:31:57.948 INFO:teuthology.orchestra.run.smithi146.stdout: "dispatch_client_request": 13, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "dispatch_server_request": 0, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_request": 11, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_session": 29, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_peer_request": 0, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "req_create_latency": { 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.949 INFO:teuthology.orchestra.run.smithi146.stdout: "req_getattr_latency": { 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 5, 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.002275120, 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000455024 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: "req_getfilelock_latency": { 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.950 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: "req_link_latency": { 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookup_latency": { 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.951 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookuphash_latency": { 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupino_latency": { 2024-04-24T08:31:57.952 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupname_latency": { 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.953 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupparent_latency": { 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupsnap_latency": { 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.954 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.955 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lssnap_latency": { 2024-04-24T08:31:57.955 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.955 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.955 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.955 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.955 INFO:teuthology.orchestra.run.smithi146.stdout: "req_mkdir_latency": { 2024-04-24T08:31:57.956 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.956 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.956 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: "req_mknod_latency": { 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: "req_mksnap_latency": { 2024-04-24T08:31:57.957 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: "req_open_latency": { 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.958 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "req_readdir_latency": { 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rename_latency": { 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.959 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.960 INFO:teuthology.orchestra.run.smithi146.stdout: "req_renamesnap_latency": { 2024-04-24T08:31:57.960 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.960 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.960 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.960 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.960 INFO:teuthology.orchestra.run.smithi146.stdout: "req_snapdiff_latency": { 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rmdir_latency": { 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.961 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rmsnap_latency": { 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rmxattr_latency": { 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.962 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setattr_latency": { 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.963 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setdirlayout_latency": { 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setfilelock_latency": { 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.964 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setlayout_latency": { 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setxattr_latency": { 2024-04-24T08:31:57.965 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.966 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.966 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.966 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.966 INFO:teuthology.orchestra.run.smithi146.stdout: "req_symlink_latency": { 2024-04-24T08:31:57.966 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.966 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.966 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: "req_unlink_latency": { 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:31:57.967 INFO:teuthology.orchestra.run.smithi146.stdout: "req_getvxattr_latency": { 2024-04-24T08:31:57.968 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.968 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.968 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.968 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:57.968 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.968 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_sessions": { 2024-04-24T08:31:57.968 INFO:teuthology.orchestra.run.smithi146.stdout: "session_count": 2, 2024-04-24T08:31:57.969 INFO:teuthology.orchestra.run.smithi146.stdout: "session_add": 5, 2024-04-24T08:31:57.969 INFO:teuthology.orchestra.run.smithi146.stdout: "session_remove": 3, 2024-04-24T08:31:57.969 INFO:teuthology.orchestra.run.smithi146.stdout: "sessions_open": 2, 2024-04-24T08:31:57.969 INFO:teuthology.orchestra.run.smithi146.stdout: "sessions_stale": 0, 2024-04-24T08:31:57.970 INFO:teuthology.orchestra.run.smithi146.stdout: "total_load": 3, 2024-04-24T08:31:57.970 INFO:teuthology.orchestra.run.smithi146.stdout: "average_load": 0, 2024-04-24T08:31:57.970 INFO:teuthology.orchestra.run.smithi146.stdout: "avg_session_uptime": 619, 2024-04-24T08:31:57.970 INFO:teuthology.orchestra.run.smithi146.stdout: "mdthresh_evicted": 0 2024-04-24T08:31:57.970 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "mempool": { 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_items": 0, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:31:57.971 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:31:57.972 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_items": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_items": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:31:57.973 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_items": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_items": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_bytes": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_items": 0, 2024-04-24T08:31:57.974 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_bytes": 273355, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_items": 112, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_bytes": 176, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_items": 2, 2024-04-24T08:31:57.975 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bytes": 0, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_items": 0, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_items": 0, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_items": 0, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_bytes": 8128, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_items": 169, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:31:57.976 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_bytes": 0, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_items": 0, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_bytes": 83953, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_items": 408, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_bytes": 0, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_items": 0, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_bytes": 0, 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_items": 0 2024-04-24T08:31:57.977 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter": { 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "op_active": 0, 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "op_laggy": 0, 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "op_send": 49, 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "op_send_bytes": 13772, 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "op_resend": 33, 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "op_reply": 49, 2024-04-24T08:31:57.978 INFO:teuthology.orchestra.run.smithi146.stdout: "op_latency": { 2024-04-24T08:31:57.979 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 49, 2024-04-24T08:31:57.979 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.289995524, 2024-04-24T08:31:57.979 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.005918276 2024-04-24T08:31:57.979 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.979 INFO:teuthology.orchestra.run.smithi146.stdout: "op_inflight": 0, 2024-04-24T08:31:57.979 INFO:teuthology.orchestra.run.smithi146.stdout: "oplen_avg": { 2024-04-24T08:31:57.979 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 49, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 73 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "op": 49, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "op_r": 0, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "op_w": 49, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "op_rmw": 0, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "op_pg": 0, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_stat": 0, 2024-04-24T08:31:57.980 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_create": 10, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_read": 0, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_write": 9, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_writefull": 9, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_writesame": 0, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_append": 0, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_zero": 0, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_truncate": 0, 2024-04-24T08:31:57.981 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_delete": 6, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_mapext": 0, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_sparse_read": 0, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_clonerange": 0, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_getxattr": 0, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_setxattr": 20, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_rmxattr": 0, 2024-04-24T08:31:57.982 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_call": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_watch": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_notify": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_pgls": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_other": 0, 2024-04-24T08:31:57.983 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_active": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_send": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_resend": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_ping": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "poolop_active": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "poolop_send": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "poolop_resend": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "poolstat_active": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "poolstat_send": 0, 2024-04-24T08:31:57.984 INFO:teuthology.orchestra.run.smithi146.stdout: "poolstat_resend": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "statfs_active": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "statfs_send": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "statfs_resend": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "command_active": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "command_send": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "command_resend": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "map_epoch": 99, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "map_full": 0, 2024-04-24T08:31:57.985 INFO:teuthology.orchestra.run.smithi146.stdout: "map_inc": 11, 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_sessions": 9, 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_session_open": 9, 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_session_close": 0, 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_laggy": 0, 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_wr": 18, 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_rd": 0, 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_del": 1 2024-04-24T08:31:57.986 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.987 INFO:teuthology.orchestra.run.smithi146.stdout: "oft": { 2024-04-24T08:31:57.987 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_objs": 1, 2024-04-24T08:31:57.987 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:31:57.987 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_updates": 0, 2024-04-24T08:31:57.987 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_removes": 0 2024-04-24T08:31:57.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.987 INFO:teuthology.orchestra.run.smithi146.stdout: "osd-slow-ops": { 2024-04-24T08:31:57.988 INFO:teuthology.orchestra.run.smithi146.stdout: "slow_ops_count": 0 2024-04-24T08:31:57.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.988 INFO:teuthology.orchestra.run.smithi146.stdout: "purge_queue": { 2024-04-24T08:31:57.988 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing_ops": 0, 2024-04-24T08:31:57.988 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:31:57.989 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing": 0, 2024-04-24T08:31:57.989 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing_high_water": 0, 2024-04-24T08:31:57.989 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executed_ops": 0, 2024-04-24T08:31:57.989 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executed": 0, 2024-04-24T08:31:57.990 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_item_in_journal": 0 2024-04-24T08:31:57.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.990 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:31:57.990 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:31:57.991 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 104857600, 2024-04-24T08:31:57.991 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:31:57.991 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 953, 2024-04-24T08:31:57.991 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 1858159, 2024-04-24T08:31:57.992 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:31:57.992 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 953, 2024-04-24T08:31:57.992 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:31:57.992 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:31:57.993 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 953, 2024-04-24T08:31:57.993 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 1858159, 2024-04-24T08:31:57.993 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:31:57.993 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.994 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.994 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.994 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:57.994 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.995 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-objecter_bytes": { 2024-04-24T08:31:57.995 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:31:57.995 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 104857600, 2024-04-24T08:31:57.995 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:31:57.996 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 0, 2024-04-24T08:31:57.996 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 0, 2024-04-24T08:31:57.996 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:31:57.996 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 0, 2024-04-24T08:31:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 49, 2024-04-24T08:31:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 13772, 2024-04-24T08:31:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 49, 2024-04-24T08:31:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 13772, 2024-04-24T08:31:57.997 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:31:57.998 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:57.998 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:57.998 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:57.998 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:57.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:57.999 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-objecter_ops": { 2024-04-24T08:31:57.999 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:31:57.999 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 1024, 2024-04-24T08:31:57.999 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 0, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 0, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 0, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 49, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 49, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 49, 2024-04-24T08:31:58.000 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 49, 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:31:58.001 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 3758096384, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 0, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 0, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 0, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:31:58.002 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 0, 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 0, 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-write_buf_throttle-0x5613a8d3a500": { 2024-04-24T08:31:58.003 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:31:58.004 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 3758096384, 2024-04-24T08:31:58.004 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:31:58.004 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 10, 2024-04-24T08:31:58.004 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 2885, 2024-04-24T08:31:58.005 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:31:58.005 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 10, 2024-04-24T08:31:58.005 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:31:58.005 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:31:58.005 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 9, 2024-04-24T08:31:58.006 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 2885, 2024-04-24T08:31:58.006 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:31:58.006 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:31:58.007 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:31:58.007 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:31:58.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:58.007 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:31:58.008 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:31:58.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:31:58 smithi153 ceph-mon[32746]: pgmap v334: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:58.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:31:58 smithi146 ceph-mon[27805]: pgmap v334: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:58.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:31:58 smithi177 ceph-mon[33046]: pgmap v334: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:31:58.904 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-24T08:31:58.904 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-24T08:31:58.904 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=6 2024-04-24T08:31:58.904 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:31:58.904 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:31:58.904 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-04-24T08:31:58.905 DEBUG:tasks.check_counter:Getting stats from d 2024-04-24T08:31:58.905 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-24T08:31:59.341 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:32:00.707 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:00 smithi146 ceph-mon[27805]: pgmap v335: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:00.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:00 smithi153 ceph-mon[32746]: pgmap v335: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:00.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:00 smithi177 ceph-mon[33046]: pgmap v335: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:01.613 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:32:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 0, 2024-04-24T08:32:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 2, 2024-04-24T08:32:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:32:01.613 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 0, 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 4, 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 0, 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.003929349, 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.000809918, 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:01.614 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:01.615 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 6, 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 170, 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 3344, 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 186320, 2024-04-24T08:32:01.616 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 5, 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.033182839, 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.024378390, 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.003758809, 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 168, 2024-04-24T08:32:01.617 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.029171567, 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000173640 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.618 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 186320 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 233, 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 221, 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 1731424, 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 136224, 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 2, 2024-04-24T08:32:01.619 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 1, 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.107852996, 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.023980911, 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.110328658, 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.010697615, 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 220, 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.021511680, 2024-04-24T08:32:01.620 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000097780 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 1731424, 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 136224 2024-04-24T08:32:01.621 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "cct": { 2024-04-24T08:32:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "total_workers": 0, 2024-04-24T08:32:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "unhealthy_workers": 0 2024-04-24T08:32:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "mempool": { 2024-04-24T08:32:01.622 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:01.623 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:01.624 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:01.625 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_items": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:01.626 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_bytes": 148977, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_items": 30, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bytes": 0, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_items": 0, 2024-04-24T08:32:01.627 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_bytes": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_items": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:01.628 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_items": 0, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_bytes": 928, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_items": 2, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_items": 0, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_items": 0 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:01.629 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 104857600, 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 235, 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 1698572, 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 235, 2024-04-24T08:32:01.630 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 235, 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 1698572, 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:01.631 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:01.632 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:32:02.557 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-24T08:32:02.557 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-24T08:32:02.557 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-24T08:32:02.558 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-24T08:32:02.558 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-24T08:32:02.558 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-24T08:32:02.558 DEBUG:tasks.check_counter:Getting stats from g 2024-04-24T08:32:02.558 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-24T08:32:02.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:02 smithi153 ceph-mon[32746]: pgmap v336: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:02.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:02 smithi146 ceph-mon[27805]: pgmap v336: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:02.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:02 smithi177 ceph-mon[33046]: pgmap v336: 97 pgs: 97 active+clean; 74 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:03.011 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:32:04.706 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:04 smithi146 ceph-mon[27805]: pgmap v337: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:32:04.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:04 smithi153 ceph-mon[32746]: pgmap v337: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:32:04.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:04 smithi177 ceph-mon[33046]: pgmap v337: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:32:05.241 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:32:05.241 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:05.241 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 13081, 2024-04-24T08:32:05.241 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 17888, 2024-04-24T08:32:05.242 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 3999760, 2024-04-24T08:32:05.242 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 11568617, 2024-04-24T08:32:05.242 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 10, 2024-04-24T08:32:05.242 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 4, 2024-04-24T08:32:05.242 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 2.681317931, 2024-04-24T08:32:05.242 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 1.546935305, 2024-04-24T08:32:05.242 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 1.511413280, 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.165764749, 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 17887, 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 1.473995080, 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000082405 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:05.243 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 16418, 2024-04-24T08:32:05.244 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.007879534, 2024-04-24T08:32:05.244 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000479 2024-04-24T08:32:05.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.244 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 4864, 2024-04-24T08:32:05.244 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 3328 2024-04-24T08:32:05.244 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 11610, 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 13610, 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 5263395, 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 9338929, 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 13, 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 6, 2024-04-24T08:32:05.245 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 1.780728948, 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.825952296, 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.708752726, 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.236582654, 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 13608, 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 1.160458575, 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000085277 2024-04-24T08:32:05.246 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 9199, 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.004573242, 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000497 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 458656 2024-04-24T08:32:05.247 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 8677, 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 8665, 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 4018237, 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 5223369, 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 48, 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 8, 2024-04-24T08:32:05.248 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.924922921, 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.319823371, 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.308371952, 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.245262775, 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 8663, 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.705147514, 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000081397 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.249 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 66, 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000010915, 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000165 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 1782576, 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 165968 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "cct": { 2024-04-24T08:32:05.250 INFO:teuthology.orchestra.run.smithi146.stdout: "total_workers": 1, 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "unhealthy_workers": 0 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "finisher-MDSRank": { 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "queue_len": 0, 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "complete_latency": { 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 13307, 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 9.115420352, 2024-04-24T08:32:05.251 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000685009 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "finisher-PurgeQueue": { 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "queue_len": 0, 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "complete_latency": { 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 1816, 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.746856332, 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000411264 2024-04-24T08:32:05.252 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "mds": { 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "request": 3651, 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "reply": 3647, 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "reply_latency": { 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 3647, 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 9.971531146, 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.002734173 2024-04-24T08:32:05.253 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "slow_reply": 0, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "forward": 4, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_fetch_complete": 0, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_fetch_keys": 0, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_commit": 11, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_split": 0, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_merge": 0, 2024-04-24T08:32:05.254 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes": 1953, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_top": 5, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_bottom": 1948, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_pin_tail": 0, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_pinned": 1932, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_expired": 1, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "inodes_with_caps": 63, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "caps": 63, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "subtrees": 84, 2024-04-24T08:32:05.255 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse": 7497, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_hit": 5192, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_forward": 4, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_discover": 84, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_remote_ino": 0, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "traverse_lock": 3, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "load_cent": 0, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "q": 0, 2024-04-24T08:32:05.256 INFO:teuthology.orchestra.run.smithi146.stdout: "exported": 0, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "exported_inodes": 0, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "imported": 82, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "imported_inodes": 1776, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "openino_dir_fetch": 0, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "openino_backtrace_fetch": 20404, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "openino_peer_discover": 0, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "root_rfiles": 3454, 2024-04-24T08:32:05.257 INFO:teuthology.orchestra.run.smithi146.stdout: "root_rbytes": 26625078, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "root_rsnaps": 6, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_backtrace_fetch": 20404, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_set_tag": 0, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_dir_inodes": 13, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:32:05.258 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_dirfrag_rstats": 13, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "scrub_file_inodes": 20391, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_revoke": 33, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_grant": 2290, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_trunc": 2, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:32:05.259 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_caps": 3370, 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_caps_dirty": 2433, 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_cap_release": 334, 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "process_request_cap_release": 2454 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_cache": { 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_update": 0, 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_update_receipt": 0, 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_try_discover": 0, 2024-04-24T08:32:05.260 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_send_discover": 78, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "dir_handle_discover": 0, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "num_strays": 0, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "num_strays_delayed": 0, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_created": 1184, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_enqueued": 1184, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_reintegrated": 0, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "strays_migrated": 0, 2024-04-24T08:32:05.261 INFO:teuthology.orchestra.run.smithi146.stdout: "num_recovering_processing": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_started": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "recovery_completed": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:32:05.262 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_exportdir": 0, 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_flush": 0, 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_fragstats": 0, 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "ireq_inodestats": 0 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_log": { 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "evlrg": 0, 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "evadd": 7474, 2024-04-24T08:32:05.263 INFO:teuthology.orchestra.run.smithi146.stdout: "evex": 0, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "evtrm": 0, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "ev": 7474, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "evexg": 0, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "evexd": 0, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "segadd": 10, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "segex": 0, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "segtrm": 0, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "seg": 10, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "segmjr": 2, 2024-04-24T08:32:05.264 INFO:teuthology.orchestra.run.smithi146.stdout: "segexg": 0, 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "segexd": 0, 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "expos": 4194304, 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "wrpos": 16690792, 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "rdpos": 0, 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "jlat": { 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 162, 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 2.411152174, 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.014883655 2024-04-24T08:32:05.265 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "replayed": 0 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_mem": { 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "ino": 1429, 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "ino+": 2613, 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "ino-": 1184, 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "dir": 106, 2024-04-24T08:32:05.266 INFO:teuthology.orchestra.run.smithi146.stdout: "dir+": 107, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "dir-": 1, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "dn": 1953, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "dn+": 3152, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "dn-": 1199, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "cap": 63, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "cap+": 2546, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "cap-": 2483, 2024-04-24T08:32:05.267 INFO:teuthology.orchestra.run.smithi146.stdout: "rss": 81688, 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "heap": 223608 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_server": { 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "dispatch_client_request": 5715, 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "dispatch_server_request": 0, 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_request": 3651, 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_client_session": 21, 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "handle_peer_request": 1929, 2024-04-24T08:32:05.268 INFO:teuthology.orchestra.run.smithi146.stdout: "req_create_latency": { 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 748, 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 1.961513581, 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.002622344 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "req_getattr_latency": { 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 3, 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.002654669, 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000884889 2024-04-24T08:32:05.269 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "req_getfilelock_latency": { 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "req_link_latency": { 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.270 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookup_latency": { 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 1676, 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 1.805094451, 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.001077025 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.271 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookuphash_latency": { 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupino_latency": { 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 1, 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000391141, 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000391141 2024-04-24T08:32:05.272 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupname_latency": { 2024-04-24T08:32:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupparent_latency": { 2024-04-24T08:32:05.273 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lookupsnap_latency": { 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.274 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "req_lssnap_latency": { 2024-04-24T08:32:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.275 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "req_mkdir_latency": { 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 1, 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.002888855, 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.002888855 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "req_mknod_latency": { 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.276 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: "req_mksnap_latency": { 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.277 INFO:teuthology.orchestra.run.smithi146.stdout: "req_open_latency": { 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 14, 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.007461204, 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000532943 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "req_readdir_latency": { 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 4, 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.003946922, 2024-04-24T08:32:05.278 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000986730 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rename_latency": { 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 13, 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.050773683, 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.003905667 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "req_renamesnap_latency": { 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.279 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "req_snapdiff_latency": { 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rmdir_latency": { 2024-04-24T08:32:05.280 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rmsnap_latency": { 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.281 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "req_rmxattr_latency": { 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setattr_latency": { 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 2, 2024-04-24T08:32:05.282 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.022753563, 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.011376781 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setdirlayout_latency": { 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setfilelock_latency": { 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.283 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setlayout_latency": { 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.284 INFO:teuthology.orchestra.run.smithi146.stdout: "req_setxattr_latency": { 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "req_symlink_latency": { 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 1, 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.001493621, 2024-04-24T08:32:05.285 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.001493621 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "req_unlink_latency": { 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 1184, 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 6.112559456, 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.005162634 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:32:05.286 INFO:teuthology.orchestra.run.smithi146.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "req_getvxattr_latency": { 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_sessions": { 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "session_count": 1, 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "session_add": 4, 2024-04-24T08:32:05.287 INFO:teuthology.orchestra.run.smithi146.stdout: "session_remove": 3, 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "sessions_open": 1, 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "sessions_stale": 0, 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "total_load": 2607, 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "average_load": 2607, 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "avg_session_uptime": 637, 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "mdthresh_evicted": 0 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.288 INFO:teuthology.orchestra.run.smithi146.stdout: "mempool": { 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:05.289 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:05.290 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:05.291 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_items": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:05.292 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_bytes": 351142, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_items": 1484, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_bytes": 880, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_items": 10, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bytes": 0, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_items": 0, 2024-04-24T08:32:05.293 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_bytes": 8128, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_items": 169, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:05.294 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_items": 0, 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_bytes": 4975478, 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_items": 60697, 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_items": 0, 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_items": 0 2024-04-24T08:32:05.295 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "objecter": { 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "op_active": 0, 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "op_laggy": 0, 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "op_send": 21894, 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "op_send_bytes": 12999747, 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "op_resend": 29, 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "op_reply": 21894, 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "op_latency": { 2024-04-24T08:32:05.296 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 21894, 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 16.981736400, 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000775634 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "op_inflight": 0, 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "oplen_avg": { 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 21894, 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 21959 2024-04-24T08:32:05.297 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "op": 21894, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "op_r": 20425, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "op_w": 1469, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "op_rmw": 0, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "op_pg": 0, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_stat": 0, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_create": 10, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_read": 21, 2024-04-24T08:32:05.298 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_write": 183, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_writefull": 23, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_writesame": 0, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_append": 0, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_zero": 0, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_truncate": 0, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_delete": 1198, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_mapext": 0, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_sparse_read": 0, 2024-04-24T08:32:05.299 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_clonerange": 0, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_getxattr": 20404, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_setxattr": 20, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_rmxattr": 0, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_call": 0, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_watch": 0, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_notify": 0, 2024-04-24T08:32:05.300 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_pgls": 0, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "osdop_other": 2, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_active": 0, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_send": 0, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_resend": 0, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "linger_ping": 0, 2024-04-24T08:32:05.301 INFO:teuthology.orchestra.run.smithi146.stdout: "poolop_active": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "poolop_send": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "poolop_resend": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "poolstat_active": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "poolstat_send": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "poolstat_resend": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "statfs_active": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "statfs_send": 0, 2024-04-24T08:32:05.302 INFO:teuthology.orchestra.run.smithi146.stdout: "statfs_resend": 0, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "command_active": 0, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "command_send": 0, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "command_resend": 0, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "map_epoch": 99, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "map_full": 0, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "map_inc": 11, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_sessions": 12, 2024-04-24T08:32:05.303 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_session_open": 12, 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_session_close": 0, 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_laggy": 0, 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_wr": 83, 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_rd": 0, 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_del": 15 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "oft": { 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_objs": 1, 2024-04-24T08:32:05.304 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_updates": 0, 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "omap_total_removes": 0 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "osd-slow-ops": { 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "slow_ops_count": 0 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "purge_queue": { 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing_ops": 0, 2024-04-24T08:32:05.305 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing_ops_high_water": 10, 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing": 0, 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executing_high_water": 10, 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executed_ops": 1184, 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_executed": 1184, 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "pq_item_in_journal": 0 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:05.306 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 104857600, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 33364, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 10259130, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 33364, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:32:05.307 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 33364, 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 10259130, 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.308 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-objecter_bytes": { 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 104857600, 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 0, 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 0, 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:05.309 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 21894, 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 13119331, 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 21894, 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 13119331, 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.310 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-objecter_ops": { 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 1024, 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 0, 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 0, 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:05.311 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 21894, 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 21894, 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 21894, 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 21894, 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.312 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 3758096384, 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 1184, 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 119584, 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:05.313 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 1184, 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 21, 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 119584, 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.314 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-write_buf_throttle-0x55fcfb754500": { 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 3758096384, 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 7474, 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 12496488, 2024-04-24T08:32:05.315 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 7474, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 162, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 12496488, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:05.316 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.317 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:05.317 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:32:06.252 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-24T08:32:06.252 INFO:tasks.check_counter:Daemon mds.g mds.imported=82 2024-04-24T08:32:06.252 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=6 2024-04-24T08:32:06.252 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:32:06.252 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:32:06.252 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-04-24T08:32:06.252 DEBUG:tasks.check_counter:Getting stats from j 2024-04-24T08:32:06.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-24T08:32:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:06 smithi153 ceph-mon[32746]: pgmap v338: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:32:06.719 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:32:06.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:06 smithi146 ceph-mon[27805]: pgmap v338: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:32:06.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:06 smithi177 ceph-mon[33046]: pgmap v338: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:32:08.652 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:08 smithi146 ceph-mon[27805]: pgmap v339: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:32:08.653 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:08 smithi146 ceph-mon[27805]: osdmap e100: 12 total, 12 up, 12 in 2024-04-24T08:32:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:08 smithi177 ceph-mon[33046]: pgmap v339: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:32:08.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:08 smithi177 ceph-mon[33046]: osdmap e100: 12 total, 12 up, 12 in 2024-04-24T08:32:08.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:08 smithi153 ceph-mon[32746]: pgmap v339: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-24T08:32:08.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:08 smithi153 ceph-mon[32746]: osdmap e100: 12 total, 12 up, 12 in 2024-04-24T08:32:08.982 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 0, 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 2, 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 0, 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 4, 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 0, 2024-04-24T08:32:08.983 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.004314216, 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.000986323, 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:08.984 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:08.985 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 6, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 173, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 3344, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 186848, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 5, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.037532332, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.026528854, 2024-04-24T08:32:08.986 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.005736854, 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 171, 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.034703920, 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000202946 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:08.987 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 186848 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_messages": 235, 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages": 223, 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_bytes": 1731968, 2024-04-24T08:32:08.988 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_bytes": 137504, 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_created_connections": 2, 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_active_connections": 1, 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_total_time": 0.088415974, 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_send_time": 0.024752841, 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_recv_time": 0.071318038, 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_running_fast_dispatch_time": 0.010086284, 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:08.989 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 222, 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.022306460, 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000100479 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.990 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_recv_encrypted_bytes": 1731968, 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: "msgr_send_encrypted_bytes": 137504 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: "cct": { 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: "total_workers": 0, 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: "unhealthy_workers": 0 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: "mempool": { 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:08.991 INFO:teuthology.orchestra.run.smithi146.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:08.992 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:08.993 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:08.994 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_items": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:08.995 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_bytes": 150008, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_anon_items": 31, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_bytes": 0, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_items": 0, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:08.996 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_bytes": 0, 2024-04-24T08:32:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_items": 0, 2024-04-24T08:32:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: "pgmap_items": 0, 2024-04-24T08:32:08.997 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_bytes": 928, 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_co_items": 2, 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_1_items": 0, 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "unittest_2_items": 0 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "val": 0, 2024-04-24T08:32:08.998 INFO:teuthology.orchestra.run.smithi146.stdout: "max": 104857600, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "get_started": 0, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "get": 237, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "get_sum": 1698832, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "get_or_fail_success": 237, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "take": 0, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "take_sum": 0, 2024-04-24T08:32:08.999 INFO:teuthology.orchestra.run.smithi146.stdout: "put": 237, 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout: "put_sum": 1698832, 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout: "wait": { 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout: "avgcount": 0, 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout: "sum": 0.000000000, 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout: "avgtime": 0.000000000 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-04-24T08:32:09.000 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-04-24T08:32:09.832 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-24T08:32:09.833 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-24T08:32:09.833 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-24T08:32:09.833 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-24T08:32:09.833 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-24T08:32:09.833 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-24T08:32:09.833 DEBUG:tasks.check_counter:Getting stats from b 2024-04-24T08:32:09.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-24T08:32:10.083 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:32:10.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:10 smithi146 ceph-mon[27805]: pgmap v341: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:32:10.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:10 smithi177 ceph-mon[33046]: pgmap v341: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:32:10.861 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:10 smithi153 ceph-mon[32746]: pgmap v341: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:32:11.707 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-24T08:32:11.708 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:11.708 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 14509, 2024-04-24T08:32:11.708 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 13771, 2024-04-24T08:32:11.708 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 4048201, 2024-04-24T08:32:11.708 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 21301241, 2024-04-24T08:32:11.708 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 23, 2024-04-24T08:32:11.708 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.945168767, 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.332739399, 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.237264053, 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.313920902, 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 13773, 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.504235931, 2024-04-24T08:32:11.709 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000036610 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2117, 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000420900, 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000198 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 369104, 2024-04-24T08:32:11.710 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 98880 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 20046, 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 18319, 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 7365413, 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 21675338, 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 16, 2024-04-24T08:32:11.711 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 6, 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 1.126179689, 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.389930269, 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.490832835, 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.285354537, 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 18322, 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2.629937817, 2024-04-24T08:32:11.712 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000143539 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 9067, 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.001408389, 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000155 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1793136, 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 495296 2024-04-24T08:32:11.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 24285, 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 41082, 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 7671176, 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 24819975, 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 150, 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 8, 2024-04-24T08:32:11.714 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 1.619181742, 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.661032997, 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.877065835, 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.235891835, 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 41072, 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.841594309, 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000020490 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.715 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 38548, 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.006065268, 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000157 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 130880, 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 170464 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-24T08:32:11.716 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 25057, 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 12.081395233, 2024-04-24T08:32:11.717 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000482156 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3634, 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.607783955, 2024-04-24T08:32:11.718 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000167249 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 8942, 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 8875, 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8875, 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 31.057027238, 2024-04-24T08:32:11.719 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.003499383 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 67, 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 194, 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 22, 2024-04-24T08:32:11.720 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 3219, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 239, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 2930, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 50, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 2878, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 1785, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 2185, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 2185, 2024-04-24T08:32:11.721 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 86, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 9217, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 8620, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 67, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 3, 2024-04-24T08:32:11.722 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 82, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 1776, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 32717, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-04-24T08:32:11.723 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 3454, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 26625078, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 6, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 32717, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 1138, 2024-04-24T08:32:11.724 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 19, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 1119, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 31579, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 5, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 103, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 8512, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 6, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:32:11.725 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 4759, 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 4246, 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 361, 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 2340 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 1, 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-24T08:32:11.726 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 83, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 2063, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 2060, 2024-04-24T08:32:11.727 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 3, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:32:11.728 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 19, 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 82, 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 22, 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.729 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 16815, 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 16815, 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-24T08:32:11.730 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 20, 2024-04-24T08:32:11.731 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-24T08:32:11.731 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-24T08:32:11.731 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 20, 2024-04-24T08:32:11.731 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 3, 2024-04-24T08:32:11.731 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-24T08:32:11.731 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-24T08:32:11.731 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-24T08:32:11.732 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 40913224, 2024-04-24T08:32:11.732 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 0, 2024-04-24T08:32:11.732 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-24T08:32:11.732 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 586, 2024-04-24T08:32:11.732 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 23.810137401, 2024-04-24T08:32:11.732 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.040631633 2024-04-24T08:32:11.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 0 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 2240, 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 6087, 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 3847, 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 242, 2024-04-24T08:32:11.733 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 274, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 32, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 3219, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 7193, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 3974, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 2185, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 6029, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 3844, 2024-04-24T08:32:11.734 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 89540, 2024-04-24T08:32:11.735 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 231800 2024-04-24T08:32:11.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.735 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-24T08:32:11.735 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 9253, 2024-04-24T08:32:11.735 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 1929, 2024-04-24T08:32:11.735 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 8942, 2024-04-24T08:32:11.735 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 120, 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 3858, 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 5933, 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 24.499300356, 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.004129327 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-24T08:32:11.736 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 164, 2024-04-24T08:32:11.737 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.343834993, 2024-04-24T08:32:11.737 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002096554 2024-04-24T08:32:11.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.737 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-24T08:32:11.737 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.737 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.737 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3, 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.003150699, 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001050233 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-24T08:32:11.738 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-04-24T08:32:11.739 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.007354240, 2024-04-24T08:32:11.739 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000919280 2024-04-24T08:32:11.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.739 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-24T08:32:11.739 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.739 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.740 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.740 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-24T08:32:11.740 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 1, 2024-04-24T08:32:11.740 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000313488, 2024-04-24T08:32:11.740 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000313488 2024-04-24T08:32:11.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.741 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.742 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.742 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-24T08:32:11.742 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.742 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.742 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.743 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-24T08:32:11.744 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 11, 2024-04-24T08:32:11.744 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.002440358, 2024-04-24T08:32:11.744 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000221850 2024-04-24T08:32:11.744 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.744 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-24T08:32:11.744 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 84, 2024-04-24T08:32:11.744 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.075974484, 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000904458 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-24T08:32:11.745 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 11, 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.071518312, 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.006501664 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 427, 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.106941814, 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000250449 2024-04-24T08:32:11.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 104, 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.665162680, 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.006395795 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 69, 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.398132280, 2024-04-24T08:32:11.747 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.005770033 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.748 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 22, 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.539028398, 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.024501290 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.749 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 5, 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.028326482, 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.005665296 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 7, 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.147906764, 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.021129537 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.751 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.002626086, 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000328260 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-24T08:32:11.752 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 8, 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.019635073, 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002454384 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.753 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 6, 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.004859193, 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000809865 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2004, 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 4.140521538, 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.002066128 2024-04-24T08:32:11.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.755 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 6, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 4, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 2, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 52, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 26, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 632, 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-24T08:32:11.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:11.757 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:11.758 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:11.759 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-24T08:32:11.760 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 387975, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 2323, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 1408, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 16, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-24T08:32:11.761 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 8128, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 169, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:11.762 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 8793935, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 100102, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:11.763 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 36361, 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 40007602, 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 37, 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 36361, 2024-04-24T08:32:11.764 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-24T08:32:11.765 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 36361, 2024-04-24T08:32:11.765 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 46.976124904, 2024-04-24T08:32:11.765 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.001291937 2024-04-24T08:32:11.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.765 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-24T08:32:11.765 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 36361, 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 36669 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 36361, 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 32795, 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 3566, 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-24T08:32:11.766 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 0, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 11, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 59, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 626, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 49, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-24T08:32:11.767 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-24T08:32:11.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 6, 2024-04-24T08:32:11.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 2596, 2024-04-24T08:32:11.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-24T08:32:11.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-24T08:32:11.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-24T08:32:11.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 32736, 2024-04-24T08:32:11.768 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 22, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:32:11.769 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 6, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-24T08:32:11.770 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-24T08:32:11.771 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-24T08:32:11.771 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-24T08:32:11.771 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-24T08:32:11.771 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-24T08:32:11.771 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-24T08:32:11.771 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-24T08:32:11.771 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 100, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 13, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 12, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 12, 2024-04-24T08:32:11.772 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 517, 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 0, 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 41 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 1, 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:32:11.773 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-24T08:32:11.774 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 10, 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 10, 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 2060, 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 2060, 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:11.775 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:11.776 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-24T08:32:11.776 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:11.776 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 58836, 2024-04-24T08:32:11.776 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 13717765, 2024-04-24T08:32:11.776 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:11.776 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 58836, 2024-04-24T08:32:11.776 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 58836, 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 13717765, 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:11.778 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 36361, 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 40215678, 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 36361, 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 40215678, 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.779 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-24T08:32:11.780 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 36361, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 36361, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 36361, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 36361, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.781 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2060, 2024-04-24T08:32:11.782 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 208076, 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2060, 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 40, 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 208076, 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:11.783 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.784 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.784 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:11.784 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x55e5e9e9c500": { 2024-04-24T08:32:11.784 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 112, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 16815, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 36718920, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 16815, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:11.785 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 586, 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 36718808, 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.786 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:11.787 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-24T08:32:12.480 INFO:tasks.check_counter:Daemon mds.b mds.exported=82 2024-04-24T08:32:12.480 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-24T08:32:12.480 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-24T08:32:12.480 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-04-24T08:32:12.480 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-04-24T08:32:12.481 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=22 2024-04-24T08:32:12.481 DEBUG:tasks.check_counter:Getting stats from e 2024-04-24T08:32:12.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-24T08:32:12.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:12 smithi153 ceph-mon[32746]: pgmap v342: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:32:12.734 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:32:12.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:12 smithi146 ceph-mon[27805]: pgmap v342: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:32:12.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:12 smithi177 ceph-mon[33046]: pgmap v342: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-24T08:32:14.313 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-24T08:32:14.314 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:14.314 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 0, 2024-04-24T08:32:14.314 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2, 2024-04-24T08:32:14.314 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:32:14.314 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 0, 2024-04-24T08:32:14.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-04-24T08:32:14.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 0, 2024-04-24T08:32:14.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.000998133, 2024-04-24T08:32:14.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:32:14.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.000117217, 2024-04-24T08:32:14.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:14.315 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:14.316 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 239, 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 225, 2024-04-24T08:32:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1734992, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 137952, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 2, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.048969228, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.014360742, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.039583069, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.005513398, 2024-04-24T08:32:14.318 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 225, 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.013134283, 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000058374 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:14.319 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1734992, 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 137952 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 2, 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 171, 2024-04-24T08:32:14.320 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 320, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 186400, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 5, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.012182520, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.008311142, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.001293836, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:14.321 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 168, 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.024831244, 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000147805 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:14.322 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 186400 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 0, 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.323 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:14.324 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:14.325 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:14.325 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:14.325 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:14.325 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:14.325 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:14.325 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:14.325 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:14.326 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-24T08:32:14.327 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 148933, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 30, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:14.328 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 0, 2024-04-24T08:32:14.329 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:14.330 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:14.330 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:14.330 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-24T08:32:14.330 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 928, 2024-04-24T08:32:14.330 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 2, 2024-04-24T08:32:14.330 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:14.330 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 237, 2024-04-24T08:32:14.331 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1698832, 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 237, 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 237, 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1698832, 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:14.332 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:14.333 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:14.333 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:14.333 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:14.333 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:14.333 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-24T08:32:14.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:14 smithi153 ceph-mon[32746]: pgmap v343: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:14.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:14 smithi146 ceph-mon[27805]: pgmap v343: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:14.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:14 smithi177 ceph-mon[33046]: pgmap v343: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:15.060 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-04-24T08:32:15.060 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-04-24T08:32:15.060 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-24T08:32:15.060 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-24T08:32:15.061 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-24T08:32:15.061 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-04-24T08:32:15.061 DEBUG:tasks.check_counter:Getting stats from h 2024-04-24T08:32:15.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-24T08:32:15.301 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:32:16.895 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-24T08:32:16.895 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:16.895 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 2358, 2024-04-24T08:32:16.895 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 2360, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 668800, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 749851, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 6, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.181099627, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.058305800, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.043650273, 2024-04-24T08:32:16.896 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.064777079, 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2358, 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.070835366, 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000030040 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.897 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 257, 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 242, 2024-04-24T08:32:16.898 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1766732, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 140682, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 3, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 2, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.047542012, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.014374622, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.037660284, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.005739238, 2024-04-24T08:32:16.899 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 241, 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.012454856, 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000051679 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.900 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1766176, 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 139984 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 8, 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 174, 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 3772, 2024-04-24T08:32:16.901 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 540910, 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 7, 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 4, 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.010540071, 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.006351213, 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.002200926, 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000015930, 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:16.902 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 171, 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.024584881, 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000143771 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.903 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 540592 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 1, 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-MDSRank": { 2024-04-24T08:32:16.904 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3958, 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.098189246, 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000024807 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: "finisher-PurgeQueue": { 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: "queue_len": 0, 2024-04-24T08:32:16.905 INFO:teuthology.orchestra.run.smithi153.stdout: "complete_latency": { 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: "request": 0, 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: "reply": 0, 2024-04-24T08:32:16.906 INFO:teuthology.orchestra.run.smithi153.stdout: "reply_latency": { 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_reply": 0, 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "forward": 0, 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_complete": 0, 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_fetch_keys": 0, 2024-04-24T08:32:16.907 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_commit": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_split": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_merge": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_top": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_bottom": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pin_tail": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_pinned": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_expired": 0, 2024-04-24T08:32:16.908 INFO:teuthology.orchestra.run.smithi153.stdout: "inodes_with_caps": 0, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "caps": 0, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "subtrees": 1, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse": 0, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_hit": 0, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_forward": 0, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_discover": 0, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:32:16.909 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_remote_ino": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "traverse_lock": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "load_cent": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "q": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "exported": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "exported_inodes": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "imported": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "imported_inodes": 0, 2024-04-24T08:32:16.910 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_dir_fetch": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "openino_peer_discover": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rfiles": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rbytes": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "root_rsnaps": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_set_tag": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:32:16.911 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "scrub_file_inodes": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_grant": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:32:16.912 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps": 0, 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_cap_release": 0, 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: "process_request_cap_release": 0 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache": { 2024-04-24T08:32:16.913 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update": 0, 2024-04-24T08:32:16.914 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_update_receipt": 0, 2024-04-24T08:32:16.914 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_try_discover": 0, 2024-04-24T08:32:16.914 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_send_discover": 0, 2024-04-24T08:32:16.914 INFO:teuthology.orchestra.run.smithi153.stdout: "dir_handle_discover": 0, 2024-04-24T08:32:16.914 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays": 0, 2024-04-24T08:32:16.914 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_delayed": 0, 2024-04-24T08:32:16.914 INFO:teuthology.orchestra.run.smithi153.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:32:16.915 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_created": 0, 2024-04-24T08:32:16.915 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_enqueued": 0, 2024-04-24T08:32:16.915 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_reintegrated": 0, 2024-04-24T08:32:16.915 INFO:teuthology.orchestra.run.smithi153.stdout: "strays_migrated": 0, 2024-04-24T08:32:16.915 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_processing": 0, 2024-04-24T08:32:16.915 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:32:16.915 INFO:teuthology.orchestra.run.smithi153.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_started": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_completed": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_exportdir": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_flush": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:32:16.916 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_fragstats": 0, 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: "ireq_inodestats": 0 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log": { 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: "evlrg": 0, 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: "evadd": 0, 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: "evex": 0, 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: "evtrm": 0, 2024-04-24T08:32:16.917 INFO:teuthology.orchestra.run.smithi153.stdout: "ev": 10, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "evexg": 0, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "evexd": 0, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "segadd": 0, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "segex": 0, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "segtrm": 0, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "seg": 2, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "segmjr": 2, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "segexg": 0, 2024-04-24T08:32:16.918 INFO:teuthology.orchestra.run.smithi153.stdout: "segexd": 0, 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: "expos": 4194304, 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: "wrpos": 4197189, 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: "rdpos": 4197048, 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: "jlat": { 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "replayed": 10 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mem": { 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "ino": 3, 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "ino+": 3, 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "ino-": 0, 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "dir": 1, 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "dir+": 1, 2024-04-24T08:32:16.920 INFO:teuthology.orchestra.run.smithi153.stdout: "dir-": 0, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "dn": 0, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "dn+": 0, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "dn-": 0, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "cap": 0, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "cap+": 0, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "cap-": 0, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "rss": 53068, 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: "heap": 215416 2024-04-24T08:32:16.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_server": { 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_client_request": 0, 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "dispatch_server_request": 0, 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_request": 0, 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_client_session": 0, 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "handle_peer_request": 0, 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "req_create_latency": { 2024-04-24T08:32:16.922 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getattr_latency": { 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.923 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getfilelock_latency": { 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: "req_link_latency": { 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.924 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookup_latency": { 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookuphash_latency": { 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.925 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupino_latency": { 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupname_latency": { 2024-04-24T08:32:16.926 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupparent_latency": { 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.927 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lookupsnap_latency": { 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: "req_lssnap_latency": { 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.929 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mkdir_latency": { 2024-04-24T08:32:16.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.929 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.929 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mknod_latency": { 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: "req_mksnap_latency": { 2024-04-24T08:32:16.930 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: "req_open_latency": { 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "req_readdir_latency": { 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rename_latency": { 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.932 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: "req_renamesnap_latency": { 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: "req_snapdiff_latency": { 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.933 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmdir_latency": { 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.934 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmsnap_latency": { 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: "req_rmxattr_latency": { 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setattr_latency": { 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setdirlayout_latency": { 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.936 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setfilelock_latency": { 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setlayout_latency": { 2024-04-24T08:32:16.937 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: "req_setxattr_latency": { 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "req_symlink_latency": { 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "req_unlink_latency": { 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.939 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: "req_getvxattr_latency": { 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.940 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessions": { 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: "session_count": 2, 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: "session_add": 5, 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: "session_remove": 3, 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_open": 4, 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: "sessions_stale": 0, 2024-04-24T08:32:16.941 INFO:teuthology.orchestra.run.smithi153.stdout: "total_load": 0, 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "average_load": 0, 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "avg_session_uptime": 3728, 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "mdthresh_evicted": 0 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:16.942 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:16.943 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:16.944 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:16.945 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 181269, 2024-04-24T08:32:16.946 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 47, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:16.947 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 8248, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 172, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 8512, 2024-04-24T08:32:16.948 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 13, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter": { 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "op_active": 0, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "op_laggy": 0, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send": 2361, 2024-04-24T08:32:16.949 INFO:teuthology.orchestra.run.smithi153.stdout: "op_send_bytes": 21, 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: "op_resend": 3, 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: "op_reply": 2361, 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: "op_latency": { 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2361, 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 1.279976074, 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000542133 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.950 INFO:teuthology.orchestra.run.smithi153.stdout: "op_inflight": 0, 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: "oplen_avg": { 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 2361, 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 2362 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: "op": 2361, 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: "op_r": 2361, 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: "op_w": 0, 2024-04-24T08:32:16.951 INFO:teuthology.orchestra.run.smithi153.stdout: "op_rmw": 0, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "op_pg": 0, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_stat": 1544, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_create": 0, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_read": 815, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_write": 0, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writefull": 0, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_writesame": 0, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_append": 0, 2024-04-24T08:32:16.952 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_zero": 0, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_truncate": 0, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_delete": 0, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_mapext": 0, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_sparse_read": 0, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_clonerange": 0, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_getxattr": 1, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_setxattr": 0, 2024-04-24T08:32:16.953 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_rmxattr": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_call": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_watch": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_notify": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls": 0, 2024-04-24T08:32:16.954 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "osdop_other": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_active": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_send": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_resend": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "linger_ping": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_active": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_send": 0, 2024-04-24T08:32:16.955 INFO:teuthology.orchestra.run.smithi153.stdout: "poolop_resend": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_active": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_send": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "poolstat_resend": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_active": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_send": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "statfs_resend": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "command_active": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "command_send": 0, 2024-04-24T08:32:16.956 INFO:teuthology.orchestra.run.smithi153.stdout: "command_resend": 0, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "map_epoch": 101, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "map_full": 0, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "map_inc": 13, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_sessions": 4, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_open": 4, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_session_close": 0, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_laggy": 0, 2024-04-24T08:32:16.957 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_wr": 0, 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_rd": 2, 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_del": 0 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: "oft": { 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_objs": 0, 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_updates": 0, 2024-04-24T08:32:16.958 INFO:teuthology.orchestra.run.smithi153.stdout: "omap_total_removes": 0 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: "osd-slow-ops": { 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: "slow_ops_count": 0 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: "purge_queue": { 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops": 0, 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing": 0, 2024-04-24T08:32:16.959 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executing_high_water": 0, 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed_ops": 0, 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_executed": 0, 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: "pq_item_in_journal": 0 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-24T08:32:16.960 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 2619, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 2187460, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 2619, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2619, 2024-04-24T08:32:16.961 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2187460, 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_bytes": { 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:16.962 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2361, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2890, 2024-04-24T08:32:16.963 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2361, 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2890, 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-objecter_ops": { 2024-04-24T08:32:16.964 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 1024, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 2361, 2024-04-24T08:32:16.965 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 2361, 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 2361, 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 2361, 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:16.967 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.968 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-write_buf_throttle-0x5571487166e0": { 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 3758096384, 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 0, 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 0, 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:16.969 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 0, 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 0, 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:16.970 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:16.971 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.971 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:16.971 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-24T08:32:17.210 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:16 smithi153 ceph-mon[32746]: pgmap v344: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:17.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:16 smithi153 ceph-mon[32746]: osdmap e101: 12 total, 12 up, 12 in 2024-04-24T08:32:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:16 smithi146 ceph-mon[27805]: pgmap v344: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:17.242 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:16 smithi146 ceph-mon[27805]: osdmap e101: 12 total, 12 up, 12 in 2024-04-24T08:32:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:16 smithi177 ceph-mon[33046]: pgmap v344: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:17.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:16 smithi177 ceph-mon[33046]: osdmap e101: 12 total, 12 up, 12 in 2024-04-24T08:32:17.592 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-24T08:32:17.592 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-24T08:32:17.592 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-24T08:32:17.592 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:32:17.593 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:32:17.593 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-04-24T08:32:17.593 DEBUG:tasks.check_counter:Getting stats from k 2024-04-24T08:32:17.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-24T08:32:17.843 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:32:18.276 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:17 smithi153 ceph-mon[32746]: pgmap v346: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-24T08:32:18.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:17 smithi177 ceph-mon[33046]: pgmap v346: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-24T08:32:18.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:17 smithi146 ceph-mon[27805]: pgmap v346: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-04-24T08:32:19.408 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-24T08:32:19.408 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:19.408 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 4, 2024-04-24T08:32:19.408 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 4, 2024-04-24T08:32:19.408 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 3024, 2024-04-24T08:32:19.408 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 448, 2024-04-24T08:32:19.408 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 4, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 0, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.001464345, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.000031116, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.000593401, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 3, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000780343, 2024-04-24T08:32:19.409 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000260114 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T08:32:19.410 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 2, 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 174, 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 320, 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 186928, 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 5, 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:19.411 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.010941309, 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.007309062, 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.001335435, 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 171, 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.026954088, 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000157626 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:19.412 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 186928 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_messages": 239, 2024-04-24T08:32:19.413 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages": 227, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_bytes": 1733056, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_bytes": 140064, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_created_connections": 2, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_active_connections": 1, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_total_time": 0.049948479, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_send_time": 0.014354348, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_recv_time": 0.040115455, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_running_fast_dispatch_time": 0.005796391, 2024-04-24T08:32:19.414 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 226, 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.012013059, 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000053155 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:19.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_recv_encrypted_bytes": 1733056, 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: "msgr_send_encrypted_bytes": 140064 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: "cct": { 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: "total_workers": 0, 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: "unhealthy_workers": 0 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool": { 2024-04-24T08:32:19.416 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:19.417 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:19.418 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:19.419 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_items": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:19.420 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_bytes": 148692, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_anon_items": 30, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bytes": 0, 2024-04-24T08:32:19.421 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_items": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_bytes": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_items": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:19.422 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "pgmap_items": 0, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_bytes": 928, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_co_items": 2, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_1_items": 0, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: "unittest_2_items": 0 2024-04-24T08:32:19.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-24T08:32:19.424 INFO:teuthology.orchestra.run.smithi153.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:19.424 INFO:teuthology.orchestra.run.smithi153.stdout: "val": 0, 2024-04-24T08:32:19.424 INFO:teuthology.orchestra.run.smithi153.stdout: "max": 104857600, 2024-04-24T08:32:19.424 INFO:teuthology.orchestra.run.smithi153.stdout: "get_started": 0, 2024-04-24T08:32:19.424 INFO:teuthology.orchestra.run.smithi153.stdout: "get": 241, 2024-04-24T08:32:19.424 INFO:teuthology.orchestra.run.smithi153.stdout: "get_sum": 1699352, 2024-04-24T08:32:19.425 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:19.425 INFO:teuthology.orchestra.run.smithi153.stdout: "get_or_fail_success": 241, 2024-04-24T08:32:19.425 INFO:teuthology.orchestra.run.smithi153.stdout: "take": 0, 2024-04-24T08:32:19.425 INFO:teuthology.orchestra.run.smithi153.stdout: "take_sum": 0, 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: "put": 241, 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: "put_sum": 1699352, 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: "wait": { 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: "avgcount": 0, 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: "sum": 0.000000000, 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: "avgtime": 0.000000000 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:19.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-24T08:32:19.427 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-24T08:32:20.145 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-24T08:32:20.146 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-24T08:32:20.146 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-24T08:32:20.146 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-24T08:32:20.146 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-24T08:32:20.146 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-04-24T08:32:20.146 DEBUG:tasks.check_counter:Getting stats from c 2024-04-24T08:32:20.146 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-24T08:32:20.397 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:32:20.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:20 smithi153 ceph-mon[32746]: pgmap v347: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:20.727 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:20 smithi177 ceph-mon[33046]: pgmap v347: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:20.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:20 smithi146 ceph-mon[27805]: pgmap v347: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:22.006 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 258, 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 243, 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1767554, 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 141588, 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-04-24T08:32:22.007 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.049725913, 2024-04-24T08:32:22.008 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.015753613, 2024-04-24T08:32:22.008 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.037629313, 2024-04-24T08:32:22.008 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.005418099, 2024-04-24T08:32:22.008 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:22.008 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 242, 2024-04-24T08:32:22.008 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.012710225, 2024-04-24T08:32:22.008 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000052521 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1766720, 2024-04-24T08:32:22.009 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 141264 2024-04-24T08:32:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 680, 2024-04-24T08:32:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 849, 2024-04-24T08:32:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 4362992, 2024-04-24T08:32:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 679129, 2024-04-24T08:32:22.010 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 7, 2024-04-24T08:32:22.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-04-24T08:32:22.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.079895752, 2024-04-24T08:32:22.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.029406451, 2024-04-24T08:32:22.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.018768378, 2024-04-24T08:32:22.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.023311737, 2024-04-24T08:32:22.011 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:22.011 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 846, 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.049361607, 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000058347 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:32:22.012 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 463520 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1652, 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1652, 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 8797201, 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 524570, 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-04-24T08:32:22.013 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.172880026, 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.052265533, 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.051452045, 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.053386516, 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1651, 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.071432897, 2024-04-24T08:32:22.014 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000043266 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T08:32:22.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3949, 2024-04-24T08:32:22.016 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.123367192, 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000031240 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.017 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-24T08:32:22.018 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-24T08:32:22.019 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 3118, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 4, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 3114, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 3113, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-24T08:32:22.020 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 84, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:32:22.021 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-24T08:32:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-24T08:32:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-24T08:32:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-24T08:32:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-24T08:32:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-24T08:32:22.022 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-24T08:32:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-24T08:32:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-24T08:32:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:32:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-24T08:32:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 0, 2024-04-24T08:32:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 0, 2024-04-24T08:32:22.023 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:32:22.024 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-24T08:32:22.025 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-24T08:32:22.026 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-24T08:32:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-24T08:32:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-24T08:32:22.027 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-24T08:32:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:32:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-24T08:32:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-24T08:32:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-24T08:32:22.028 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-24T08:32:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-24T08:32:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:32:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:32:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-24T08:32:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-24T08:32:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:32:22.029 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:32:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:32:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-24T08:32:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-24T08:32:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:32:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-24T08:32:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-24T08:32:22.030 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 7474, 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-24T08:32:22.031 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 10, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-24T08:32:22.032 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-24T08:32:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 16690792, 2024-04-24T08:32:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 16690218, 2024-04-24T08:32:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-24T08:32:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 7474 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 1426, 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 2610, 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 1184, 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 103, 2024-04-24T08:32:22.034 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 103, 2024-04-24T08:32:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 0, 2024-04-24T08:32:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 3118, 2024-04-24T08:32:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 3118, 2024-04-24T08:32:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 0, 2024-04-24T08:32:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-24T08:32:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-24T08:32:22.035 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 62568, 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-24T08:32:22.036 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.037 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-24T08:32:22.038 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-24T08:32:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.039 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-24T08:32:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.040 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-24T08:32:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.041 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.042 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-24T08:32:22.043 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-24T08:32:22.044 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.046 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-24T08:32:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.047 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-24T08:32:22.048 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.049 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-24T08:32:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-24T08:32:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.050 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-24T08:32:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.052 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-24T08:32:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.053 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.054 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-24T08:32:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-24T08:32:22.056 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.057 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:32:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:32:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-24T08:32:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 4, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 3, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 4, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 6898, 2024-04-24T08:32:22.059 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:22.060 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:22.061 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:22.062 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:22.063 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 250593, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 1472, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:22.064 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8128, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 169, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:22.065 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 5603698, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 45862, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:22.066 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 2327, 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 21, 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 3, 2024-04-24T08:32:22.067 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 2327, 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2327, 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1.653953145, 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000710766 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2327, 2024-04-24T08:32:22.068 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2328 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 2327, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 2327, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 1477, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-24T08:32:22.069 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 848, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-24T08:32:22.070 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 1, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:32:22.071 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-24T08:32:22.072 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-24T08:32:22.073 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-24T08:32:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-24T08:32:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-24T08:32:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-24T08:32:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-24T08:32:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 101, 2024-04-24T08:32:22.074 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 13, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 6, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 6, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-24T08:32:22.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-24T08:32:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-24T08:32:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:32:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-24T08:32:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-24T08:32:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.076 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-24T08:32:22.077 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 2586, 2024-04-24T08:32:22.078 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 14678812, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 2586, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2586, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 14678812, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.079 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:22.080 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2327, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 12498761, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2327, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 12498761, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.081 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:22.082 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2327, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 2327, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2327, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2327, 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:22.083 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:22.084 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:22.085 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x55ed3511c6e0": { 2024-04-24T08:32:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:22.086 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-24T08:32:22.087 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-24T08:32:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:22.088 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-24T08:32:22.544 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:22 smithi177 ceph-mon[33046]: pgmap v348: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:22.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:22 smithi153 ceph-mon[32746]: pgmap v348: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:22.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:22 smithi146 ceph-mon[27805]: pgmap v348: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:32:22.771 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-24T08:32:22.772 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-24T08:32:22.772 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-04-24T08:32:22.772 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:32:22.772 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:32:22.772 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-04-24T08:32:22.772 DEBUG:tasks.check_counter:Getting stats from f 2024-04-24T08:32:22.773 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-24T08:32:23.031 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:32:24.663 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:24 smithi177 ceph-mon[33046]: pgmap v349: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:32:24.664 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-24T08:32:24.664 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 237, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 225, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1732512, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 138784, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 0, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.051159475, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.014528621, 2024-04-24T08:32:24.665 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.040057401, 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.005895665, 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 224, 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.014533451, 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000064881 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:24.666 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:24.667 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:24.667 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:24.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.667 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1732512, 2024-04-24T08:32:24.667 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 138784 2024-04-24T08:32:24.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 172, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 320, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 186576, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.011383143, 2024-04-24T08:32:24.668 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.007768687, 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.001369207, 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 169, 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.024105037, 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000142633 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.669 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 186576 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.670 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 4, 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 4, 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 3024, 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 448, 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 2, 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.000804520, 2024-04-24T08:32:24.671 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.000042050, 2024-04-24T08:32:24.672 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000723489, 2024-04-24T08:32:24.672 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:24.672 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:24.672 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3, 2024-04-24T08:32:24.672 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000846020, 2024-04-24T08:32:24.673 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000282006 2024-04-24T08:32:24.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.673 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:24.673 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:24.674 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:24.674 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:24.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 0, 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.675 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:24.676 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:24.677 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:24.677 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:24.677 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:24.677 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:24.677 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:24.677 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:24.677 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:24.678 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:24.678 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:24.678 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:24.678 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:24.678 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:24.678 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:24.678 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:24.679 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:24.680 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:24.680 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-24T08:32:24.680 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:24.681 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:24.681 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:24.681 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:24.681 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 148995, 2024-04-24T08:32:24.681 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 31, 2024-04-24T08:32:24.682 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:24.682 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:24.682 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-24T08:32:24.682 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-24T08:32:24.682 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:24.682 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:24.682 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 928, 2024-04-24T08:32:24.683 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 2, 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:24.684 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 239, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 1699092, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 239, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 239, 2024-04-24T08:32:24.685 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1699092, 2024-04-24T08:32:24.686 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:24.686 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:24.686 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:24.686 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:24.686 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:24.686 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:24.686 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-24T08:32:24.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:24 smithi146 ceph-mon[27805]: pgmap v349: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:32:24.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:24 smithi153 ceph-mon[32746]: pgmap v349: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:32:25.288 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-04-24T08:32:25.289 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-04-24T08:32:25.289 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-04-24T08:32:25.289 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-04-24T08:32:25.289 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-04-24T08:32:25.289 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-04-24T08:32:25.289 DEBUG:tasks.check_counter:Getting stats from i 2024-04-24T08:32:25.290 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-24T08:32:25.530 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:32:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:26 smithi177 ceph-mon[33046]: pgmap v350: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:32:26.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:26 smithi146 ceph-mon[27805]: pgmap v350: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:32:26.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:26 smithi153 ceph-mon[32746]: pgmap v350: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:32:27.111 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-24T08:32:27.111 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:27.112 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2181, 2024-04-24T08:32:27.112 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2183, 2024-04-24T08:32:27.112 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 12176324, 2024-04-24T08:32:27.112 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 693568, 2024-04-24T08:32:27.112 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 7, 2024-04-24T08:32:27.112 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:27.112 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.207693060, 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.065493919, 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.058143562, 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.065438291, 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2181, 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.084621524, 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000038799 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.113 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.114 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 364, 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 349, 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 18571922, 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 174876, 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.067241676, 2024-04-24T08:32:27.115 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.018224893, 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.050768067, 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.008211499, 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 348, 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.019622765, 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000056387 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.116 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.117 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.117 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1766608, 2024-04-24T08:32:27.117 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 140784 2024-04-24T08:32:27.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.117 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:27.117 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 283, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 450, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 8460485, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 553671, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 7, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.041441410, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.016357897, 2024-04-24T08:32:27.118 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.011704660, 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.008798612, 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 448, 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.038825808, 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000086664 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:27.119 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 465584 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-24T08:32:27.120 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 4255, 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.126722292, 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000029781 2024-04-24T08:32:27.121 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.122 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.123 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.124 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-24T08:32:27.124 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-24T08:32:27.124 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-24T08:32:27.124 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-24T08:32:27.124 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-24T08:32:27.124 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 4938, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 0, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 4938, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 4938, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-24T08:32:27.125 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-24T08:32:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 84, 2024-04-24T08:32:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-24T08:32:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-24T08:32:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-24T08:32:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-24T08:32:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:32:27.126 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-24T08:32:27.127 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-24T08:32:27.128 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:32:27.128 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-24T08:32:27.128 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 3454, 2024-04-24T08:32:27.128 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 26625078, 2024-04-24T08:32:27.128 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 6, 2024-04-24T08:32:27.128 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:32:27.128 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:32:27.129 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-24T08:32:27.130 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-24T08:32:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-24T08:32:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-24T08:32:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-24T08:32:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-24T08:32:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-24T08:32:27.131 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-24T08:32:27.132 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-24T08:32:27.132 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-24T08:32:27.132 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:32:27.132 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-24T08:32:27.132 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-24T08:32:27.132 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-24T08:32:27.132 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:32:27.133 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-24T08:32:27.134 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-24T08:32:27.134 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:32:27.134 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-24T08:32:27.134 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-24T08:32:27.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.134 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-24T08:32:27.134 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-24T08:32:27.135 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-24T08:32:27.135 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-24T08:32:27.135 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-24T08:32:27.135 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 16815, 2024-04-24T08:32:27.135 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-24T08:32:27.135 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-24T08:32:27.135 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 20, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 3, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-24T08:32:27.136 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 40913224, 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 40913112, 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 16815 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.137 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-24T08:32:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 2231, 2024-04-24T08:32:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 6062, 2024-04-24T08:32:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 3831, 2024-04-24T08:32:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 240, 2024-04-24T08:32:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 272, 2024-04-24T08:32:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 32, 2024-04-24T08:32:27.138 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 4938, 2024-04-24T08:32:27.139 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 6762, 2024-04-24T08:32:27.139 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 1824, 2024-04-24T08:32:27.139 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-24T08:32:27.139 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-24T08:32:27.139 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-24T08:32:27.139 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 65752, 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-24T08:32:27.140 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.141 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.142 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.144 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.145 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-24T08:32:27.145 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.145 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.145 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.145 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.146 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.147 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-24T08:32:27.147 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.147 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.147 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.148 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-24T08:32:27.148 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.148 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.149 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.150 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-24T08:32:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.151 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.152 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.153 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.154 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.154 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-24T08:32:27.154 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.154 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.154 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.155 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.156 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.158 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-24T08:32:27.158 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.158 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.158 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.158 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-24T08:32:27.158 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 6, 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 4, 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 5, 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-24T08:32:27.161 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 4567, 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:27.162 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:27.163 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:27.164 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:27.165 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-24T08:32:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:27.166 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 289101, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 2299, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-24T08:32:27.167 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8128, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 169, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:27.168 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 8955271, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 76351, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.169 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 2565, 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 26, 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 4, 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 2565, 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-24T08:32:27.170 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2565, 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1.323962478, 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000516164 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2565, 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2566 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.171 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 2565, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 2565, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 1627, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 935, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-24T08:32:27.172 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-24T08:32:27.173 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 2, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-24T08:32:27.174 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-24T08:32:27.175 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-24T08:32:27.176 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 101, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 13, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 7, 2024-04-24T08:32:27.177 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 7, 2024-04-24T08:32:27.178 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-24T08:32:27.178 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-24T08:32:27.178 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-24T08:32:27.178 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-24T08:32:27.178 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-24T08:32:27.178 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.178 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-24T08:32:27.179 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.180 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-24T08:32:27.180 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-24T08:32:27.180 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:32:27.180 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-24T08:32:27.180 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-24T08:32:27.180 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-24T08:32:27.180 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-24T08:32:27.181 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-24T08:32:27.181 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.181 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:27.181 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:27.181 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-24T08:32:27.181 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:27.181 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 2824, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 38938382, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 2824, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2824, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 38938382, 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:27.182 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-24T08:32:27.183 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2565, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 36719227, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2565, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 36719227, 2024-04-24T08:32:27.184 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-24T08:32:27.185 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2565, 2024-04-24T08:32:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 2565, 2024-04-24T08:32:27.186 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2565, 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2565, 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:32:27.187 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:27.188 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-24T08:32:27.188 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:27.188 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:27.188 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:27.188 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:27.188 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:27.188 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x562be63b86e0": { 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:27.190 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:27.191 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:27.192 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.192 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:27.192 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-24T08:32:27.866 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-24T08:32:27.867 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-24T08:32:27.867 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-24T08:32:27.867 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:32:27.867 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:32:27.867 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-24T08:32:27.867 DEBUG:tasks.check_counter:Getting stats from l 2024-04-24T08:32:27.868 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid cca14746-0211-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-24T08:32:28.104 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:32:28.485 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:28 smithi177 ceph-mon[33046]: pgmap v351: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:32:28.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:28 smithi146 ceph-mon[27805]: pgmap v351: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:32:28.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:28 smithi153 ceph-mon[32746]: pgmap v351: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 89 B/s wr, 8 op/s 2024-04-24T08:32:29.642 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-24T08:32:29.642 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:32:29.642 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 0, 2024-04-24T08:32:29.642 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2, 2024-04-24T08:32:29.642 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 0, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 0, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.001044672, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.000149564, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:29.643 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:29.644 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 246, 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 232, 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1736896, 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 142432, 2024-04-24T08:32:29.645 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 2, 2024-04-24T08:32:29.646 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 1, 2024-04-24T08:32:29.646 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.051985496, 2024-04-24T08:32:29.646 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.016071583, 2024-04-24T08:32:29.646 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.040054035, 2024-04-24T08:32:29.646 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.005896535, 2024-04-24T08:32:29.646 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:29.646 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 232, 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.013746342, 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000059251 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:29.647 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1736896, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 142432 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 3, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 178, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 480, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 266048, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-24T08:32:29.648 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.012303945, 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.008296113, 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.001448985, 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 175, 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.028960631, 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000165489 2024-04-24T08:32:29.649 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.650 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:32:29.650 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:29.650 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:29.650 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:29.650 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.650 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-24T08:32:29.650 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 266048 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 0, 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:32:29.651 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-24T08:32:29.652 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:32:29.652 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:32:29.652 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:32:29.652 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:32:29.652 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:32:29.652 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:32:29.652 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-24T08:32:29.653 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:32:29.654 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:32:29.655 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 148692, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 30, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:32:29.656 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 928, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 2, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-24T08:32:29.657 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-24T08:32:29.658 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 245, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 1699769, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 245, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 245, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1699769, 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-24T08:32:29.659 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-24T08:32:29.660 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-24T08:32:29.660 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-24T08:32:29.660 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:29.660 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-24T08:32:29.660 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-24T08:32:30.315 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-24T08:32:30.315 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-24T08:32:30.315 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-24T08:32:30.315 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-24T08:32:30.315 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-24T08:32:30.315 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-04-24T08:32:30.316 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-24T08:32:30.327 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-24T08:32:30.327 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:32:30.327 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:32:30.385 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-04-24T08:32:30.385 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:32:30.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:32:30.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:30 smithi177 ceph-mon[33046]: pgmap v352: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1629585863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad0108e40 msgr2=0x7f8ad01152b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1629585863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad0108e40 0x7f8ad01152b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009be0 tx=0x7f8ac802a750 comp rx=0 tx=0).stop 2024-04-24T08:32:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1629585863 shutdown_connections 2024-04-24T08:32:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1629585863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad01084a0 0x7f8ad0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1629585863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad0108e40 0x7f8ad01152b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.643 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1629585863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ad010d3c0 0x7f8ad010d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1629585863 >> 172.21.15.146:0/1629585863 conn(0x7f8ad0077b80 msgr2=0x7f8ad0079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:30.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1629585863 shutdown_connections 2024-04-24T08:32:30.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.642+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1629585863 wait complete. 2024-04-24T08:32:30.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.643+0000 7f8ad4e00640 1 Processor -- start 2024-04-24T08:32:30.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.643+0000 7f8ad4e00640 1 -- start start 2024-04-24T08:32:30.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8ad4e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad01084a0 0x7f8ad01a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8ad4e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ad0108e40 0x7f8ad01a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8ad4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad010d3c0 0x7f8ad019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8ad4e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad01187e0 con 0x7f8ad0108e40 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8ad4e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad0118660 con 0x7f8ad01084a0 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8ad4e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad0118960 con 0x7f8ad010d3c0 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8acfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad010d3c0 0x7f8ad019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8ac7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ad0108e40 0x7f8ad01a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:30.646 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8acfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad010d3c0 0x7f8ad019ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:43524/0 (socket says 172.21.15.146:43524) 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8acf400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad01084a0 0x7f8ad01a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.644+0000 7f8acfe00640 1 -- 172.21.15.146:0/1599720124 learned_addr learned my addr 172.21.15.146:0/1599720124 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8acfe00640 1 -- 172.21.15.146:0/1599720124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad01084a0 msgr2=0x7f8ad01a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8acfe00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad01084a0 0x7f8ad01a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8acfe00640 1 -- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ad0108e40 msgr2=0x7f8ad01a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8acfe00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ad0108e40 0x7f8ad01a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8acfe00640 1 -- 172.21.15.146:0/1599720124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad019f200 con 0x7f8ad010d3c0 2024-04-24T08:32:30.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8acfe00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad010d3c0 0x7f8ad019ea80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009b30 tx=0x7f8ac80305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:30.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8accc00640 1 -- 172.21.15.146:0/1599720124 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8041020 con 0x7f8ad010d3c0 2024-04-24T08:32:30.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8accc00640 1 -- 172.21.15.146:0/1599720124 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ac8003e50 con 0x7f8ad010d3c0 2024-04-24T08:32:30.648 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8accc00640 1 -- 172.21.15.146:0/1599720124 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8036040 con 0x7f8ad010d3c0 2024-04-24T08:32:30.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.645+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad019f490 con 0x7f8ad010d3c0 2024-04-24T08:32:30.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.646+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad01ab790 con 0x7f8ad010d3c0 2024-04-24T08:32:30.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.647+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad0069190 con 0x7f8ad010d3c0 2024-04-24T08:32:30.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.649+0000 7f8accc00640 1 -- 172.21.15.146:0/1599720124 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f8ac8044020 con 0x7f8ad010d3c0 2024-04-24T08:32:30.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.650+0000 7f8accc00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ab40a3d20 0x7f8ab40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:30.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.650+0000 7f8accc00640 1 -- 172.21.15.146:0/1599720124 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7f8ac80f14b0 con 0x7f8ad010d3c0 2024-04-24T08:32:30.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.650+0000 7f8acf400640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ab40a3d20 0x7f8ab40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:30.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.651+0000 7f8acf400640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ab40a3d20 0x7f8ab40a61e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00096c0 tx=0x7f8ac0009260 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:30.654 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.654+0000 7f8accc00640 1 -- 172.21.15.146:0/1599720124 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ac804b020 con 0x7f8ad010d3c0 2024-04-24T08:32:30.742 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:30 smithi146 ceph-mon[27805]: pgmap v352: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:30.858 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.858+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ad0110960 con 0x7f8ad010d3c0 2024-04-24T08:32:30.860 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.860+0000 7f8accc00640 1 -- 172.21.15.146:0/1599720124 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+23161 (secure 0 0 0) 0x7f8ac80bdae0 con 0x7f8ad010d3c0 2024-04-24T08:32:30.861 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:32:30.861 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":101,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:32:15.934761+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:0/4151754658":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2355630587":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6800/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/393585939":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/4023745007":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:32:30.863 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.862+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ab40a3d20 msgr2=0x7f8ab40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.862+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ab40a3d20 0x7f8ab40a61e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00096c0 tx=0x7f8ac0009260 comp rx=0 tx=0).stop 2024-04-24T08:32:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad010d3c0 msgr2=0x7f8ad019ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad010d3c0 0x7f8ad019ea80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009b30 tx=0x7f8ac80305d0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 shutdown_connections 2024-04-24T08:32:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ab40a3d20 0x7f8ab40a61e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ad010d3c0 0x7f8ad019ea80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.864 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ad0108e40 0x7f8ad01a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 --2- 172.21.15.146:0/1599720124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ad01084a0 0x7f8ad01a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 >> 172.21.15.146:0/1599720124 conn(0x7f8ad0077b80 msgr2=0x7f8ad00784d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 shutdown_connections 2024-04-24T08:32:30.865 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:30.863+0000 7f8ad4e00640 1 -- 172.21.15.146:0/1599720124 wait complete. 2024-04-24T08:32:30.882 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:13:40.471709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:44.805814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:45.883831+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 100, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:32:30.882 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:32:30.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:30 smithi153 ceph-mon[32746]: pgmap v352: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:31.089 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 -- 172.21.15.146:0/3909541258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b88100e10 msgr2=0x7f4b881011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 --2- 172.21.15.146:0/3909541258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b88100e10 0x7f4b881011f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009900 tx=0x7f4b7802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 -- 172.21.15.146:0/3909541258 shutdown_connections 2024-04-24T08:32:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 --2- 172.21.15.146:0/3909541258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b8810e9c0 0x7f4b88110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 --2- 172.21.15.146:0/3909541258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88101730 0x7f4b8810e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 --2- 172.21.15.146:0/3909541258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b88100e10 0x7f4b881011f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.090 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 -- 172.21.15.146:0/3909541258 >> 172.21.15.146:0/3909541258 conn(0x7f4b88077280 msgr2=0x7f4b88077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:31.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.088+0000 7f4b90400640 1 -- 172.21.15.146:0/3909541258 shutdown_connections 2024-04-24T08:32:31.091 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.089+0000 7f4b90400640 1 -- 172.21.15.146:0/3909541258 wait complete. 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.089+0000 7f4b90400640 1 Processor -- start 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b90400640 1 -- start start 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b90400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88100e10 0x7f4b881a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b90400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b88101730 0x7f4b881a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b90400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b8810e9c0 0x7f4b8819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b90400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f4b881145d0 con 0x7f4b88100e10 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b90400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88114450 con 0x7f4b8810e9c0 2024-04-24T08:32:31.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b90400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88114750 con 0x7f4b88101730 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b8da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88100e10 0x7f4b881a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b8da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88100e10 0x7f4b881a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46738/0 (socket says 172.21.15.146:46738) 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b8da00640 1 -- 172.21.15.146:0/3857267986 learned_addr learned my addr 172.21.15.146:0/3857267986 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b8d000640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b88101730 0x7f4b881a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.090+0000 7f4b8e400640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b8810e9c0 0x7f4b8819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b8da00640 1 -- 172.21.15.146:0/3857267986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b88101730 msgr2=0x7f4b881a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b8da00640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b88101730 0x7f4b881a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b8da00640 1 -- 172.21.15.146:0/3857267986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b8810e9c0 msgr2=0x7f4b8819a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b8da00640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b8810e9c0 0x7f4b8819a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b8da00640 1 -- 172.21.15.146:0/3857267986 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8819b0b0 con 0x7f4b88100e10 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b8d000640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b88101730 0x7f4b881a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b8da00640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88100e10 0x7f4b881a0780 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4b78006010 tx=0x7f4b780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.091+0000 7f4b86000640 1 -- 172.21.15.146:0/3857267986 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7803f030 con 0x7f4b88100e10 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.092+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8819b340 con 0x7f4b88100e10 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.092+0000 7f4b86000640 1 -- 172.21.15.146:0/3857267986 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b78031990 con 0x7f4b88100e10 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.092+0000 7f4b86000640 1 -- 172.21.15.146:0/3857267986 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78035070 con 0x7f4b88100e10 2024-04-24T08:32:31.095 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.092+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b881a7640 con 0x7f4b88100e10 2024-04-24T08:32:31.096 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.093+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b8806ada0 con 0x7f4b88100e10 2024-04-24T08:32:31.097 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.096+0000 7f4b86000640 1 -- 172.21.15.146:0/3857267986 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f4b78042030 con 0x7f4b88100e10 2024-04-24T08:32:31.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.098+0000 7f4b86000640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4b6c0a3d20 0x7f4b6c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.098+0000 7f4b86000640 1 -- 172.21.15.146:0/3857267986 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7f4b780f0e70 con 0x7f4b88100e10 2024-04-24T08:32:31.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.098+0000 7f4b8d000640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4b6c0a3d20 0x7f4b6c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.099+0000 7f4b8d000640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4b6c0a3d20 0x7f4b6c0a61e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4b88100750 tx=0x7f4b74005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:31.101 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.100+0000 7f4b86000640 1 -- 172.21.15.146:0/3857267986 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b780bcf10 con 0x7f4b88100e10 2024-04-24T08:32:31.302 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.301+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4b88109b50 con 0x7f4b88100e10 2024-04-24T08:32:31.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.302+0000 7f4b86000640 1 -- 172.21.15.146:0/3857267986 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f4b780c1210 con 0x7f4b88100e10 2024-04-24T08:32:31.303 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 1 2024-04-24T08:32:31.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4b6c0a3d20 msgr2=0x7f4b6c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4b6c0a3d20 0x7f4b6c0a61e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4b88100750 tx=0x7f4b74005f90 comp rx=0 tx=0).stop 2024-04-24T08:32:31.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88100e10 msgr2=0x7f4b881a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88100e10 0x7f4b881a0780 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4b78006010 tx=0x7f4b780042f0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 shutdown_connections 2024-04-24T08:32:31.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f4b6c0a3d20 0x7f4b6c0a61e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b88101730 0x7f4b881a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4b8810e9c0 0x7f4b8819a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 --2- 172.21.15.146:0/3857267986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b88100e10 0x7f4b881a0780 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 >> 172.21.15.146:0/3857267986 conn(0x7f4b88077280 msgr2=0x7f4b88110010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 shutdown_connections 2024-04-24T08:32:31.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.305+0000 7f4b90400640 1 -- 172.21.15.146:0/3857267986 wait complete. 2024-04-24T08:32:31.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:32:31.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.522+0000 7f8c6fe00640 1 -- 172.21.15.146:0/269365714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c70104cd0 msgr2=0x7f8c70108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.522+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/269365714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c70104cd0 0x7f8c70108d10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009900 tx=0x7f8c5802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.523+0000 7f8c6fe00640 1 -- 172.21.15.146:0/269365714 shutdown_connections 2024-04-24T08:32:31.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.523+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/269365714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c701093f0 0x7f8c70110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.523+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/269365714 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c70104cd0 0x7f8c70108d10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.524 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.523+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/269365714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c70104320 0x7f8c70104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.523+0000 7f8c6fe00640 1 -- 172.21.15.146:0/269365714 >> 172.21.15.146:0/269365714 conn(0x7f8c700ffb40 msgr2=0x7f8c70101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:31.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.523+0000 7f8c6fe00640 1 -- 172.21.15.146:0/269365714 shutdown_connections 2024-04-24T08:32:31.525 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.523+0000 7f8c6fe00640 1 -- 172.21.15.146:0/269365714 wait complete. 2024-04-24T08:32:31.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 Processor -- start 2024-04-24T08:32:31.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 -- start start 2024-04-24T08:32:31.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c70104320 0x7f8c701a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c701093f0 0x7f8c701a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c7019a810 0x7f8c7019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8c701144b0 con 0x7f8c70104320 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70114330 con 0x7f8c701093f0 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.524+0000 7f8c6fe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70114630 con 0x7f8c7019a810 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c70104320 0x7f8c701a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c7019a810 0x7f8c7019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c7019a810 0x7f8c7019acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:35862/0 (socket says 172.21.15.146:35862) 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 -- 172.21.15.146:0/2871697954 learned_addr learned my addr 172.21.15.146:0/2871697954 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c66000640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c701093f0 0x7f8c701a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.528 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 -- 172.21.15.146:0/2871697954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c701093f0 msgr2=0x7f8c701a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c701093f0 0x7f8c701a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 -- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c70104320 msgr2=0x7f8c701a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c70104320 0x7f8c701a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.525+0000 7f8c6f400640 1 -- 172.21.15.146:0/2871697954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7019b3a0 con 0x7f8c7019a810 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.526+0000 7f8c6f400640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c7019a810 0x7f8c7019acc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8c6800efd0 tx=0x7f8c6800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.526+0000 7f8c74a00640 1 -- 172.21.15.146:0/2871697954 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68003b70 con 0x7f8c7019a810 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.526+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7019b690 con 0x7f8c7019a810 2024-04-24T08:32:31.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.526+0000 7f8c74a00640 1 -- 172.21.15.146:0/2871697954 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c68003d10 con 0x7f8c7019a810 2024-04-24T08:32:31.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.526+0000 7f8c74a00640 1 -- 172.21.15.146:0/2871697954 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c680109f0 con 0x7f8c7019a810 2024-04-24T08:32:31.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.526+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c701a77f0 con 0x7f8c7019a810 2024-04-24T08:32:31.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.527+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c701057f0 con 0x7f8c7019a810 2024-04-24T08:32:31.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.529+0000 7f8c74a00640 1 -- 172.21.15.146:0/2871697954 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f8c68004020 con 0x7f8c7019a810 2024-04-24T08:32:31.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.530+0000 7f8c74a00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8c540a3d20 0x7f8c540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.530+0000 7f8c74a00640 1 -- 172.21.15.146:0/2871697954 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7f8c680d0c30 con 0x7f8c7019a810 2024-04-24T08:32:31.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.531+0000 7f8c6ea00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8c540a3d20 0x7f8c540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.532 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.531+0000 7f8c6ea00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8c540a3d20 0x7f8c540a61e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8c600095f0 tx=0x7f8c60005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:31.535 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.534+0000 7f8c74a00640 1 -- 172.21.15.146:0/2871697954 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c7019b690 con 0x7f8c7019a810 2024-04-24T08:32:31.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:31 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1599720124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:32:31.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:31 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3857267986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:32:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.742+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8c70104cd0 con 0x7f8c7019a810 2024-04-24T08:32:31.744 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.742+0000 7f8c74a00640 1 -- 172.21.15.146:0/2871697954 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) ==== 104+0+11 (secure 0 0 0) 0x7f8c6809d260 con 0x7f8c7019a810 2024-04-24T08:32:31.744 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 32 2024-04-24T08:32:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.745+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8c540a3d20 msgr2=0x7f8c540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.746 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.745+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8c540a3d20 0x7f8c540a61e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8c600095f0 tx=0x7f8c60005f90 comp rx=0 tx=0).stop 2024-04-24T08:32:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.745+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c7019a810 msgr2=0x7f8c7019acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.745+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c7019a810 0x7f8c7019acc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8c6800efd0 tx=0x7f8c6800c6a0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 shutdown_connections 2024-04-24T08:32:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8c540a3d20 0x7f8c540a61e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c7019a810 0x7f8c7019acc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8c701093f0 0x7f8c701a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.747 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 --2- 172.21.15.146:0/2871697954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c70104320 0x7f8c701a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 >> 172.21.15.146:0/2871697954 conn(0x7f8c700ffb40 msgr2=0x7f8c70100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:31.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 shutdown_connections 2024-04-24T08:32:31.748 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.746+0000 7f8c6fe00640 1 -- 172.21.15.146:0/2871697954 wait complete. 2024-04-24T08:32:31.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:32:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:31 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1599720124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:32:31.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:31 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3857267986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:32:31.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.956+0000 7f337a000640 1 -- 172.21.15.146:0/1670720899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076530 msgr2=0x7f3374076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.956+0000 7f337a000640 1 --2- 172.21.15.146:0/1670720899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076530 0x7f3374076910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3368009900 tx=0x7f336802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.956+0000 7f337a000640 1 -- 172.21.15.146:0/1670720899 shutdown_connections 2024-04-24T08:32:31.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.956+0000 7f337a000640 1 --2- 172.21.15.146:0/1670720899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374112d40 0x7f3374115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.956+0000 7f337a000640 1 --2- 172.21.15.146:0/1670720899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374076e50 0x7f3374112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.956+0000 7f337a000640 1 --2- 172.21.15.146:0/1670720899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076530 0x7f3374076910 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.956+0000 7f337a000640 1 -- 172.21.15.146:0/1670720899 >> 172.21.15.146:0/1670720899 conn(0x7f33740ffb70 msgr2=0x7f3374101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:31.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.957+0000 7f337a000640 1 -- 172.21.15.146:0/1670720899 shutdown_connections 2024-04-24T08:32:31.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.957+0000 7f337a000640 1 -- 172.21.15.146:0/1670720899 wait complete. 2024-04-24T08:32:31.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 Processor -- start 2024-04-24T08:32:31.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 -- start start 2024-04-24T08:32:31.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374076530 0x7f337419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076e50 0x7f337419f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374112d40 0x7f33741a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f33741188e0 con 0x7f3374076e50 2024-04-24T08:32:31.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3374118760 con 0x7f3374112d40 2024-04-24T08:32:31.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f337a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3374118a60 con 0x7f3374076530 2024-04-24T08:32:31.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f3378c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374076530 0x7f337419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f3378c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374076530 0x7f337419ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:35884/0 (socket says 172.21.15.146:35884) 2024-04-24T08:32:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f3378c00640 1 -- 172.21.15.146:0/3978837804 learned_addr learned my addr 172.21.15.146:0/3978837804 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.958+0000 7f3373e00640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076e50 0x7f337419f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3379600640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374112d40 0x7f33741a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3378c00640 1 -- 172.21.15.146:0/3978837804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374112d40 msgr2=0x7f33741a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3378c00640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374112d40 0x7f33741a3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3378c00640 1 -- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076e50 msgr2=0x7f337419f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3378c00640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076e50 0x7f337419f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3378c00640 1 -- 172.21.15.146:0/3978837804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33741a3da0 con 0x7f3374076530 2024-04-24T08:32:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3379600640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374112d40 0x7f33741a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:32:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3378c00640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374076530 0x7f337419ee00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f336802a9e0 tx=0x7f3368003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3371600640 1 -- 172.21.15.146:0/3978837804 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336803f070 con 0x7f3374076530 2024-04-24T08:32:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33741a4030 con 0x7f3374076530 2024-04-24T08:32:31.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3371600640 1 -- 172.21.15.146:0/3978837804 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3368031c30 con 0x7f3374076530 2024-04-24T08:32:31.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.959+0000 7f3371600640 1 -- 172.21.15.146:0/3978837804 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368035030 con 0x7f3374076530 2024-04-24T08:32:31.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.960+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33741ab920 con 0x7f3374076530 2024-04-24T08:32:31.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.961+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3374069190 con 0x7f3374076530 2024-04-24T08:32:31.967 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:31 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1599720124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:32:31.967 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:31 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3857267986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:32:31.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.963+0000 7f3371600640 1 -- 172.21.15.146:0/3978837804 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3368042070 con 0x7f3374076530 2024-04-24T08:32:31.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.964+0000 7f3371600640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f335c0a3d20 0x7f335c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:31.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.964+0000 7f3371600640 1 -- 172.21.15.146:0/3978837804 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7f33680f15d0 con 0x7f3374076530 2024-04-24T08:32:31.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.964+0000 7f3373e00640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f335c0a3d20 0x7f335c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:31.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.965+0000 7f3373e00640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f335c0a3d20 0x7f335c0a61e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3374079130 tx=0x7f3360005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:31.968 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:31.966+0000 7f3371600640 1 -- 172.21.15.146:0/3978837804 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3368031de0 con 0x7f3374076530 2024-04-24T08:32:32.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.168+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f337410ded0 con 0x7f3374076530 2024-04-24T08:32:32.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.170+0000 7f3371600640 1 -- 172.21.15.146:0/3978837804 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) ==== 100+0+11 (secure 0 0 0) 0x7f33680bdc00 con 0x7f3374076530 2024-04-24T08:32:32.171 INFO:teuthology.orchestra.run.smithi146.stdout:pg_num: 64 2024-04-24T08:32:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f335c0a3d20 msgr2=0x7f335c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:32.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f335c0a3d20 0x7f335c0a61e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3374079130 tx=0x7f3360005f90 comp rx=0 tx=0).stop 2024-04-24T08:32:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374076530 msgr2=0x7f337419ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374076530 0x7f337419ee00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f336802a9e0 tx=0x7f3368003c10 comp rx=0 tx=0).stop 2024-04-24T08:32:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 shutdown_connections 2024-04-24T08:32:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f335c0a3d20 0x7f335c0a61e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3374112d40 0x7f33741a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3374076e50 0x7f337419f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.172+0000 7f337a000640 1 --2- 172.21.15.146:0/3978837804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3374076530 0x7f337419ee00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.173+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 >> 172.21.15.146:0/3978837804 conn(0x7f33740ffb70 msgr2=0x7f3374101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.173+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 shutdown_connections 2024-04-24T08:32:32.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.173+0000 7f337a000640 1 -- 172.21.15.146:0/3978837804 wait complete. 2024-04-24T08:32:32.193 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:32:32.409 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 -- 172.21.15.146:0/515861747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 msgr2=0x7f84c0110d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 --2- 172.21.15.146:0/515861747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c0110d80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009ca0 tx=0x7f84b802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 -- 172.21.15.146:0/515861747 shutdown_connections 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 --2- 172.21.15.146:0/515861747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84c0101700 0x7f84c010e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 --2- 172.21.15.146:0/515861747 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c0110d80 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 --2- 172.21.15.146:0/515861747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c0100de0 0x7f84c01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 -- 172.21.15.146:0/515861747 >> 172.21.15.146:0/515861747 conn(0x7f84c0077280 msgr2=0x7f84c0077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 -- 172.21.15.146:0/515861747 shutdown_connections 2024-04-24T08:32:32.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.408+0000 7f84c6800640 1 -- 172.21.15.146:0/515861747 wait complete. 2024-04-24T08:32:32.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.409+0000 7f84c6800640 1 Processor -- start 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84c6800640 1 -- start start 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84c6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84c0100de0 0x7f84c01a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84c6800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c0101700 0x7f84c01a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84c6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84c6800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f84c01145a0 con 0x7f84c010e990 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84c6800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84c0114420 con 0x7f84c0101700 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84c6800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84c0114720 con 0x7f84c0100de0 2024-04-24T08:32:32.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.410+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46782/0 (socket says 172.21.15.146:46782) 2024-04-24T08:32:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 -- 172.21.15.146:0/1561561051 learned_addr learned my addr 172.21.15.146:0/1561561051 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bea00640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c0101700 0x7f84c01a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bf400640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84c0100de0 0x7f84c01a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 -- 172.21.15.146:0/1561561051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84c0100de0 msgr2=0x7f84c01a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84c0100de0 0x7f84c01a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 -- 172.21.15.146:0/1561561051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c0101700 msgr2=0x7f84c01a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c0101700 0x7f84c01a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 -- 172.21.15.146:0/1561561051 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c019b0c0 con 0x7f84c010e990 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bea00640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c0101700 0x7f84c01a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.411+0000 7f84bfe00640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c019a940 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f84b802ade0 tx=0x7f84b80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.412+0000 7f84b7e00640 1 -- 172.21.15.146:0/1561561051 <== mon.0 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8003b70 con 0x7f84c010e990 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.412+0000 7f84b7e00640 1 -- 172.21.15.146:0/1561561051 <== mon.0 v2:172.21.15.146:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84b8003d10 con 0x7f84c010e990 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.412+0000 7f84b7e00640 1 -- 172.21.15.146:0/1561561051 <== mon.0 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b802fc40 con 0x7f84c010e990 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.412+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c019b350 con 0x7f84c010e990 2024-04-24T08:32:32.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.412+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c01a7650 con 0x7f84c010e990 2024-04-24T08:32:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.413+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c006ae20 con 0x7f84c010e990 2024-04-24T08:32:32.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.415+0000 7f84b7e00640 1 -- 172.21.15.146:0/1561561051 <== mon.0 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f84b8004050 con 0x7f84c010e990 2024-04-24T08:32:32.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.417+0000 7f84b7e00640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f84a40a3da0 0x7f84a40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:32.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.417+0000 7f84b7e00640 1 -- 172.21.15.146:0/1561561051 <== mon.0 v2:172.21.15.146:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7f84b80f1800 con 0x7f84c010e990 2024-04-24T08:32:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.417+0000 7f84bf400640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f84a40a3da0 0x7f84a40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:32.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.418+0000 7f84bf400640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f84a40a3da0 0x7f84a40a6260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84c019ba80 tx=0x7f84b0005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:32.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.421+0000 7f84b7e00640 1 -- 172.21.15.146:0/1561561051 <== mon.0 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84b80be0c0 con 0x7f84c010e990 2024-04-24T08:32:32.639 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.639+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f84c0069270 con 0x7f84c010e990 2024-04-24T08:32:32.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.640+0000 7f84b7e00640 1 -- 172.21.15.146:0/1561561051 <== mon.0 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v101) ==== 81+0+956 (secure 0 0 0) 0x7f84b80bd9c0 con 0x7f84c010e990 2024-04-24T08:32:32.641 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/4151754658 2024-04-25T08:21:11.077472+0000 2024-04-24T08:32:32.641 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6801/1475219332 2024-04-25T08:21:11.077472+0000 2024-04-24T08:32:32.641 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/2355630587 2024-04-25T08:21:11.077472+0000 2024-04-24T08:32:32.641 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6800/1475219332 2024-04-25T08:21:11.077472+0000 2024-04-24T08:32:32.641 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/393585939 2024-04-25T08:21:11.077472+0000 2024-04-24T08:32:32.641 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6801/888738583 2024-04-25T08:10:56.306756+0000 2024-04-24T08:32:32.641 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/3118900567 2024-04-25T08:10:56.306756+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6800/520435616 2024-04-25T08:10:22.202345+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/1190010075 2024-04-25T08:10:56.306756+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6800/888738583 2024-04-25T08:10:56.306756+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/3978832813 2024-04-25T08:10:56.306756+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/2983645773 2024-04-25T08:10:22.202345+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/4023745007 2024-04-25T08:21:11.077472+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/2776821726 2024-04-25T08:10:22.202345+0000 2024-04-24T08:32:32.642 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:6801/520435616 2024-04-25T08:10:22.202345+0000 2024-04-24T08:32:32.643 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146:0/677277692 2024-04-25T08:10:22.202345+0000 2024-04-24T08:32:32.643 INFO:teuthology.orchestra.run.smithi146.stderr:listed 16 entries 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.642+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f84a40a3da0 msgr2=0x7f84a40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.642+0000 7f84c6800640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f84a40a3da0 0x7f84a40a6260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84c019ba80 tx=0x7f84b0005f90 comp rx=0 tx=0).stop 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.642+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 msgr2=0x7f84c019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c019a940 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f84b802ade0 tx=0x7f84b80319f0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 shutdown_connections 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f84a40a3da0 0x7f84a40a6260 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c010e990 0x7f84c019a940 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.644 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84c0101700 0x7f84c01a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 --2- 172.21.15.146:0/1561561051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84c0100de0 0x7f84c01a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 >> 172.21.15.146:0/1561561051 conn(0x7f84c0077280 msgr2=0x7f84c010ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 shutdown_connections 2024-04-24T08:32:32.645 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:32.643+0000 7f84c6800640 1 -- 172.21.15.146:0/1561561051 wait complete. 2024-04-24T08:32:32.663 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-24T08:32:32.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:32:32.664 DEBUG:teuthology.orchestra.run.smithi146:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-24T08:32:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:32 smithi177 ceph-mon[33046]: pgmap v353: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:32 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/2871697954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:32:32.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:32 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/3978837804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:32:32.835 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:32 smithi146 ceph-mon[27805]: pgmap v353: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:32.835 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:32 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/2871697954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:32:32.835 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:32 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/3978837804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:32:32.839 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:32:32.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:32:32.839 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:32:32.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:32:32.903 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-04-24T08:32:32.940 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:32:32.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:32:32.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:32:32.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:32 smithi153 ceph-mon[32746]: pgmap v353: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:32:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:32 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/2871697954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:32:32.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:32 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/3978837804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:32:33.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:32:33.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-04-24T08:32:33.157 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-24T08:32:33.168 INFO:tasks.cephadm:Teardown begin 2024-04-24T08:32:33.169 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:32:33.169 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:32:33.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 -- 172.21.15.146:0/730816375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 msgr2=0x7fdbf010d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/730816375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf010d7a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009900 tx=0x7fdbe002a480 comp rx=0 tx=0).stop 2024-04-24T08:32:33.352 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 -- 172.21.15.146:0/730816375 shutdown_connections 2024-04-24T08:32:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/730816375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0108e40 0x7fdbf01152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/730816375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbf01084a0 0x7fdbf0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/730816375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf010d7a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 -- 172.21.15.146:0/730816375 >> 172.21.15.146:0/730816375 conn(0x7fdbf0077b80 msgr2=0x7fdbf0079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 -- 172.21.15.146:0/730816375 shutdown_connections 2024-04-24T08:32:33.353 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.351+0000 7fdbf5c00640 1 -- 172.21.15.146:0/730816375 wait complete. 2024-04-24T08:32:33.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.352+0000 7fdbf5c00640 1 Processor -- start 2024-04-24T08:32:33.355 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5c00640 1 -- start start 2024-04-24T08:32:33.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbf01084a0 0x7fdbf01a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0108e40 0x7fdbf01a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf0118800 con 0x7fdbf01084a0 2024-04-24T08:32:33.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf0118680 con 0x7fdbf010d3c0 2024-04-24T08:32:33.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf0118980 con 0x7fdbf0108e40 2024-04-24T08:32:33.356 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbefe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbf01084a0 0x7fdbf01a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbefe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbf01084a0 0x7fdbf01a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46802/0 (socket says 172.21.15.146:46802) 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbf5200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48892/0 (socket says 172.21.15.146:48892) 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0108e40 0x7fdbf01a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.353+0000 7fdbefe00640 1 -- 172.21.15.146:0/1544587535 learned_addr learned my addr 172.21.15.146:0/1544587535 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5200640 1 -- 172.21.15.146:0/1544587535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0108e40 msgr2=0x7fdbf01a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5200640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0108e40 0x7fdbf01a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.357 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5200640 1 -- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbf01084a0 msgr2=0x7fdbf01a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5200640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbf01084a0 0x7fdbf01a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5200640 1 -- 172.21.15.146:0/1544587535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf019f2c0 con 0x7fdbf010d3c0 2024-04-24T08:32:33.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5200640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf019eb40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009fd0 tx=0x7fdbe800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:33.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbecc00640 1 -- 172.21.15.146:0/1544587535 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe800ce20 con 0x7fdbf010d3c0 2024-04-24T08:32:33.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbecc00640 1 -- 172.21.15.146:0/1544587535 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe8005020 con 0x7fdbf010d3c0 2024-04-24T08:32:33.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbecc00640 1 -- 172.21.15.146:0/1544587535 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8010720 con 0x7fdbf010d3c0 2024-04-24T08:32:33.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf019f5b0 con 0x7fdbf010d3c0 2024-04-24T08:32:33.359 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.354+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf01ab800 con 0x7fdbf010d3c0 2024-04-24T08:32:33.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.356+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbf0069190 con 0x7fdbf010d3c0 2024-04-24T08:32:33.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.358+0000 7fdbecc00640 1 -- 172.21.15.146:0/1544587535 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7fdbe8005540 con 0x7fdbf010d3c0 2024-04-24T08:32:33.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.359+0000 7fdbecc00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdbd40a3da0 0x7fdbd40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.359+0000 7fdbecc00640 1 -- 172.21.15.146:0/1544587535 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7fdbe80d0ee0 con 0x7fdbf010d3c0 2024-04-24T08:32:33.360 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.359+0000 7fdbefe00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdbd40a3da0 0x7fdbd40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.360+0000 7fdbefe00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdbd40a3da0 0x7fdbd40a6260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00096c0 tx=0x7fdbe0005a20 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:33.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.361+0000 7fdbecc00640 1 -- 172.21.15.146:0/1544587535 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbe809d510 con 0x7fdbf010d3c0 2024-04-24T08:32:33.565 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:33 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1561561051' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T08:32:33.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.606+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbf019ffe0 con 0x7fdbf010d3c0 2024-04-24T08:32:33.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.607+0000 7fdbecc00640 1 -- 172.21.15.146:0/1544587535 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7fdbf019ffe0 con 0x7fdbf010d3c0 2024-04-24T08:32:33.608 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:32:33.608 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:32:33.609 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:32:33.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdbd40a3da0 msgr2=0x7fdbd40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdbd40a3da0 0x7fdbd40a6260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00096c0 tx=0x7fdbe0005a20 comp rx=0 tx=0).stop 2024-04-24T08:32:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 msgr2=0x7fdbf019eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf019eb40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009fd0 tx=0x7fdbe800ef10 comp rx=0 tx=0).stop 2024-04-24T08:32:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 shutdown_connections 2024-04-24T08:32:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7fdbd40a3da0 0x7fdbd40a6260 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf0108e40 0x7fdbf01a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdbf010d3c0 0x7fdbf019eb40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 --2- 172.21.15.146:0/1544587535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbf01084a0 0x7fdbf01a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 >> 172.21.15.146:0/1544587535 conn(0x7fdbf0077b80 msgr2=0x7fdbf0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:33.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 shutdown_connections 2024-04-24T08:32:33.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.610+0000 7fdbf5c00640 1 -- 172.21.15.146:0/1544587535 wait complete. 2024-04-24T08:32:33.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:32:33.837 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 -- 172.21.15.146:0/1078527595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f141084a0 msgr2=0x7f8f14108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 --2- 172.21.15.146:0/1078527595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f141084a0 0x7f8f14108900 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009900 tx=0x7f8efc02a480 comp rx=0 tx=0).stop 2024-04-24T08:32:33.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 -- 172.21.15.146:0/1078527595 shutdown_connections 2024-04-24T08:32:33.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 --2- 172.21.15.146:0/1078527595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f14108e40 0x7f8f141152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 --2- 172.21.15.146:0/1078527595 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f141084a0 0x7f8f14108900 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.838 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 --2- 172.21.15.146:0/1078527595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f1410d3c0 0x7f8f1410d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 -- 172.21.15.146:0/1078527595 >> 172.21.15.146:0/1078527595 conn(0x7f8f14077b80 msgr2=0x7f8f14079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:33.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 -- 172.21.15.146:0/1078527595 shutdown_connections 2024-04-24T08:32:33.841 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.836+0000 7f8f18e00640 1 -- 172.21.15.146:0/1078527595 wait complete. 2024-04-24T08:32:33.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.837+0000 7f8f18e00640 1 Processor -- start 2024-04-24T08:32:33.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f18e00640 1 -- start start 2024-04-24T08:32:33.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f18e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141084a0 0x7f8f1419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f18e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f14108e40 0x7f8f1419f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f18e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f1410d3c0 0x7f8f141a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.845 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f18e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f8f14118850 con 0x7f8f1410d3c0 2024-04-24T08:32:33.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f18e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8f141186d0 con 0x7f8f14108e40 2024-04-24T08:32:33.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f18e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8f141189d0 con 0x7f8f141084a0 2024-04-24T08:32:33.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141084a0 0x7f8f1419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f0be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f14108e40 0x7f8f1419f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141084a0 0x7f8f1419ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:35942/0 (socket says 172.21.15.146:35942) 2024-04-24T08:32:33.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f13400640 1 -- 172.21.15.146:0/725279967 learned_addr learned my addr 172.21.15.146:0/725279967 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:33.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.838+0000 7f8f13e00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f1410d3c0 0x7f8f141a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f13400640 1 -- 172.21.15.146:0/725279967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f14108e40 msgr2=0x7f8f1419f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f13400640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f14108e40 0x7f8f1419f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f13400640 1 -- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f1410d3c0 msgr2=0x7f8f141a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:33.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f13400640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f1410d3c0 0x7f8f141a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:33.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f13400640 1 -- 172.21.15.146:0/725279967 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f141a3d40 con 0x7f8f141084a0 2024-04-24T08:32:33.847 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f13400640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141084a0 0x7f8f1419ed50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400e9f0 tx=0x7f8f0400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f10c00640 1 -- 172.21.15.146:0/725279967 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0400cd80 con 0x7f8f141084a0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f10c00640 1 -- 172.21.15.146:0/725279967 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f04005500 con 0x7f8f141084a0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f141a4030 con 0x7f8f141084a0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f10c00640 1 -- 172.21.15.146:0/725279967 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04010690 con 0x7f8f141084a0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.839+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f141ab8c0 con 0x7f8f141084a0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.840+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f14069190 con 0x7f8f141084a0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.842+0000 7f8f10c00640 1 -- 172.21.15.146:0/725279967 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f8f040046c0 con 0x7f8f141084a0 2024-04-24T08:32:33.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.844+0000 7f8f10c00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ef80a3d20 0x7f8ef80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:33.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.844+0000 7f8f10c00640 1 -- 172.21.15.146:0/725279967 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7f8f040d0bf0 con 0x7f8f141084a0 2024-04-24T08:32:33.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.844+0000 7f8f0be00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ef80a3d20 0x7f8ef80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:33.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.845+0000 7f8f0be00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ef80a3d20 0x7f8ef80a61e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8efc000c00 tx=0x7f8efc005b50 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:33.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:33.848+0000 7f8f10c00640 1 -- 172.21.15.146:0/725279967 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f0409d220 con 0x7f8f141084a0 2024-04-24T08:32:33.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:33 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1561561051' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T08:32:33.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:33 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1561561051' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T08:32:34.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.092+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f14110900 con 0x7f8f141084a0 2024-04-24T08:32:34.094 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.094+0000 7f8f10c00640 1 -- 172.21.15.146:0/725279967 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11891 (secure 0 0 0) 0x7f8f0409cb20 con 0x7f8f141084a0 2024-04-24T08:32:34.095 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:32:34.095 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/3628851028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3628851028},{"type":"v1","addr":"172.21.15.177:6833","nonce":3628851028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/3377566039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":3377566039},{"type":"v1","addr":"172.21.15.153:6833","nonce":3377566039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24464,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2947430860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2947430860},{"type":"v1","addr":"172.21.15.146:6835","nonce":2947430860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1079190046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1079190046},{"type":"v1","addr":"172.21.15.146:6837","nonce":1079190046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/3655889612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":3655889612},{"type":"v1","addr":"172.21.15.153:6835","nonce":3655889612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/2429639426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":2429639426},{"type":"v1","addr":"172.21.15.177:6835","nonce":2429639426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:18:46.894792+0000","modified":"2024-04-24T08:26:42.216076+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24568,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2495345664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2495345664},{"type":"v1","addr":"172.21.15.177:6837","nonce":2495345664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6837/565470680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":565470680},{"type":"v1","addr":"172.21.15.153:6837","nonce":565470680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/1533073551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1533073551},{"type":"v1","addr":"172.21.15.177:6839","nonce":1533073551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.146:6839/2207966302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6838","nonce":2207966302},{"type":"v1","addr":"172.21.15.146:6839","nonce":2207966302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24568":{"gid":24568,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.146:6841/1590374458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6840","nonce":1590374458},{"type":"v1","addr":"172.21.15.146:6841","nonce":1590374458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.153:6839/2023799727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2023799727},{"type":"v1","addr":"172.21.15.153:6839","nonce":2023799727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24553,24568,24569]},"id":1}]} 2024-04-24T08:32:34.095 INFO:teuthology.orchestra.run.smithi146.stderr:dumped fsmap epoch 26 2024-04-24T08:32:34.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ef80a3d20 msgr2=0x7f8ef80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ef80a3d20 0x7f8ef80a61e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8efc000c00 tx=0x7f8efc005b50 comp rx=0 tx=0).stop 2024-04-24T08:32:34.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141084a0 msgr2=0x7f8f1419ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141084a0 0x7f8f1419ed50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400e9f0 tx=0x7f8f0400edf0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 shutdown_connections 2024-04-24T08:32:34.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f8ef80a3d20 0x7f8ef80a61e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f1410d3c0 0x7f8f141a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f14108e40 0x7f8f1419f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 --2- 172.21.15.146:0/725279967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f141084a0 0x7f8f1419ed50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 >> 172.21.15.146:0/725279967 conn(0x7f8f14077b80 msgr2=0x7f8f14078180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:34.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 shutdown_connections 2024-04-24T08:32:34.099 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.097+0000 7f8f18e00640 1 -- 172.21.15.146:0/725279967 wait complete. 2024-04-24T08:32:34.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:32:34.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 -- 172.21.15.146:0/3860995129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b741084a0 msgr2=0x7f6b74108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/3860995129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b741084a0 0x7f6b74108900 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009900 tx=0x7f6b6002a480 comp rx=0 tx=0).stop 2024-04-24T08:32:34.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 -- 172.21.15.146:0/3860995129 shutdown_connections 2024-04-24T08:32:34.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/3860995129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b741084a0 0x7f6b74108900 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/3860995129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b74108e40 0x7f6b741152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/3860995129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b7410d3c0 0x7f6b7410d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 -- 172.21.15.146:0/3860995129 >> 172.21.15.146:0/3860995129 conn(0x7f6b74077b80 msgr2=0x7f6b74079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:34.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 -- 172.21.15.146:0/3860995129 shutdown_connections 2024-04-24T08:32:34.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.321+0000 7f6b7aa00640 1 -- 172.21.15.146:0/3860995129 wait complete. 2024-04-24T08:32:34.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.322+0000 7f6b7aa00640 1 Processor -- start 2024-04-24T08:32:34.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.322+0000 7f6b7aa00640 1 -- start start 2024-04-24T08:32:34.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b741084a0 0x7f6b741a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b74108e40 0x7f6b741a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b7410d3c0 0x7f6b7419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7aa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741187e0 con 0x7f6b741084a0 2024-04-24T08:32:34.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7aa00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6b74118660 con 0x7f6b74108e40 2024-04-24T08:32:34.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6b74118960 con 0x7f6b7410d3c0 2024-04-24T08:32:34.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b7410d3c0 0x7f6b7419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b79600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b741084a0 0x7f6b741a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b7410d3c0 0x7f6b7419eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:35964/0 (socket says 172.21.15.146:35964) 2024-04-24T08:32:34.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b7a000640 1 -- 172.21.15.146:0/375128701 learned_addr learned my addr 172.21.15.146:0/375128701 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:34.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.323+0000 7f6b78c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b74108e40 0x7f6b741a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7a000640 1 -- 172.21.15.146:0/375128701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b74108e40 msgr2=0x7f6b741a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7a000640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b74108e40 0x7f6b741a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7a000640 1 -- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b741084a0 msgr2=0x7f6b741a48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7a000640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b741084a0 0x7f6b741a48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7a000640 1 -- 172.21.15.146:0/375128701 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7419f220 con 0x7f6b7410d3c0 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7a000640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b7410d3c0 0x7f6b7419eaa0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00eb10 tx=0x7f6b6c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b72000640 1 -- 172.21.15.146:0/375128701 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c004b20 con 0x7f6b7410d3c0 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b72000640 1 -- 172.21.15.146:0/375128701 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b6c004cc0 con 0x7f6b7410d3c0 2024-04-24T08:32:34.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7419f510 con 0x7f6b7410d3c0 2024-04-24T08:32:34.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b72000640 1 -- 172.21.15.146:0/375128701 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c010760 con 0x7f6b7410d3c0 2024-04-24T08:32:34.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.324+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b741ab7b0 con 0x7f6b7410d3c0 2024-04-24T08:32:34.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.325+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b74069190 con 0x7f6b7410d3c0 2024-04-24T08:32:34.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.328+0000 7f6b72000640 1 -- 172.21.15.146:0/375128701 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f6b6c005020 con 0x7f6b7410d3c0 2024-04-24T08:32:34.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.329+0000 7f6b72000640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6b5c0a3d20 0x7f6b5c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.329+0000 7f6b72000640 1 -- 172.21.15.146:0/375128701 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7f6b6c0d0df0 con 0x7f6b7410d3c0 2024-04-24T08:32:34.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.329+0000 7f6b79600640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6b5c0a3d20 0x7f6b5c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.330+0000 7f6b79600640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6b5c0a3d20 0x7f6b5c0a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6b680095f0 tx=0x7f6b68005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:34.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.333+0000 7f6b72000640 1 -- 172.21.15.146:0/375128701 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b6c09d420 con 0x7f6b7410d3c0 2024-04-24T08:32:34.540 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.539+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b7402d620 con 0x7f6b7410d3c0 2024-04-24T08:32:34.542 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.542+0000 7f6b72000640 1 -- 172.21.15.146:0/375128701 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+23161 (secure 0 0 0) 0x7f6b6c09cd20 con 0x7f6b7410d3c0 2024-04-24T08:32:34.543 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-04-24T08:32:34.543 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":101,"fsid":"cca14746-0211-11ef-bc93-c7b262605968","created":"2024-04-24T08:09:48.695117+0000","modified":"2024-04-24T08:32:15.934761+0000","last_up_change":"2024-04-24T08:17:52.533829+0000","last_in_change":"2024-04-24T08:17:32.602083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:13:40.471709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:44.805814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:45.883831+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1eee291d-ca8f-49ff-8a55-6e95cfb18c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6803","nonce":3767707346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6805","nonce":3767707346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6809","nonce":3767707346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":3767707346},{"type":"v1","addr":"172.21.15.146:6807","nonce":3767707346}]},"public_addr":"172.21.15.146:6803/3767707346","cluster_addr":"172.21.15.146:6805/3767707346","heartbeat_back_addr":"172.21.15.146:6809/3767707346","heartbeat_front_addr":"172.21.15.146:6807/3767707346","state":["exists","up"]},{"osd":1,"uuid":"8e5912a1-9d40-4f2b-9651-551bc2797a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6801","nonce":1792775739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6803","nonce":1792775739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6807","nonce":1792775739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1792775739},{"type":"v1","addr":"172.21.15.153:6805","nonce":1792775739}]},"public_addr":"172.21.15.153:6801/1792775739","cluster_addr":"172.21.15.153:6803/1792775739","heartbeat_back_addr":"172.21.15.153:6807/1792775739","heartbeat_front_addr":"172.21.15.153:6805/1792775739","state":["exists","up"]},{"osd":2,"uuid":"a25edc87-3db2-4652-82ee-c406cbe5dd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6801","nonce":96699540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6803","nonce":96699540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6807","nonce":96699540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":96699540},{"type":"v1","addr":"172.21.15.177:6805","nonce":96699540}]},"public_addr":"172.21.15.177:6801/96699540","cluster_addr":"172.21.15.177:6803/96699540","heartbeat_back_addr":"172.21.15.177:6807/96699540","heartbeat_front_addr":"172.21.15.177:6805/96699540","state":["exists","up"]},{"osd":3,"uuid":"c0de0716-915a-4554-bb92-b04edb82af4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6811","nonce":602409571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6813","nonce":602409571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6817","nonce":602409571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":602409571},{"type":"v1","addr":"172.21.15.146:6815","nonce":602409571}]},"public_addr":"172.21.15.146:6811/602409571","cluster_addr":"172.21.15.146:6813/602409571","heartbeat_back_addr":"172.21.15.146:6817/602409571","heartbeat_front_addr":"172.21.15.146:6815/602409571","state":["exists","up"]},{"osd":4,"uuid":"3f96f1d5-f2a7-4b27-8f7e-1d440866acd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6809","nonce":3654502994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6811","nonce":3654502994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6815","nonce":3654502994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3654502994},{"type":"v1","addr":"172.21.15.153:6813","nonce":3654502994}]},"public_addr":"172.21.15.153:6809/3654502994","cluster_addr":"172.21.15.153:6811/3654502994","heartbeat_back_addr":"172.21.15.153:6815/3654502994","heartbeat_front_addr":"172.21.15.153:6813/3654502994","state":["exists","up"]},{"osd":5,"uuid":"ad461bd7-3a6c-47cc-aa4c-39793fc77bc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6809","nonce":3773516414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6811","nonce":3773516414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6815","nonce":3773516414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3773516414},{"type":"v1","addr":"172.21.15.177:6813","nonce":3773516414}]},"public_addr":"172.21.15.177:6809/3773516414","cluster_addr":"172.21.15.177:6811/3773516414","heartbeat_back_addr":"172.21.15.177:6815/3773516414","heartbeat_front_addr":"172.21.15.177:6813/3773516414","state":["exists","up"]},{"osd":6,"uuid":"c3257f43-689c-4fa9-b182-d9783ef1aa31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6819","nonce":3036960082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6821","nonce":3036960082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6825","nonce":3036960082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3036960082},{"type":"v1","addr":"172.21.15.146:6823","nonce":3036960082}]},"public_addr":"172.21.15.146:6819/3036960082","cluster_addr":"172.21.15.146:6821/3036960082","heartbeat_back_addr":"172.21.15.146:6825/3036960082","heartbeat_front_addr":"172.21.15.146:6823/3036960082","state":["exists","up"]},{"osd":7,"uuid":"7b062222-8a11-47ad-9389-ad3a5b1ed19d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6817","nonce":2204927697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6819","nonce":2204927697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6823","nonce":2204927697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2204927697},{"type":"v1","addr":"172.21.15.153:6821","nonce":2204927697}]},"public_addr":"172.21.15.153:6817/2204927697","cluster_addr":"172.21.15.153:6819/2204927697","heartbeat_back_addr":"172.21.15.153:6823/2204927697","heartbeat_front_addr":"172.21.15.153:6821/2204927697","state":["exists","up"]},{"osd":8,"uuid":"30f8bd53-eba7-4942-8c3d-1f327abe42fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6817","nonce":4270133379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6819","nonce":4270133379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6823","nonce":4270133379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":4270133379},{"type":"v1","addr":"172.21.15.177:6821","nonce":4270133379}]},"public_addr":"172.21.15.177:6817/4270133379","cluster_addr":"172.21.15.177:6819/4270133379","heartbeat_back_addr":"172.21.15.177:6823/4270133379","heartbeat_front_addr":"172.21.15.177:6821/4270133379","state":["exists","up"]},{"osd":9,"uuid":"18bd8323-8ae1-4ea1-bf14-17d4f4cdce6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6827","nonce":2032154724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6829","nonce":2032154724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6833","nonce":2032154724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2032154724},{"type":"v1","addr":"172.21.15.146:6831","nonce":2032154724}]},"public_addr":"172.21.15.146:6827/2032154724","cluster_addr":"172.21.15.146:6829/2032154724","heartbeat_back_addr":"172.21.15.146:6833/2032154724","heartbeat_front_addr":"172.21.15.146:6831/2032154724","state":["exists","up"]},{"osd":10,"uuid":"e7cb8a28-80cc-473d-91d1-dda23d2790af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6825","nonce":679103416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6827","nonce":679103416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6831","nonce":679103416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":679103416},{"type":"v1","addr":"172.21.15.153:6829","nonce":679103416}]},"public_addr":"172.21.15.153:6825/679103416","cluster_addr":"172.21.15.153:6827/679103416","heartbeat_back_addr":"172.21.15.153:6831/679103416","heartbeat_front_addr":"172.21.15.153:6829/679103416","state":["exists","up"]},{"osd":11,"uuid":"3a48d77c-4350-46bf-8fdf-49827d3957c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6825","nonce":2574161099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6827","nonce":2574161099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6831","nonce":2574161099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":2574161099},{"type":"v1","addr":"172.21.15.177:6829","nonce":2574161099}]},"public_addr":"172.21.15.177:6825/2574161099","cluster_addr":"172.21.15.177:6827/2574161099","heartbeat_back_addr":"172.21.15.177:6831/2574161099","heartbeat_front_addr":"172.21.15.177:6829/2574161099","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:12:47.743666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:13.336252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:13:37.470869+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:09.885486+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:14:35.983286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:00.062202+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:32.852598+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:15:59.225602+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:24.425109+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:16:58.110696+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:25.111175+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:17:50.509588+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:0/4151754658":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2355630587":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6800/1475219332":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/393585939":"2024-04-25T08:21:11.077472+0000","172.21.15.146:6801/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3118900567":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/1190010075":"2024-04-25T08:10:56.306756+0000","172.21.15.146:6800/888738583":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/3978832813":"2024-04-25T08:10:56.306756+0000","172.21.15.146:0/2983645773":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/4023745007":"2024-04-25T08:21:11.077472+0000","172.21.15.146:0/2776821726":"2024-04-25T08:10:22.202345+0000","172.21.15.146:6801/520435616":"2024-04-25T08:10:22.202345+0000","172.21.15.146:0/677277692":"2024-04-25T08:10:22.202345+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:32:34.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.544+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6b5c0a3d20 msgr2=0x7f6b5c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.544+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6b5c0a3d20 0x7f6b5c0a61e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6b680095f0 tx=0x7f6b68005f90 comp rx=0 tx=0).stop 2024-04-24T08:32:34.545 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.544+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b7410d3c0 msgr2=0x7f6b7419eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.544+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b7410d3c0 0x7f6b7419eaa0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00eb10 tx=0x7f6b6c00c2d0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 shutdown_connections 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f6b5c0a3d20 0x7f6b5c0a61e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6b7410d3c0 0x7f6b7419eaa0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b74108e40 0x7f6b741a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 --2- 172.21.15.146:0/375128701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b741084a0 0x7f6b741a48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 >> 172.21.15.146:0/375128701 conn(0x7f6b74077b80 msgr2=0x7f6b740781c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:34.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 shutdown_connections 2024-04-24T08:32:34.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.545+0000 7f6b7aa00640 1 -- 172.21.15.146:0/375128701 wait complete. 2024-04-24T08:32:34.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:32:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.766+0000 7ffa47800640 1 -- 172.21.15.146:0/796171282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 msgr2=0x7ffa401153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.766+0000 7ffa47800640 1 --2- 172.21.15.146:0/796171282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 0x7ffa401153c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009ca0 tx=0x7ffa3802a950 comp rx=0 tx=0).stop 2024-04-24T08:32:34.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.766+0000 7ffa47800640 1 -- 172.21.15.146:0/796171282 shutdown_connections 2024-04-24T08:32:34.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.766+0000 7ffa47800640 1 --2- 172.21.15.146:0/796171282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa40074cc0 0x7ffa40112a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.766+0000 7ffa47800640 1 --2- 172.21.15.146:0/796171282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 0x7ffa401153c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.766+0000 7ffa47800640 1 --2- 172.21.15.146:0/796171282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa400743a0 0x7ffa40074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.768 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.766+0000 7ffa47800640 1 -- 172.21.15.146:0/796171282 >> 172.21.15.146:0/796171282 conn(0x7ffa400fda70 msgr2=0x7ffa400ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:34.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.767+0000 7ffa47800640 1 -- 172.21.15.146:0/796171282 shutdown_connections 2024-04-24T08:32:34.769 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.767+0000 7ffa47800640 1 -- 172.21.15.146:0/796171282 wait complete. 2024-04-24T08:32:34.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.768+0000 7ffa47800640 1 Processor -- start 2024-04-24T08:32:34.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.768+0000 7ffa47800640 1 -- start start 2024-04-24T08:32:34.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa47800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa400743a0 0x7ffa4019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa47800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40074cc0 0x7ffa4019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.771 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa47800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 0x7ffa401a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa47800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7ffa401189d0 con 0x7ffa40112fd0 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa47800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa40118850 con 0x7ffa40074cc0 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa47800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa40118b50 con 0x7ffa400743a0 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40074cc0 0x7ffa4019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40074cc0 0x7ffa4019f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48934/0 (socket says 172.21.15.146:48934) 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 -- 172.21.15.146:0/553180533 learned_addr learned my addr 172.21.15.146:0/553180533 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa45800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 0x7ffa401a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.772 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa44e00640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa400743a0 0x7ffa4019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 -- 172.21.15.146:0/553180533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa400743a0 msgr2=0x7ffa4019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa400743a0 0x7ffa4019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 -- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 msgr2=0x7ffa401a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 0x7ffa401a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa3fe00640 1 -- 172.21.15.146:0/553180533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa401a3ee0 con 0x7ffa40074cc0 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.769+0000 7ffa45800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 0x7ffa401a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.770+0000 7ffa3fe00640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40074cc0 0x7ffa4019f430 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00ea60 tx=0x7ffa2c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:34.773 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.770+0000 7ffa3d600640 1 -- 172.21.15.146:0/553180533 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c00cde0 con 0x7ffa40074cc0 2024-04-24T08:32:34.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.770+0000 7ffa3d600640 1 -- 172.21.15.146:0/553180533 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa2c0035f0 con 0x7ffa40074cc0 2024-04-24T08:32:34.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.770+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa401a41d0 con 0x7ffa40074cc0 2024-04-24T08:32:34.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.770+0000 7ffa3d600640 1 -- 172.21.15.146:0/553180533 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c010670 con 0x7ffa40074cc0 2024-04-24T08:32:34.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.770+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa401aba60 con 0x7ffa40074cc0 2024-04-24T08:32:34.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.771+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa40068e20 con 0x7ffa40074cc0 2024-04-24T08:32:34.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.773+0000 7ffa3d600640 1 -- 172.21.15.146:0/553180533 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7ffa2c0037a0 con 0x7ffa40074cc0 2024-04-24T08:32:34.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.774+0000 7ffa3d600640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa280a3d20 0x7ffa280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:34.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.775+0000 7ffa3d600640 1 -- 172.21.15.146:0/553180533 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8542+0+0 (secure 0 0 0) 0x7ffa2c0d0b40 con 0x7ffa40074cc0 2024-04-24T08:32:34.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.775+0000 7ffa44e00640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa280a3d20 0x7ffa280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:34.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.775+0000 7ffa44e00640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa280a3d20 0x7ffa280a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffa340095f0 tx=0x7ffa34005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:34.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:34.779+0000 7ffa3d600640 1 -- 172.21.15.146:0/553180533 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa2c09d170 con 0x7ffa40074cc0 2024-04-24T08:32:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:34 smithi177 ceph-mon[33046]: pgmap v354: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:32:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:34 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1544587535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:32:34.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:34 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/725279967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:32:34.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:34 smithi153 ceph-mon[32746]: pgmap v354: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:32:34.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:34 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1544587535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:32:34.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:34 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/725279967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:32:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:34 smithi146 ceph-mon[27805]: pgmap v354: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:32:34.992 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:34 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1544587535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:32:34.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:34 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/725279967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:32:35.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.022+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ffa40105480 con 0x7ffa40074cc0 2024-04-24T08:32:35.033 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.032+0000 7ffa3d600640 1 -- 172.21.15.146:0/553180533 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7ffa2c09ca70 con 0x7ffa40074cc0 2024-04-24T08:32:35.033 INFO:teuthology.orchestra.run.smithi146.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:32:35.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.036+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa280a3d20 msgr2=0x7ffa280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:35.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.036+0000 7ffa47800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa280a3d20 0x7ffa280a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffa340095f0 tx=0x7ffa34005f90 comp rx=0 tx=0).stop 2024-04-24T08:32:35.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.036+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40074cc0 msgr2=0x7ffa4019f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:35.037 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.036+0000 7ffa47800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40074cc0 0x7ffa4019f430 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00ea60 tx=0x7ffa2c00ef20 comp rx=0 tx=0).stop 2024-04-24T08:32:35.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 shutdown_connections 2024-04-24T08:32:35.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7ffa280a3d20 0x7ffa280a61e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffa40112fd0 0x7ffa401a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.038 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa40074cc0 0x7ffa4019f430 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 --2- 172.21.15.146:0/553180533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa400743a0 0x7ffa4019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 >> 172.21.15.146:0/553180533 conn(0x7ffa400fda70 msgr2=0x7ffa400ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:35.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 shutdown_connections 2024-04-24T08:32:35.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.037+0000 7ffa47800640 1 -- 172.21.15.146:0/553180533 wait complete. 2024-04-24T08:32:35.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:32:35.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 -- 172.21.15.146:0/1512122617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102bf0 msgr2=0x7f5294106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:35.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 --2- 172.21.15.146:0/1512122617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102bf0 0x7f5294106c30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5280009900 tx=0x7f528002a400 comp rx=0 tx=0).stop 2024-04-24T08:32:35.242 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 -- 172.21.15.146:0/1512122617 shutdown_connections 2024-04-24T08:32:35.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 --2- 172.21.15.146:0/1512122617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5294107310 0x7f529410eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 --2- 172.21.15.146:0/1512122617 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102bf0 0x7f5294106c30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 --2- 172.21.15.146:0/1512122617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5294102240 0x7f5294102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 -- 172.21.15.146:0/1512122617 >> 172.21.15.146:0/1512122617 conn(0x7f5294079450 msgr2=0x7f5294079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:35.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 -- 172.21.15.146:0/1512122617 shutdown_connections 2024-04-24T08:32:35.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.241+0000 7f529ba00640 1 -- 172.21.15.146:0/1512122617 wait complete. 2024-04-24T08:32:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.242+0000 7f529ba00640 1 Processor -- start 2024-04-24T08:32:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.242+0000 7f529ba00640 1 -- start start 2024-04-24T08:32:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f529ba00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102240 0x7f52941a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f529ba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5294107310 0x7f52941a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f529ba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529419a9d0 0x7f529419ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f529ba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f52941123a0 con 0x7f529419a9d0 2024-04-24T08:32:35.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f529ba00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5294112220 con 0x7f5294102240 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f529ba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5294112520 con 0x7f5294107310 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102240 0x7f52941a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529419a9d0 0x7f529419ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102240 0x7f52941a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:48998/0 (socket says 172.21.15.146:48998) 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 -- 172.21.15.146:0/1198680210 learned_addr learned my addr 172.21.15.146:0/1198680210 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5293e00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5294107310 0x7f52941a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 -- 172.21.15.146:0/1198680210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5294107310 msgr2=0x7f52941a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:35.247 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5294107310 0x7f52941a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 -- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529419a9d0 msgr2=0x7f529419ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529419a9d0 0x7f529419ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299000640 1 -- 172.21.15.146:0/1198680210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529419b4f0 con 0x7f5294102240 2024-04-24T08:32:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5293e00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5294107310 0x7f52941a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:32:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.243+0000 7f5299a00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529419a9d0 0x7f529419ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:32:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.244+0000 7f5299000640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102240 0x7f52941a0780 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f528800e9f0 tx=0x7f528800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:35.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.244+0000 7f5291600640 1 -- 172.21.15.146:0/1198680210 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528800cd80 con 0x7f5294102240 2024-04-24T08:32:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.244+0000 7f5291600640 1 -- 172.21.15.146:0/1198680210 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5288005500 con 0x7f5294102240 2024-04-24T08:32:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.244+0000 7f5291600640 1 -- 172.21.15.146:0/1198680210 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288010690 con 0x7f5294102240 2024-04-24T08:32:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.244+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529419b7e0 con 0x7f5294102240 2024-04-24T08:32:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.244+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52941a7940 con 0x7f5294102240 2024-04-24T08:32:35.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.245+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5294069270 con 0x7f5294102240 2024-04-24T08:32:35.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.247+0000 7f5291600640 1 -- 172.21.15.146:0/1198680210 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f52880046c0 con 0x7f5294102240 2024-04-24T08:32:35.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.248+0000 7f5291600640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f527c0a3da0 0x7f527c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:35.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.248+0000 7f5291600640 1 -- 172.21.15.146:0/1198680210 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8756+0+0 (secure 0 0 0) 0x7f52880d0d90 con 0x7f5294102240 2024-04-24T08:32:35.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.248+0000 7f5293e00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f527c0a3da0 0x7f527c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:35.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.249+0000 7f5293e00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f527c0a3da0 0x7f527c0a6260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5280000c00 tx=0x7f5280005a20 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:35.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.252+0000 7f5291600640 1 -- 172.21.15.146:0/1198680210 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f528809d2e0 con 0x7f5294102240 2024-04-24T08:32:35.461 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: ignoring --setuser ceph since I am not root 2024-04-24T08:32:35.461 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: ignoring --setgroup ceph since I am not root 2024-04-24T08:32:35.462 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mds[62796]: -- 172.21.15.153:0/2596605741 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8582d7a40 con 0x55f8582ab000 2024-04-24T08:32:35.462 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: starting mds.b at 2024-04-24T08:32:35.462 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h[62476]: ignoring --setuser ceph since I am not root 2024-04-24T08:32:35.462 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h[62476]: ignoring --setgroup ceph since I am not root 2024-04-24T08:32:35.462 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mds[62485]: -- 172.21.15.153:0/3193285343 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55acab5ffa40 con 0x55acab5d2800 2024-04-24T08:32:35.462 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h[62476]: starting mds.h at 2024-04-24T08:32:35.492 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a[27801]: 2024-04-24T08:32:35.024+0000 7f4fba400700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:32:35.493 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a[69454]: ignoring --setuser ceph since I am not root 2024-04-24T08:32:35.493 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a[69454]: ignoring --setgroup ceph since I am not root 2024-04-24T08:32:35.493 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mds[69459]: -- 172.21.15.146:0/793100155 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631901a9a40 con 0x56319017d000 2024-04-24T08:32:35.493 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a[69454]: starting mds.a at 2024-04-24T08:32:35.494 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g[69067]: ignoring --setuser ceph since I am not root 2024-04-24T08:32:35.494 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g[69067]: ignoring --setgroup ceph since I am not root 2024-04-24T08:32:35.494 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mds[69089]: -- 172.21.15.146:0/2536130213 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645ccb07a40 con 0x5645ccada800 2024-04-24T08:32:35.494 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g[69067]: starting mds.g at 2024-04-24T08:32:35.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:35.500+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5294102d80 con 0x7f5294102240 2024-04-24T08:32:35.518 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i[64442]: ignoring --setuser ceph since I am not root 2024-04-24T08:32:35.518 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i[64442]: ignoring --setgroup ceph since I am not root 2024-04-24T08:32:35.518 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mds[64446]: -- 172.21.15.177:0/3042567664 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55944cac3a40 con 0x55944ca96800 2024-04-24T08:32:35.518 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i[64442]: starting mds.i at 2024-04-24T08:32:35.519 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c[64134]: ignoring --setuser ceph since I am not root 2024-04-24T08:32:35.519 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c[64134]: ignoring --setgroup ceph since I am not root 2024-04-24T08:32:35.519 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mds[64138]: -- 172.21.15.177:0/3398209028 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5606678fda40 con 0x5606678d1000 2024-04-24T08:32:35.519 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c[64134]: starting mds.c at 2024-04-24T08:32:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/375128701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:32:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/553180533' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:32:35.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:32:35.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T08:32:35.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:32:35.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T08:32:35.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: osdmap e102: 12 total, 12 up, 12 in 2024-04-24T08:32:35.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-24T08:32:35.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1198680210' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:32:35.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:35 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:32:35.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/375128701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:32:35.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/553180533' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:32:35.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:32:35.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T08:32:35.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:32:35.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T08:32:35.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: osdmap e102: 12 total, 12 up, 12 in 2024-04-24T08:32:35.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-24T08:32:35.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1198680210' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:32:35.962 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:35 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:32:35.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/375128701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:32:35.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/553180533' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:32:35.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:32:35.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T08:32:35.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:32:35.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T08:32:35.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: osdmap e102: 12 total, 12 up, 12 in 2024-04-24T08:32:35.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-24T08:32:35.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1198680210' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:32:35.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:35 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:32:36.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.041+0000 7f5291600640 1 -- 172.21.15.146:0/1198680210 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7f528809cbe0 con 0x7f5294102240 2024-04-24T08:32:36.045 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f527c0a3da0 msgr2=0x7f527c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:36.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f527c0a3da0 0x7f527c0a6260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5280000c00 tx=0x7f5280005a20 comp rx=0 tx=0).stop 2024-04-24T08:32:36.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102240 msgr2=0x7f52941a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:36.046 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102240 0x7f52941a0780 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f528800e9f0 tx=0x7f528800edf0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 shutdown_connections 2024-04-24T08:32:36.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f527c0a3da0 0x7f527c0a6260 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5294107310 0x7f52941a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529419a9d0 0x7f529419ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 --2- 172.21.15.146:0/1198680210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5294102240 0x7f52941a0780 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.047 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.045+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 >> 172.21.15.146:0/1198680210 conn(0x7f5294079450 msgr2=0x7f52940ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:36.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.047+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 shutdown_connections 2024-04-24T08:32:36.048 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.047+0000 7f529ba00640 1 -- 172.21.15.146:0/1198680210 wait complete. 2024-04-24T08:32:36.070 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:32:36.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 -- 172.21.15.146:0/4263737471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18100770 msgr2=0x7f2b18100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:36.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 --2- 172.21.15.146:0/4263737471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18100770 0x7f2b18100b50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009900 tx=0x7f2b0802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 -- 172.21.15.146:0/4263737471 shutdown_connections 2024-04-24T08:32:36.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 --2- 172.21.15.146:0/4263737471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b18101120 0x7f2b18106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 --2- 172.21.15.146:0/4263737471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18107380 0x7f2b1810ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 --2- 172.21.15.146:0/4263737471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18100770 0x7f2b18100b50 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 -- 172.21.15.146:0/4263737471 >> 172.21.15.146:0/4263737471 conn(0x7f2b18077430 msgr2=0x7f2b180fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:36.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 -- 172.21.15.146:0/4263737471 shutdown_connections 2024-04-24T08:32:36.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.255+0000 7f2b1e400640 1 -- 172.21.15.146:0/4263737471 wait complete. 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.256+0000 7f2b1e400640 1 Processor -- start 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.256+0000 7f2b1e400640 1 -- start start 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b1e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18100770 0x7f2b181016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b1e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b18107380 0x7f2b18101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b1e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18102160 0x7f2b181a6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b1e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f2b18112410 con 0x7f2b18102160 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b1e400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2b18112290 con 0x7f2b18100770 2024-04-24T08:32:36.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b1e400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2b18112590 con 0x7f2b18107380 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b17400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18100770 0x7f2b181016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b17400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18100770 0x7f2b181016e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:49016/0 (socket says 172.21.15.146:49016) 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b17400640 1 -- 172.21.15.146:0/945549905 learned_addr learned my addr 172.21.15.146:0/945549905 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b17e00640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18102160 0x7f2b181a6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.257+0000 7f2b0fe00640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b18107380 0x7f2b18101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b17400640 1 -- 172.21.15.146:0/945549905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b18107380 msgr2=0x7f2b18101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b17400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b18107380 0x7f2b18101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b17400640 1 -- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18102160 msgr2=0x7f2b181a6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b17400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18102160 0x7f2b181a6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b17400640 1 -- 172.21.15.146:0/945549905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b181a71c0 con 0x7f2b18100770 2024-04-24T08:32:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b17400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18100770 0x7f2b181016e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2b08004770 tx=0x7f2b08003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b14c00640 1 -- 172.21.15.146:0/945549905 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0803f070 con 0x7f2b18100770 2024-04-24T08:32:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b14c00640 1 -- 172.21.15.146:0/945549905 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b08031ca0 con 0x7f2b18100770 2024-04-24T08:32:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b181a7450 con 0x7f2b18100770 2024-04-24T08:32:36.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b14c00640 1 -- 172.21.15.146:0/945549905 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08035030 con 0x7f2b18100770 2024-04-24T08:32:36.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.258+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b181a7990 con 0x7f2b18100770 2024-04-24T08:32:36.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.259+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b18103800 con 0x7f2b18100770 2024-04-24T08:32:36.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.262+0000 7f2b14c00640 1 -- 172.21.15.146:0/945549905 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f2b08042070 con 0x7f2b18100770 2024-04-24T08:32:36.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.263+0000 7f2b14c00640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2af80a3dd0 0x7f2af80a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:36.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.263+0000 7f2b14c00640 1 -- 172.21.15.146:0/945549905 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8756+0+0 (secure 0 0 0) 0x7f2b080f11b0 con 0x7f2b18100770 2024-04-24T08:32:36.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.263+0000 7f2b0fe00640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2af80a3dd0 0x7f2af80a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:36.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.264+0000 7f2b0fe00640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2af80a3dd0 0x7f2af80a6290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2b18103170 tx=0x7f2b0000b390 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:36.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.265+0000 7f2b14c00640 1 -- 172.21.15.146:0/945549905 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b080bd1c0 con 0x7f2b18100770 2024-04-24T08:32:36.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:36.471+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2b1810a530 con 0x7f2b18100770 2024-04-24T08:32:36.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: pgmap v356: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-24T08:32:36.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T08:32:36.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T08:32:36.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: Cluster is now healthy 2024-04-24T08:32:36.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:32:36.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:32:36.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:32:36.816 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.153:6836/362827972,v1:172.21.15.153:6837/362827972] up:boot 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.177:6836/4147045037,v1:172.21.15.177:6838/4147045037] up:boot 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.177:6837/3271213888,v1:172.21.15.177:6839/3271213888] up:boot 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.153:6838/2612121917,v1:172.21.15.153:6839/2612121917] up:boot 2024-04-24T08:32:36.817 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6840/1279154158,v1:172.21.15.146:6841/1279154158] up:boot 2024-04-24T08:32:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: mds.? [v2:172.21.15.146:6838/930259841,v1:172.21.15.146:6839/930259841] up:boot 2024-04-24T08:32:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: fsmap 12 up:standby 2024-04-24T08:32:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/945549905' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:36.818 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:36 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:36.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: pgmap v356: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-24T08:32:36.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T08:32:36.863 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: Cluster is now healthy 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:32:36.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.153:6836/362827972,v1:172.21.15.153:6837/362827972] up:boot 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.177:6836/4147045037,v1:172.21.15.177:6838/4147045037] up:boot 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.177:6837/3271213888,v1:172.21.15.177:6839/3271213888] up:boot 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.153:6838/2612121917,v1:172.21.15.153:6839/2612121917] up:boot 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6840/1279154158,v1:172.21.15.146:6841/1279154158] up:boot 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: mds.? [v2:172.21.15.146:6838/930259841,v1:172.21.15.146:6839/930259841] up:boot 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: fsmap 12 up:standby 2024-04-24T08:32:36.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/945549905' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:36.866 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:36 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:36.993 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: pgmap v356: 97 pgs: 97 active+clean; 74 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-24T08:32:36.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T08:32:36.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T08:32:36.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: Cluster is now healthy 2024-04-24T08:32:36.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:32:36.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:32:36.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:32:36.994 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='mgr.15087 172.21.15.146:0/3363007498' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.153:6836/362827972,v1:172.21.15.153:6837/362827972] up:boot 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.177:6836/4147045037,v1:172.21.15.177:6838/4147045037] up:boot 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.177:6837/3271213888,v1:172.21.15.177:6839/3271213888] up:boot 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.153:6838/2612121917,v1:172.21.15.153:6839/2612121917] up:boot 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6840/1279154158,v1:172.21.15.146:6841/1279154158] up:boot 2024-04-24T08:32:36.995 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: mds.? [v2:172.21.15.146:6838/930259841,v1:172.21.15.146:6839/930259841] up:boot 2024-04-24T08:32:36.996 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: fsmap 12 up:standby 2024-04-24T08:32:36.996 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/945549905' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:36.996 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:36 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.047+0000 7f2b14c00640 1 -- 172.21.15.146:0/945549905 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) ==== 181+0+0 (secure 0 0 0) 0x7f2b080c14c0 con 0x7f2b18100770 2024-04-24T08:32:37.049 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:32:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2af80a3dd0 msgr2=0x7f2af80a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2af80a3dd0 0x7f2af80a6290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2b18103170 tx=0x7f2b0000b390 comp rx=0 tx=0).stop 2024-04-24T08:32:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18100770 msgr2=0x7f2b181016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18100770 0x7f2b181016e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2b08004770 tx=0x7f2b08003c10 comp rx=0 tx=0).stop 2024-04-24T08:32:37.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 shutdown_connections 2024-04-24T08:32:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f2af80a3dd0 0x7f2af80a6290 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b18107380 0x7f2b18101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18102160 0x7f2b181a6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 --2- 172.21.15.146:0/945549905 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b18100770 0x7f2b181016e0 secure :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2b08004770 tx=0x7f2b08003c10 comp rx=0 tx=0).stop 2024-04-24T08:32:37.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.052+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 >> 172.21.15.146:0/945549905 conn(0x7f2b18077430 msgr2=0x7f2b180fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.053+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 shutdown_connections 2024-04-24T08:32:37.055 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.054+0000 7f2b1e400640 1 -- 172.21.15.146:0/945549905 wait complete. 2024-04-24T08:32:37.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:32:37.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 -- 172.21.15.146:0/1815347548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b54102010 msgr2=0x7f3b541023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:37.280 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1815347548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b54102010 0x7f3b541023f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3b44009900 tx=0x7f3b4402a410 comp rx=0 tx=0).stop 2024-04-24T08:32:37.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 -- 172.21.15.146:0/1815347548 shutdown_connections 2024-04-24T08:32:37.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1815347548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b541070e0 0x7f3b5410ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1815347548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b541029c0 0x7f3b54106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1815347548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b54102010 0x7f3b541023f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 -- 172.21.15.146:0/1815347548 >> 172.21.15.146:0/1815347548 conn(0x7f3b540fd8d0 msgr2=0x7f3b540ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:37.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 -- 172.21.15.146:0/1815347548 shutdown_connections 2024-04-24T08:32:37.281 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.279+0000 7f3b5a600640 1 -- 172.21.15.146:0/1815347548 wait complete. 2024-04-24T08:32:37.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.280+0000 7f3b5a600640 1 Processor -- start 2024-04-24T08:32:37.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.280+0000 7f3b5a600640 1 -- start start 2024-04-24T08:32:37.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b5a600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b54102010 0x7f3b541986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:37.284 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b5a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b541070e0 0x7f3b54198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:37.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b5a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5419cf20 0x7f3b5419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:37.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b5a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54112170 con 0x7f3b5419cf20 2024-04-24T08:32:37.285 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b5a600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54111ff0 con 0x7f3b54102010 2024-04-24T08:32:37.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b5a600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b541122f0 con 0x7f3b541070e0 2024-04-24T08:32:37.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b4be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b541070e0 0x7f3b54198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:37.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b54102010 0x7f3b541986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:37.286 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5419cf20 0x7f3b5419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:37.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b54102010 0x7f3b541986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.146:49030/0 (socket says 172.21.15.146:49030) 2024-04-24T08:32:37.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b4be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b541070e0 0x7f3b54198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.146:36046/0 (socket says 172.21.15.146:36046) 2024-04-24T08:32:37.287 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 -- 172.21.15.146:0/1268829996 learned_addr learned my addr 172.21.15.146:0/1268829996 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-04-24T08:32:37.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 -- 172.21.15.146:0/1268829996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b541070e0 msgr2=0x7f3b54198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:37.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b541070e0 0x7f3b54198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 -- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5419cf20 msgr2=0x7f3b5419d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:37.288 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5419cf20 0x7f3b5419d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:37.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59200640 1 -- 172.21.15.146:0/1268829996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5419da10 con 0x7f3b54102010 2024-04-24T08:32:37.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.281+0000 7f3b59c00640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5419cf20 0x7f3b5419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:32:37.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.282+0000 7f3b59200640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b54102010 0x7f3b541986b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3b44006010 tx=0x7f3b44004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:37.289 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.282+0000 7f3b52000640 1 -- 172.21.15.146:0/1268829996 <== mon.2 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4403f030 con 0x7f3b54102010 2024-04-24T08:32:37.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.282+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5419dca0 con 0x7f3b54102010 2024-04-24T08:32:37.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.282+0000 7f3b52000640 1 -- 172.21.15.146:0/1268829996 <== mon.2 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b44004440 con 0x7f3b54102010 2024-04-24T08:32:37.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.282+0000 7f3b52000640 1 -- 172.21.15.146:0/1268829996 <== mon.2 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44035070 con 0x7f3b54102010 2024-04-24T08:32:37.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.282+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b541a5460 con 0x7f3b54102010 2024-04-24T08:32:37.292 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.283+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b54069190 con 0x7f3b54102010 2024-04-24T08:32:37.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.286+0000 7f3b52000640 1 -- 172.21.15.146:0/1268829996 <== mon.2 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 27) ==== 138778+0+0 (secure 0 0 0) 0x7f3b44042030 con 0x7f3b54102010 2024-04-24T08:32:37.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.287+0000 7f3b52000640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3b380a3d20 0x7f3b380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:32:37.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.287+0000 7f3b52000640 1 -- 172.21.15.146:0/1268829996 <== mon.2 v2:172.21.15.153:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8231+0+0 (secure 0 0 0) 0x7f3b440f0cf0 con 0x7f3b54102010 2024-04-24T08:32:37.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.287+0000 7f3b4be00640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3b380a3d20 0x7f3b380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:32:37.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.288+0000 7f3b4be00640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3b380a3d20 0x7f3b380a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3b54101010 tx=0x7f3b3c005f90 comp rx=0 tx=0).ready entity=mgr.15087 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:32:37.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.290+0000 7f3b52000640 1 -- 172.21.15.146:0/1268829996 <== mon.2 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b440bce00 con 0x7f3b54102010 2024-04-24T08:32:37.508 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:37.507+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3b541999b0 con 0x7f3b54102010 2024-04-24T08:32:38.058 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.057+0000 7f3b52000640 1 -- 172.21.15.146:0/1268829996 <== mon.2 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) ==== 169+0+0 (secure 0 0 0) 0x7f3b440c1100 con 0x7f3b54102010 2024-04-24T08:32:38.058 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'cephfs_data' removed 2024-04-24T08:32:38.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.062+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3b380a3d20 msgr2=0x7f3b380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:38.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.062+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3b380a3d20 0x7f3b380a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3b54101010 tx=0x7f3b3c005f90 comp rx=0 tx=0).stop 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.062+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b54102010 msgr2=0x7f3b541986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.062+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b54102010 0x7f3b541986b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3b44006010 tx=0x7f3b44004040 comp rx=0 tx=0).stop 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.063+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 shutdown_connections 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.063+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:6800/827025587,v1:172.21.15.146:6801/827025587] conn(0x7f3b380a3d20 0x7f3b380a61e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.063+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b541070e0 0x7f3b54198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.063+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b5419cf20 0x7f3b5419d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.063+0000 7f3b5a600640 1 --2- 172.21.15.146:0/1268829996 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b54102010 0x7f3b541986b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:32:38.065 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.063+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 >> 172.21.15.146:0/1268829996 conn(0x7f3b540fd8d0 msgr2=0x7f3b540ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:32:38.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.063+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 shutdown_connections 2024-04-24T08:32:38.066 INFO:teuthology.orchestra.run.smithi146.stderr:2024-04-24T08:32:38.064+0000 7f3b5a600640 1 -- 172.21.15.146:0/1268829996 wait complete. 2024-04-24T08:32:38.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:32:38.155 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:38 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:32:38.156 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:38 smithi153 ceph-mon[32746]: osdmap e103: 12 total, 12 up, 12 in 2024-04-24T08:32:38.156 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:38 smithi153 ceph-mon[32746]: pgmap v358: 65 pgs: 65 active+clean; 26 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:32:38.156 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:38 smithi153 ceph-mon[32746]: from='client.? 172.21.15.146:0/1268829996' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:38.156 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:38 smithi153 ceph-mon[32746]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:38.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:32:38.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:32:38.220 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-24T08:32:38.220 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T08:32:38.276 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T08:32:38.290 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T08:32:38.305 INFO:tasks.cephadm:Stopping all daemons... 2024-04-24T08:32:38.306 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-24T08:32:38.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a 2024-04-24T08:32:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:38 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:32:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:38 smithi177 ceph-mon[33046]: osdmap e103: 12 total, 12 up, 12 in 2024-04-24T08:32:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:38 smithi177 ceph-mon[33046]: pgmap v358: 65 pgs: 65 active+clean; 26 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:32:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:38 smithi177 ceph-mon[33046]: from='client.? 172.21.15.146:0/1268829996' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:38.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:38 smithi177 ceph-mon[33046]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:38.379 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:32:38.379 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 ceph-mon[27805]: osdmap e103: 12 total, 12 up, 12 in 2024-04-24T08:32:38.380 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 ceph-mon[27805]: pgmap v358: 65 pgs: 65 active+clean; 26 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:32:38.380 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 ceph-mon[27805]: from='client.? 172.21.15.146:0/1268829996' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:38.380 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 ceph-mon[27805]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:32:38.743 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 systemd[1]: Stopping Ceph mon.a for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:39.170 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a[27801]: 2024-04-24T08:32:38.882+0000 7f4fc1200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:32:39.170 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a[27801]: 2024-04-24T08:32:38.882+0000 7f4fc1200700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-24T08:32:39.493 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:39 smithi146 podman[100511]: 2024-04-24 08:32:39.169130433 +0000 UTC m=+0.441124221 container died efc8b69559a513a7efdf615e24e590399949638fbdf77ad57eef3c4043fcafce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-24T08:32:39.916 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:39 smithi146 podman[100511]: 2024-04-24 08:32:39.639951499 +0000 UTC m=+0.911945347 container cleanup efc8b69559a513a7efdf615e24e590399949638fbdf77ad57eef3c4043fcafce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-24T08:32:39.916 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:39 smithi146 bash[100511]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a 2024-04-24T08:32:40.197 INFO:journalctl@ceph.mon.a.smithi146.stdout:Apr 24 08:32:39 smithi146 podman[100525]: 2024-04-24 08:32:39.915922366 +0000 UTC m=+0.735736805 container remove efc8b69559a513a7efdf615e24e590399949638fbdf77ad57eef3c4043fcafce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:32:40.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.a.service' 2024-04-24T08:32:40.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:32:40.327 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-24T08:32:40.328 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-24T08:32:40.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.b 2024-04-24T08:32:40.628 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:40 smithi153 systemd[1]: Stopping Ceph mon.b for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:40.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-b[32742]: 2024-04-24T08:32:40.627+0000 7f1a8c198700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:32:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-b[32742]: 2024-04-24T08:32:40.627+0000 7f1a8c198700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-24T08:32:40.961 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:40 smithi153 podman[68201]: 2024-04-24 08:32:40.842105682 +0000 UTC m=+0.320976743 container died 42f097a73970a7c7d1b07bd4cc64445e405e5f46ce64246f692bc9c705596fcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-24T08:32:41.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:41 smithi153 podman[68201]: 2024-04-24 08:32:41.244107522 +0000 UTC m=+0.722978633 container cleanup 42f097a73970a7c7d1b07bd4cc64445e405e5f46ce64246f692bc9c705596fcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True) 2024-04-24T08:32:41.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 24 08:32:41 smithi153 bash[68201]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-b 2024-04-24T08:32:41.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.b.service' 2024-04-24T08:32:41.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:32:41.704 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-24T08:32:41.704 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-24T08:32:41.704 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.c 2024-04-24T08:32:42.023 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:41 smithi177 systemd[1]: Stopping Ceph mon.c for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:42.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:42 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c[33029]: 2024-04-24T08:32:42.021+0000 7f2fa873c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:32:42.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:42 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c[33029]: 2024-04-24T08:32:42.021+0000 7f2fa873c700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-24T08:32:42.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:42 smithi177 podman[69982]: 2024-04-24 08:32:42.226118447 +0000 UTC m=+0.329785049 container died 7230be4a4d0bde41f33bae0674f3b47ab4e2df76f667e8556f6289dafec7de54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T08:32:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:42 smithi177 podman[69982]: 2024-04-24 08:32:42.621613611 +0000 UTC m=+0.725280210 container cleanup 7230be4a4d0bde41f33bae0674f3b47ab4e2df76f667e8556f6289dafec7de54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-24T08:32:42.815 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 24 08:32:42 smithi177 bash[69982]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mon-c 2024-04-24T08:32:43.002 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mon.c.service' 2024-04-24T08:32:43.045 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:32:43.046 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-24T08:32:43.046 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-24T08:32:43.046 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.x 2024-04-24T08:32:43.492 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:32:43 smithi146 systemd[1]: Stopping Ceph mgr.x for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:43.993 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:32:43 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:32:43.625+0000 7f16b6800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:32:43.993 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:32:43 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x[28032]: 2024-04-24T08:32:43.625+0000 7f16b6800700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-24T08:32:43.994 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:32:43 smithi146 podman[100729]: 2024-04-24 08:32:43.719801273 +0000 UTC m=+0.242312050 container died 55a3410c0b3bf0977c82a65f076e47796d0a19b1f2c80da2afddc02cb277852c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5) 2024-04-24T08:32:44.413 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:32:44 smithi146 podman[100729]: 2024-04-24 08:32:44.210788881 +0000 UTC m=+0.733299626 container cleanup 55a3410c0b3bf0977c82a65f076e47796d0a19b1f2c80da2afddc02cb277852c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T08:32:44.413 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:32:44 smithi146 bash[100729]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x 2024-04-24T08:32:44.742 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Apr 24 08:32:44 smithi146 podman[100743]: 2024-04-24 08:32:44.412109906 +0000 UTC m=+0.678039297 container remove 55a3410c0b3bf0977c82a65f076e47796d0a19b1f2c80da2afddc02cb277852c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-24T08:32:44.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-24T08:32:45.027 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:32:45.027 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-24T08:32:45.027 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-24T08:32:45.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.y 2024-04-24T08:32:45.352 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:32:45 smithi153 systemd[1]: Stopping Ceph mgr.y for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:45.684 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:32:45 smithi153 podman[68325]: 2024-04-24 08:32:45.352132206 +0000 UTC m=+0.124582835 container died a1ab8f6ee4fb11520898884247d2dabd24e99ff684deb4a647bc1c964364dec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:32:45.960 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:32:45 smithi153 podman[68325]: 2024-04-24 08:32:45.766066639 +0000 UTC m=+0.538517269 container cleanup a1ab8f6ee4fb11520898884247d2dabd24e99ff684deb4a647bc1c964364dec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-24T08:32:45.961 INFO:journalctl@ceph.mgr.y.smithi153.stdout:Apr 24 08:32:45 smithi153 bash[68325]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-y 2024-04-24T08:32:46.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-24T08:32:46.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:32:46.144 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-24T08:32:46.144 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-24T08:32:46.144 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.z 2024-04-24T08:32:46.481 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:32:46 smithi177 systemd[1]: Stopping Ceph mgr.z for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:46.768 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:32:46 smithi177 podman[70105]: 2024-04-24 08:32:46.48016105 +0000 UTC m=+0.149350248 container died 78209c103d096f244e29fa401744e682e25b245b73ca8021eae1a3fb567c9fb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:32:47.065 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:32:46 smithi177 podman[70105]: 2024-04-24 08:32:46.858975057 +0000 UTC m=+0.528164254 container cleanup 78209c103d096f244e29fa401744e682e25b245b73ca8021eae1a3fb567c9fb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-24T08:32:47.065 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 24 08:32:46 smithi177 bash[70105]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mgr-z 2024-04-24T08:32:47.232 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-24T08:32:47.265 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:32:47.265 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-24T08:32:47.266 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-24T08:32:47.266 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.0 2024-04-24T08:32:47.742 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:47 smithi146 systemd[1]: Stopping Ceph osd.0 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:48.243 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0[42306]: 2024-04-24T08:32:47.871+0000 7fc880c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:32:48.244 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0[42306]: 2024-04-24T08:32:47.871+0000 7fc880c00700 -1 osd.0 104 *** Got signal Terminated *** 2024-04-24T08:32:48.244 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0[42306]: 2024-04-24T08:32:47.871+0000 7fc880c00700 -1 osd.0 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:32:53.556 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:53 smithi146 podman[100850]: 2024-04-24 08:32:53.252204259 +0000 UTC m=+5.546521743 container died 0a1d796058389fe1d408e5720b21b5667553e9704856e56757a1209592f590de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:32:53.815 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:53 smithi146 podman[100850]: 2024-04-24 08:32:53.672702384 +0000 UTC m=+5.967019908 container cleanup 0a1d796058389fe1d408e5720b21b5667553e9704856e56757a1209592f590de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:32:53.815 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:53 smithi146 bash[100850]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0 2024-04-24T08:32:54.242 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:53 smithi146 podman[100868]: 2024-04-24 08:32:53.982046706 +0000 UTC m=+0.712730786 container remove 0a1d796058389fe1d408e5720b21b5667553e9704856e56757a1209592f590de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-24T08:32:55.054 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:54 smithi146 podman[100956]: 2024-04-24 08:32:54.651854365 +0000 UTC m=+0.037319805 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:32:55.055 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:54 smithi146 podman[100956]: 2024-04-24 08:32:54.783016806 +0000 UTC m=+0.168482283 container create 2343adbe06c3a7b1aaa40e2da120da34163bc6eba0118cb3b4a5d1d40a0d2a3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-24T08:32:55.055 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:55 smithi146 podman[100956]: 2024-04-24 08:32:55.053346998 +0000 UTC m=+0.438812482 container init 2343adbe06c3a7b1aaa40e2da120da34163bc6eba0118cb3b4a5d1d40a0d2a3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:32:55.493 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:55 smithi146 podman[100956]: 2024-04-24 08:32:55.065033981 +0000 UTC m=+0.450499465 container start 2343adbe06c3a7b1aaa40e2da120da34163bc6eba0118cb3b4a5d1d40a0d2a3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-24T08:32:55.493 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:55 smithi146 podman[100956]: 2024-04-24 08:32:55.202042773 +0000 UTC m=+0.587508250 container attach 2343adbe06c3a7b1aaa40e2da120da34163bc6eba0118cb3b4a5d1d40a0d2a3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True) 2024-04-24T08:32:55.879 INFO:journalctl@ceph.osd.0.smithi146.stdout:Apr 24 08:32:55 smithi146 podman[100956]: 2024-04-24 08:32:55.517696028 +0000 UTC m=+0.903161511 container died 2343adbe06c3a7b1aaa40e2da120da34163bc6eba0118cb3b4a5d1d40a0d2a3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T08:32:56.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.0.service' 2024-04-24T08:32:56.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:32:56.394 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-24T08:32:56.394 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-24T08:32:56.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.1 2024-04-24T08:32:56.679 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:32:56 smithi153 systemd[1]: Stopping Ceph osd.1 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:32:56.960 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:32:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1[37340]: 2024-04-24T08:32:56.676+0000 7f1d91162700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:32:56.961 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:32:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1[37340]: 2024-04-24T08:32:56.676+0000 7f1d91162700 -1 osd.1 104 *** Got signal Terminated *** 2024-04-24T08:32:56.961 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:32:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1[37340]: 2024-04-24T08:32:56.676+0000 7f1d91162700 -1 osd.1 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:02.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:02 smithi153 podman[68449]: 2024-04-24 08:33:02.163374959 +0000 UTC m=+5.574341605 container died 26631e674a319f2815b4389a2ddc8cd4c274ea66a6799802e071bd2478064d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-24T08:33:02.800 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:02 smithi153 podman[68449]: 2024-04-24 08:33:02.557721998 +0000 UTC m=+5.968688643 container cleanup 26631e674a319f2815b4389a2ddc8cd4c274ea66a6799802e071bd2478064d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-04-24T08:33:02.801 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:02 smithi153 bash[68449]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1 2024-04-24T08:33:03.126 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:02 smithi153 podman[68466]: 2024-04-24 08:33:02.800533201 +0000 UTC m=+0.640798023 container remove 26631e674a319f2815b4389a2ddc8cd4c274ea66a6799802e071bd2478064d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:33:03.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:03 smithi153 podman[68553]: 2024-04-24 08:33:03.126336141 +0000 UTC m=+0.111619070 container create 33732a765f4e81b0472031e82fc87bd1616aae516f6cedd6eedacbb8bf62dbb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-24T08:33:03.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:03 smithi153 podman[68553]: 2024-04-24 08:33:03.03348408 +0000 UTC m=+0.018767026 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:33:03.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:03 smithi153 podman[68553]: 2024-04-24 08:33:03.353624778 +0000 UTC m=+0.338907707 container init 33732a765f4e81b0472031e82fc87bd1616aae516f6cedd6eedacbb8bf62dbb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-04-24T08:33:03.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:03 smithi153 podman[68553]: 2024-04-24 08:33:03.356782298 +0000 UTC m=+0.342065231 container start 33732a765f4e81b0472031e82fc87bd1616aae516f6cedd6eedacbb8bf62dbb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:33:03.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:03 smithi153 podman[68553]: 2024-04-24 08:33:03.42864086 +0000 UTC m=+0.413923804 container attach 33732a765f4e81b0472031e82fc87bd1616aae516f6cedd6eedacbb8bf62dbb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-24T08:33:03.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:03 smithi153 podman[68553]: 2024-04-24 08:33:03.64687673 +0000 UTC m=+0.632159654 container died 33732a765f4e81b0472031e82fc87bd1616aae516f6cedd6eedacbb8bf62dbb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-24T08:33:04.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:04 smithi153 podman[68579]: 2024-04-24 08:33:04.207482387 +0000 UTC m=+0.550876007 container remove 33732a765f4e81b0472031e82fc87bd1616aae516f6cedd6eedacbb8bf62dbb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True) 2024-04-24T08:33:04.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:04 smithi153 systemd[1]: ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-24T08:33:04.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:04 smithi153 systemd[1]: Stopped Ceph osd.1 for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:33:04.225 INFO:journalctl@ceph.osd.1.smithi153.stdout:Apr 24 08:33:04 smithi153 systemd[1]: ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.1.service: Consumed 15.379s CPU time. 2024-04-24T08:33:04.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.1.service' 2024-04-24T08:33:04.265 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:33:04.265 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-24T08:33:04.265 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-24T08:33:04.265 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.2 2024-04-24T08:33:04.564 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:04 smithi177 systemd[1]: Stopping Ceph osd.2 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:33:04.815 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2[38174]: 2024-04-24T08:33:04.595+0000 7efeb502b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:33:04.815 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2[38174]: 2024-04-24T08:33:04.595+0000 7efeb502b700 -1 osd.2 104 *** Got signal Terminated *** 2024-04-24T08:33:04.815 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2[38174]: 2024-04-24T08:33:04.595+0000 7efeb502b700 -1 osd.2 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:10.315 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:09 smithi177 podman[70243]: 2024-04-24 08:33:09.94668791 +0000 UTC m=+5.483287422 container died e1d67b4161fe06305c5a6f01666ff3c4aadfe35d18779c22dcb63d75257b4e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:33:10.655 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:10 smithi177 podman[70243]: 2024-04-24 08:33:10.412197878 +0000 UTC m=+5.948797389 container cleanup e1d67b4161fe06305c5a6f01666ff3c4aadfe35d18779c22dcb63d75257b4e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:33:10.655 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:10 smithi177 bash[70243]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2 2024-04-24T08:33:10.963 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:10 smithi177 podman[70260]: 2024-04-24 08:33:10.654809451 +0000 UTC m=+0.704001894 container remove e1d67b4161fe06305c5a6f01666ff3c4aadfe35d18779c22dcb63d75257b4e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-24T08:33:11.283 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:10 smithi177 podman[70352]: 2024-04-24 08:33:10.866525189 +0000 UTC m=+0.018341092 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:33:11.284 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:10 smithi177 podman[70352]: 2024-04-24 08:33:10.980633264 +0000 UTC m=+0.132449204 container create 0eef0b4336ae5055cc8b327ccb0add1c627695cd0fb0612f23cbb1e1689a049f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:33:11.284 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:11 smithi177 podman[70352]: 2024-04-24 08:33:11.199826355 +0000 UTC m=+0.351642249 container init 0eef0b4336ae5055cc8b327ccb0add1c627695cd0fb0612f23cbb1e1689a049f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-24T08:33:11.284 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:11 smithi177 podman[70352]: 2024-04-24 08:33:11.202824885 +0000 UTC m=+0.354640780 container start 0eef0b4336ae5055cc8b327ccb0add1c627695cd0fb0612f23cbb1e1689a049f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-24T08:33:11.284 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:11 smithi177 podman[70352]: 2024-04-24 08:33:11.283124762 +0000 UTC m=+0.434940657 container attach 0eef0b4336ae5055cc8b327ccb0add1c627695cd0fb0612f23cbb1e1689a049f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-24T08:33:11.565 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 24 08:33:11 smithi177 podman[70352]: 2024-04-24 08:33:11.500001949 +0000 UTC m=+0.651817844 container died 0eef0b4336ae5055cc8b327ccb0add1c627695cd0fb0612f23cbb1e1689a049f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-04-24T08:33:12.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.2.service' 2024-04-24T08:33:12.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:33:12.113 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-24T08:33:12.113 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-24T08:33:12.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.3 2024-04-24T08:33:12.492 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:12 smithi146 systemd[1]: Stopping Ceph osd.3 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:33:12.993 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3[49262]: 2024-04-24T08:33:12.687+0000 7f816ae00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:33:12.994 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3[49262]: 2024-04-24T08:33:12.687+0000 7f816ae00700 -1 osd.3 104 *** Got signal Terminated *** 2024-04-24T08:33:12.994 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:12 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3[49262]: 2024-04-24T08:33:12.687+0000 7f816ae00700 -1 osd.3 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:14.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:13.901+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:14.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:14 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:14.546+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:15.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:14.886+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:15.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:14.901+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:15.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:15 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:15.533+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:15.743 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:15 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:15.301+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:16.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:16.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:15.858+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:16.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:16 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:16.489+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:16.743 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:16 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:16.305+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:17.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:16 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:16.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:17.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:16.906+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:17.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:17.046+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:17.742 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:17 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:17.256+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:17.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:17 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:17.475+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:18.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:17 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:17.897+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:18.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:17.900+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:18.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:18.053+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:18.472 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:18 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:18.231+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:18.472 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:18 smithi146 podman[101073]: 2024-04-24 08:33:18.178192845 +0000 UTC m=+5.624893611 container died d239f799fda4b565b6783e2160c8d6418d4b9bc9f91c80aa3a1447a610cc9846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T08:33:18.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:18 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:18.510+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:18.743 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:18 smithi146 podman[101073]: 2024-04-24 08:33:18.632268001 +0000 UTC m=+6.078968804 container cleanup d239f799fda4b565b6783e2160c8d6418d4b9bc9f91c80aa3a1447a610cc9846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-24T08:33:18.743 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:18 smithi146 bash[101073]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3 2024-04-24T08:33:19.207 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:18 smithi146 podman[101092]: 2024-04-24 08:33:18.866846372 +0000 UTC m=+0.674582935 container remove d239f799fda4b565b6783e2160c8d6418d4b9bc9f91c80aa3a1447a610cc9846 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0) 2024-04-24T08:33:19.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:18 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:18.917+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:19.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:18.875+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:19.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:19.083+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:19.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:19 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:19.205+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:19.810 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:19 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:19.539+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:19.811 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:19 smithi146 podman[101178]: 2024-04-24 08:33:19.534434539 +0000 UTC m=+0.034259786 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:33:19.811 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:19 smithi146 podman[101178]: 2024-04-24 08:33:19.659412761 +0000 UTC m=+0.159238045 container create 2cf1d5c96b015ecca6359ada5b594ef9e850725be09a26e1e987eb06eab4257a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-24T08:33:20.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:19 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:19.924+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:20.213 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:19 smithi146 podman[101178]: 2024-04-24 08:33:19.94049254 +0000 UTC m=+0.440317824 container init 2cf1d5c96b015ecca6359ada5b594ef9e850725be09a26e1e987eb06eab4257a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-24T08:33:20.213 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:19 smithi146 podman[101178]: 2024-04-24 08:33:19.952091176 +0000 UTC m=+0.451916467 container start 2cf1d5c96b015ecca6359ada5b594ef9e850725be09a26e1e987eb06eab4257a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-24T08:33:20.213 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:20 smithi146 podman[101178]: 2024-04-24 08:33:20.028208561 +0000 UTC m=+0.528033905 container attach 2cf1d5c96b015ecca6359ada5b594ef9e850725be09a26e1e987eb06eab4257a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:33:20.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:19.907+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:20.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:20.060+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:20.493 INFO:journalctl@ceph.osd.3.smithi146.stdout:Apr 24 08:33:20 smithi146 podman[101178]: 2024-04-24 08:33:20.401829193 +0000 UTC m=+0.901654440 container died 2cf1d5c96b015ecca6359ada5b594ef9e850725be09a26e1e987eb06eab4257a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_CLEAN=True) 2024-04-24T08:33:20.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:20 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:20.211+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:20.777 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:20 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:20.505+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:21.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:20 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:20.908+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:21.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.3.service' 2024-04-24T08:33:21.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:20.872+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:21.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:21.087+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:21.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:33:21.321 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-24T08:33:21.321 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-24T08:33:21.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.4 2024-04-24T08:33:21.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:21 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:21.207+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:21.710 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:21 smithi153 systemd[1]: Stopping Ceph osd.4 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:33:21.711 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:21 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:33:21.592+0000 7f43e7585700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:33:21.711 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:21 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:33:21.592+0000 7f43e7585700 -1 osd.4 104 *** Got signal Terminated *** 2024-04-24T08:33:21.711 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:21 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:33:21.592+0000 7f43e7585700 -1 osd.4 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:21.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:21 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:21.531+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:22.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:21 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:21.934+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:22.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:21.904+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:22.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:22.068+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:22.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:22 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:22.223+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:22.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:22 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:22.540+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:23.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:22.896+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:23.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:22 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:22.917+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:23.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:22 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:22.917+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:23.322 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:23.086+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:23.322 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:23.130+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:23.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:23 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:23.216+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:23.961 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:33:23.658+0000 7f43dfb71700 -1 osd.4 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.793474+0000 front 2024-04-24T08:32:58.793509+0000 (oldest deadline 2024-04-24T08:33:23.493624+0000) 2024-04-24T08:33:23.961 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:23.901+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:23.961 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:23 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:23.901+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:23.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:23 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:23.579+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:24.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:23.941+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:24.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:24 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:24.112+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:24.316 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:24 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:24.104+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:24.532 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:24 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:24.244+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:24.921 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:24 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:33:24.662+0000 7f43dfb71700 -1 osd.4 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.793474+0000 front 2024-04-24T08:32:58.793509+0000 (oldest deadline 2024-04-24T08:33:23.493624+0000) 2024-04-24T08:33:24.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:24 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:24.531+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:25.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:24 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:24.921+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:25.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:24 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:24.921+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:25.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:24 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:24.977+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:25.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:24 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:24.901+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:25.315 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:25 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:25.123+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:25.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:25 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:25.105+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:25.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:25 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:25.241+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:25.932 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:25 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4[43860]: 2024-04-24T08:33:25.613+0000 7f43dfb71700 -1 osd.4 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.793474+0000 front 2024-04-24T08:32:58.793509+0000 (oldest deadline 2024-04-24T08:33:23.493624+0000) 2024-04-24T08:33:25.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:25 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:25.579+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:26.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:25 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:25.936+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:26.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:25 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:25.931+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:26.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:25 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:25.931+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:26.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:25 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:25.922+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:26.315 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:26 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:26.126+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:26.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:26 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:26.069+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:26.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:26 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:26.069+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:26.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:26 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:26.196+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:26.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:26 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:26.619+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:26.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:26 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:26.619+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:27.210 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:27 smithi153 podman[68671]: 2024-04-24 08:33:27.045542195 +0000 UTC m=+5.530258245 container died 12e4dc0d8ba3a2adc7b74702bb113e8f93f682034cf73367f13e76696bd2f0e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5) 2024-04-24T08:33:27.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:26 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:26.929+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:27.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:26 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:26.916+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:27.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:26 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:26.916+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:27.315 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:26 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:26.953+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:27.315 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:27 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:27.158+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:27.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:27 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:27.058+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:27.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:27 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:27.058+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:27.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:27 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:27.172+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:27.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:27 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:27.172+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:27.677 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:27 smithi153 podman[68671]: 2024-04-24 08:33:27.409230076 +0000 UTC m=+5.893946123 container cleanup 12e4dc0d8ba3a2adc7b74702bb113e8f93f682034cf73367f13e76696bd2f0e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-24T08:33:27.677 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:27 smithi153 bash[68671]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4 2024-04-24T08:33:27.934 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:27 smithi153 podman[68688]: 2024-04-24 08:33:27.676843989 +0000 UTC m=+0.629494557 container remove 12e4dc0d8ba3a2adc7b74702bb113e8f93f682034cf73367f13e76696bd2f0e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-24T08:33:27.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:27 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:27.605+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:27.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:27 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:27.605+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:28.188 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:27 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:27.933+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:28.188 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:27 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:27.965+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:28.189 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:27 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:27.965+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:28.189 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:28 smithi153 podman[68778]: 2024-04-24 08:33:27.934506694 +0000 UTC m=+0.018105229 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:33:28.189 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:28 smithi153 podman[68778]: 2024-04-24 08:33:28.061444087 +0000 UTC m=+0.145042625 container create fc88a6c9d5f27babceb609e23de54af391cd20762c65d8338b0fa858a834873e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-24T08:33:28.315 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:28 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:28.170+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:28.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:28 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:28.080+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:28.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:28 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:28.080+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:28.316 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:27 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:27.921+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:28.316 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:27 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:27.921+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:56.462186+0000 front 2024-04-24T08:32:56.462267+0000 (oldest deadline 2024-04-24T08:33:27.662743+0000) 2024-04-24T08:33:28.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:28.171+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:28.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:28.171+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:28.572 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:28 smithi153 podman[68778]: 2024-04-24 08:33:28.280719362 +0000 UTC m=+0.364317898 container init fc88a6c9d5f27babceb609e23de54af391cd20762c65d8338b0fa858a834873e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-24T08:33:28.572 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:28 smithi153 podman[68778]: 2024-04-24 08:33:28.284456603 +0000 UTC m=+0.368055139 container start fc88a6c9d5f27babceb609e23de54af391cd20762c65d8338b0fa858a834873e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-24T08:33:28.572 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:28 smithi153 podman[68778]: 2024-04-24 08:33:28.355380675 +0000 UTC m=+0.438979228 container attach fc88a6c9d5f27babceb609e23de54af391cd20762c65d8338b0fa858a834873e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-24T08:33:28.866 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 24 08:33:28 smithi153 podman[68778]: 2024-04-24 08:33:28.572576283 +0000 UTC m=+0.656174820 container died fc88a6c9d5f27babceb609e23de54af391cd20762c65d8338b0fa858a834873e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-24T08:33:28.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:28.634+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:28.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:28 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:28.634+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:29.151 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:28 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:28.909+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:29.151 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:28 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:28.943+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:29.151 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:28 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:28.943+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:29.172 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.4.service' 2024-04-24T08:33:29.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:33:29.210 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-24T08:33:29.210 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-24T08:33:29.211 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.5 2024-04-24T08:33:29.237 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:29.146+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:29.238 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:28 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:28.925+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:29.238 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:28 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:28.925+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:56.462186+0000 front 2024-04-24T08:32:56.462267+0000 (oldest deadline 2024-04-24T08:33:27.662743+0000) 2024-04-24T08:33:29.238 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:29.066+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:29.238 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:29.066+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:29.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:29 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:29.140+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:29.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:29 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:29.140+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:29.565 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:29 smithi177 systemd[1]: Stopping Ceph osd.5 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:33:29.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:29 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:29.646+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:29.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:29 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:29.646+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:30.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:29.679+0000 7f791b83e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:33:30.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:29.679+0000 7f791b83e700 -1 osd.5 104 *** Got signal Terminated *** 2024-04-24T08:33:30.065 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:29.679+0000 7f791b83e700 -1 osd.5 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:30.066 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:29.877+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:30.066 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:29 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:29.877+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:56.462186+0000 front 2024-04-24T08:32:56.462267+0000 (oldest deadline 2024-04-24T08:33:27.662743+0000) 2024-04-24T08:33:30.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:29 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:29.898+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:30.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:29 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:29.932+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:30.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:29 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:29.932+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:30.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:30 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:30.144+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:30.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:30 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:30.144+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:30.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:30 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:30.152+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:30.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:30 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:30.112+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:30.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:30 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:30.112+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:30.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:30 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:30.629+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:30.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:30 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:30.629+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:30.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:30 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:30.629+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:31.181 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:31 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:31.090+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:31.181 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:31 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:31.090+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:31.182 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:30 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:30.873+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:31.182 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:30 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:30.873+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:56.462186+0000 front 2024-04-24T08:32:56.462267+0000 (oldest deadline 2024-04-24T08:33:27.662743+0000) 2024-04-24T08:33:31.182 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:30 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:30.873+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.663108+0000 front 2024-04-24T08:33:07.663250+0000 (oldest deadline 2024-04-24T08:33:29.963223+0000) 2024-04-24T08:33:31.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:30 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:30.893+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:31.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:30 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:30.910+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:31.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:30 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:30.910+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:31.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:31 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:31.161+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:31.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:31 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:31.161+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:31.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:31 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:31.180+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:31.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:31 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:31.585+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:31.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:31 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:31.585+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:31.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:31 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:31.585+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:32.130 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:31 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:31.877+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:32.131 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:31 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:31.877+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:56.462186+0000 front 2024-04-24T08:32:56.462267+0000 (oldest deadline 2024-04-24T08:33:27.662743+0000) 2024-04-24T08:33:32.131 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:31 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:31.877+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.663108+0000 front 2024-04-24T08:33:07.663250+0000 (oldest deadline 2024-04-24T08:33:29.963223+0000) 2024-04-24T08:33:32.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:31 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:31.898+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:32.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:31 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:31.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:32.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:31 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:31.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:32.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:31 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:31.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:32.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:32 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:32.183+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:32.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:32 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:32.183+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:32.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:32 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:32.157+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:32.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:32 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:32.129+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:32.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:32 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:32.129+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:32.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:32 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:32.129+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:32.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:32 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:32.626+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:32.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:32 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:32.626+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:32.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:32 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:32.626+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:33.168 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:32 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:32.868+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:33.168 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:32 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:32.868+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:56.462186+0000 front 2024-04-24T08:32:56.462267+0000 (oldest deadline 2024-04-24T08:33:27.662743+0000) 2024-04-24T08:33:33.169 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:32 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:32.868+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.663108+0000 front 2024-04-24T08:33:07.663250+0000 (oldest deadline 2024-04-24T08:33:29.963223+0000) 2024-04-24T08:33:33.169 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:33.090+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:33.169 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:33.090+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:33.169 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:33.090+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:33.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:32 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:32.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:33.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:32 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:32.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:33.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:32 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:32.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:33.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:32 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:32.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:33.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:32 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:32.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:33.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:33 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:33.190+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:33.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:33 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:33.190+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:33.517 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:33 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:33.190+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:33.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:33.167+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:33.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:33.167+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:33.958 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:33 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:33.906+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:33.958 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:33 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:33.906+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:33.958 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:33 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:33.906+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:33.958 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:33 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:33.956+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:33.958 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:33 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:33.956+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:33.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:33 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:33.647+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:33.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:33 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:33.647+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:33.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:33 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:33.647+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:34.189 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:33.867+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:47.160886+0000 front 2024-04-24T08:32:47.160950+0000 (oldest deadline 2024-04-24T08:33:13.561268+0000) 2024-04-24T08:33:34.189 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:33.867+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:56.462186+0000 front 2024-04-24T08:32:56.462267+0000 (oldest deadline 2024-04-24T08:33:27.662743+0000) 2024-04-24T08:33:34.189 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:33 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5[45111]: 2024-04-24T08:33:33.867+0000 7f7913e2a700 -1 osd.5 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.663108+0000 front 2024-04-24T08:33:07.663250+0000 (oldest deadline 2024-04-24T08:33:29.963223+0000) 2024-04-24T08:33:34.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:34 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:34.101+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:34.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:34 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:34.101+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:34.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:34 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:34.101+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:34.492 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:34 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:34.141+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:34.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:34 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:34.141+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:34.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:34 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:34.141+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:34.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:34 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:34.188+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:34.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:34 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:34.188+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:34.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:34 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:34.624+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:34.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:34 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:34.624+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:34.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:34 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:34.624+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:35.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:34 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:34.989+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:35.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:34 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:34.989+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:35.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:34 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:34.945+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:35.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:34 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:34.945+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:35.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:34 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:34.945+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:35.438 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:35 smithi177 podman[70470]: 2024-04-24 08:33:35.101409804 +0000 UTC m=+5.696856015 container died 31264f21173c5c37b981f9ee5c73fa1874bde920bd3d0c1ed65b3e588b39e25d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-24T08:33:35.441 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:35.160+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:35.441 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:35.160+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:35.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:35.094+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:35.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:35.094+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:35.442 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:35 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:35.094+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:35.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:35.192+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:35.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:35.192+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:35.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:35.192+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:35.815 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:35 smithi177 podman[70470]: 2024-04-24 08:33:35.437874543 +0000 UTC m=+6.033320744 container cleanup 31264f21173c5c37b981f9ee5c73fa1874bde920bd3d0c1ed65b3e588b39e25d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:33:35.815 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:35 smithi177 bash[70470]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5 2024-04-24T08:33:35.816 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:35 smithi177 podman[70488]: 2024-04-24 08:33:35.605606005 +0000 UTC m=+0.503350069 container remove 31264f21173c5c37b981f9ee5c73fa1874bde920bd3d0c1ed65b3e588b39e25d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:33:35.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:35.597+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:35.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:35.597+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:35.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:35 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:35.597+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:36.267 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:36 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:36.168+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:36.267 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:36 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:36.168+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:36.268 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:36 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:36.069+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:36.268 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:36 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:36.069+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:36.268 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:36 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:36.069+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:36.268 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:36 smithi177 podman[70577]: 2024-04-24 08:33:35.905562118 +0000 UTC m=+0.017851990 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:33:36.268 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:36 smithi177 podman[70577]: 2024-04-24 08:33:36.006303448 +0000 UTC m=+0.118593311 container create 4447c4b942a168e28e226acb506b505e90845d2b5874f29bda391e037c58bfb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T08:33:36.269 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:36 smithi177 podman[70577]: 2024-04-24 08:33:36.210220408 +0000 UTC m=+0.322510271 container init 4447c4b942a168e28e226acb506b505e90845d2b5874f29bda391e037c58bfb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:33:36.269 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:36 smithi177 podman[70577]: 2024-04-24 08:33:36.213264431 +0000 UTC m=+0.325554294 container start 4447c4b942a168e28e226acb506b505e90845d2b5874f29bda391e037c58bfb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:33:36.269 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:36 smithi177 podman[70577]: 2024-04-24 08:33:36.267121337 +0000 UTC m=+0.379411218 container attach 4447c4b942a168e28e226acb506b505e90845d2b5874f29bda391e037c58bfb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:33:36.461 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:36 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:36.014+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:36.462 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:36 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:36.014+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:36.462 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:35.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:36.462 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:35.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:36.462 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:35 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:35.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:36.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:36 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:36.191+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:36.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:36 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:36.191+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:36.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:36 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:36.191+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:36.565 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 24 08:33:36 smithi177 podman[70577]: 2024-04-24 08:33:36.503034617 +0000 UTC m=+0.615324476 container died 4447c4b942a168e28e226acb506b505e90845d2b5874f29bda391e037c58bfb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0) 2024-04-24T08:33:36.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:36 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:36.608+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:36.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:36 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:36.608+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:36.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:36 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:36.608+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:37.091 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.5.service' 2024-04-24T08:33:37.170 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:33:37.170 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-24T08:33:37.170 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-24T08:33:37.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.6 2024-04-24T08:33:37.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:36 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:36.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:37.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:36 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:36.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:37.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:36 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:36.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:37.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:37 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:37.014+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:37.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:37 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:37.014+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:37.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:37.241+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:37.494 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:37.241+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:37.494 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:37.241+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:37.494 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:37 smithi146 systemd[1]: Stopping Ceph osd.6 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:33:37.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:37 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:37.210+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:37.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:37 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:37.211+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:37.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:37 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:37.098+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:37.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:37 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:37.098+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:37.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:37 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:37.098+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:37.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:37.611+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:37.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:37.611+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:37.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:37.611+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:37.993 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:37.749+0000 7fdfd9800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:33:37.993 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:37.749+0000 7fdfd9800700 -1 osd.6 104 *** Got signal Terminated *** 2024-04-24T08:33:37.994 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:37 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:37.749+0000 7fdfd9800700 -1 osd.6 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:38.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:38 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:38.008+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:38.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:38 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:38.008+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:38.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:37 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:37.906+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:38.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:37 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:37.906+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:38.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:37 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:37.906+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:38.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:38.232+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:38.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:38.233+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:38.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:38.233+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:38.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:38 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:38.195+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:38.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:38 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:38.195+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:38.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:38 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:38.081+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:38.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:38 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:38.081+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:38.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:38 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:38.081+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:38.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:38.584+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:38.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:38.584+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:38.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:38.584+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:39.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:38 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:38.977+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:39.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:38 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:38.977+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:39.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:38 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:38.948+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:39.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:38 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:38.948+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:39.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:38 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:38.948+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:39.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:39.207+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:39.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:39.207+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:39.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:39.207+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:39.494 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:39.207+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:12.996850+0000 front 2024-04-24T08:33:12.996648+0000 (oldest deadline 2024-04-24T08:33:38.296550+0000) 2024-04-24T08:33:39.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:39 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:39.183+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:39.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:39 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:39.183+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:39.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:39 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:39.126+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:39.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:39 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:39.126+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:39.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:39 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:39.126+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:39.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:39.570+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:39.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:39.570+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:39.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:39 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:39.570+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:40.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:39 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:39.969+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:40.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:39 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:39.969+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:40.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:39 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:39.913+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:40.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:39 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:39.913+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:40.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:39 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:39.913+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:40.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:40 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:40.190+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:40.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:40 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:40.190+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:40.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:40 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:40.190+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:40.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:40 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:40.190+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:12.996850+0000 front 2024-04-24T08:33:12.996648+0000 (oldest deadline 2024-04-24T08:33:38.296550+0000) 2024-04-24T08:33:40.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:40 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:40.187+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:40.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:40 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:40.187+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:40.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:40 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:40.121+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:40.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:40 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:40.121+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:40.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:40 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:40.121+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:40.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:40 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:40.558+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:40.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:40 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:40.558+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:40.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:40 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:40.558+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:41.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:40.922+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:41.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:40.922+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:41.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:40.954+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:41.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:40.954+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:41.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:40.954+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:41.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:40 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:40.954+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:41.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:41 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:41.206+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:41.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:41 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:41.206+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:41.493 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:41 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:41.206+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:41.494 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:41 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:41.206+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:12.996850+0000 front 2024-04-24T08:33:12.996648+0000 (oldest deadline 2024-04-24T08:33:38.296550+0000) 2024-04-24T08:33:41.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:41 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:41.180+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:41.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:41 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:41.180+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:41.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:41 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:41.127+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:41.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:41 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:41.127+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:41.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:41 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:41.127+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:41.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:41 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:41.573+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:41.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:41 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:41.573+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:41.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:41 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:41.573+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:42.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:41 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:41.935+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:42.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:41 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:41.935+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:42.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:41 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:41.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:42.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:41 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:41.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:42.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:41 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:41.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:42.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:41 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:41.942+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:42.533 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:42 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:42.254+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:52.589625+0000 front 2024-04-24T08:32:52.590177+0000 (oldest deadline 2024-04-24T08:33:14.289833+0000) 2024-04-24T08:33:42.534 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:42 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:42.254+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:57.093256+0000 front 2024-04-24T08:32:57.093053+0000 (oldest deadline 2024-04-24T08:33:27.094393+0000) 2024-04-24T08:33:42.534 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:42 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:42.254+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:07.095894+0000 front 2024-04-24T08:33:07.095338+0000 (oldest deadline 2024-04-24T08:33:32.995407+0000) 2024-04-24T08:33:42.534 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:42 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6[55795]: 2024-04-24T08:33:42.254+0000 7fdfd5000700 -1 osd.6 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:12.996850+0000 front 2024-04-24T08:33:12.996648+0000 (oldest deadline 2024-04-24T08:33:38.296550+0000) 2024-04-24T08:33:42.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:42 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:42.180+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:42.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:42 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:42.180+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:42.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:42 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:42.107+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:42.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:42 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:42.107+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:42.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:42 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:42.107+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:42.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:42 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:42.532+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:42.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:42 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:42.532+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:42.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:42 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:42.532+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:43.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:42.908+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:43.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:42.908+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:43.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:42.908+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:43.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:42.988+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:43.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:42.988+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:43.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:42.988+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:43.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:42 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:42.988+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:43.543 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:43 smithi146 podman[101294]: 2024-04-24 08:33:43.283437501 +0000 UTC m=+5.680201890 container died 2c57afc8ac37eec474d0b0b9f04171d2037070bd5a9390cf8cab63f9281a83b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-24T08:33:43.544 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:43 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:43.511+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:43.544 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:43 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:43.511+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:43.544 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:43 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:43.511+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:43.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:43 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:43.222+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:43.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:43 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:43.222+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:43.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:43 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:43.222+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:43.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:43 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:43.117+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:43.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:43 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:43.117+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:43.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:43 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:43.117+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:43.885 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:43 smithi146 podman[101294]: 2024-04-24 08:33:43.700237625 +0000 UTC m=+6.097002045 container cleanup 2c57afc8ac37eec474d0b0b9f04171d2037070bd5a9390cf8cab63f9281a83b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-24T08:33:43.886 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:43 smithi146 bash[101294]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6 2024-04-24T08:33:44.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:43 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:43.928+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:44.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:43 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:43.928+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:44.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:43 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:43.928+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:44.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:44 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:44.001+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:44.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:44 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:44.001+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:44.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:44 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:44.001+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:44.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:44 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:44.001+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:44.242 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:43 smithi146 podman[101312]: 2024-04-24 08:33:43.884508615 +0000 UTC m=+0.585641627 container remove 2c57afc8ac37eec474d0b0b9f04171d2037070bd5a9390cf8cab63f9281a83b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:33:44.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:44 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:44.206+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:44.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:44 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:44.206+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:44.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:44 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:44.206+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:44.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:44 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:44.075+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:44.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:44 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:44.075+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:44.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:44 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:44.075+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:44.828 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:44 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:44.511+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:44.828 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:44 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:44.511+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:44.829 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:44 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:44.511+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:44.829 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:44 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:44.511+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:44.829 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:44 smithi146 podman[101400]: 2024-04-24 08:33:44.549509341 +0000 UTC m=+0.034194283 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:33:44.829 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:44 smithi146 podman[101400]: 2024-04-24 08:33:44.651870582 +0000 UTC m=+0.136555550 container create ba9dcbc5bac9a455e833e9a68aa2b888d45121b52e80f749ce1bdf5211018e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-24T08:33:45.242 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:44 smithi146 podman[101400]: 2024-04-24 08:33:44.977247266 +0000 UTC m=+0.461932272 container init ba9dcbc5bac9a455e833e9a68aa2b888d45121b52e80f749ce1bdf5211018e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T08:33:45.243 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:44 smithi146 podman[101400]: 2024-04-24 08:33:44.988818371 +0000 UTC m=+0.473503378 container start ba9dcbc5bac9a455e833e9a68aa2b888d45121b52e80f749ce1bdf5211018e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:33:45.243 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:45 smithi146 podman[101400]: 2024-04-24 08:33:45.070971111 +0000 UTC m=+0.555656141 container attach ba9dcbc5bac9a455e833e9a68aa2b888d45121b52e80f749ce1bdf5211018e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-24T08:33:45.378 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:44 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:44.975+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:45.379 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:44 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:44.975+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:45.379 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:44 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:44.975+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:45.379 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:45 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.012+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:45.379 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:45 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.012+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:45.379 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:45 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.012+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:45.379 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:45 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.012+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:45.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:45 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:45.223+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:45.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:45 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:45.223+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:45.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:45 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:45.223+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:45.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:45 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:45.105+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:45.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:45 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:45.105+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:45.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:45 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:45.105+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:45.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:45 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:45.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:45.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:45 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:45.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:45.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:45 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:45.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:45.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:45 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:45.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:45.743 INFO:journalctl@ceph.osd.6.smithi146.stdout:Apr 24 08:33:45 smithi146 podman[101400]: 2024-04-24 08:33:45.43915279 +0000 UTC m=+0.923837738 container died ba9dcbc5bac9a455e833e9a68aa2b888d45121b52e80f749ce1bdf5211018e60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:33:46.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.6.service' 2024-04-24T08:33:46.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:45 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:45.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:46.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:45 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:45.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:46.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:45 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:45.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:46.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.998+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:46.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.998+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:46.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.998+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:46.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:45.998+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:46.312 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:33:46.312 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-24T08:33:46.313 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-24T08:33:46.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.7 2024-04-24T08:33:46.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:46 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:46.186+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:46.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:46 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:46.186+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:46.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:46 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:46.186+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:46.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:46 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:46.064+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:46.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:46 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:46.064+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:46.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:46 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:46.064+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:46.620 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:46 smithi153 systemd[1]: Stopping Ceph osd.7 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:33:46.913 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:46.618+0000 7f5a31847700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:33:46.913 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:46.618+0000 7f5a31847700 -1 osd.7 104 *** Got signal Terminated *** 2024-04-24T08:33:46.913 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:46.618+0000 7f5a31847700 -1 osd.7 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:46.913 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:46.911+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:46.913 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:46.911+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:46.914 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:46.911+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:46.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:46 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:46.584+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:46.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:46 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:46.584+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:46.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:46 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:46.584+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:46.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:46 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:46.584+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:47.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:46.953+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:47.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:46.953+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:47.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:46.953+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:47.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:46 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:46.953+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:47.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:47 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:47.195+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:47.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:47 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:47.195+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:47.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:47 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:47.195+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:47.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:47 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:47.095+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:47.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:47 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:47.095+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:47.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:47 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:47.095+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:47.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:47.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:47.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:47.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:47.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:47.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:47.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:47 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:47.541+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:48.210 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:47.907+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:48.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:47.907+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:48.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:47.907+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:48.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:47.921+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:48.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:47.921+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:48.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:47.921+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:48.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:47.921+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:48.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:47 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:47.921+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:48.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:48 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:48.160+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:48.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:48 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:48.160+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:48.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:48 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:48.160+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:48.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:48 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:48.130+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:48.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:48 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:48.130+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:48.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:48 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:48.130+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:48.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:48 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:48.540+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:48.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:48 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:48.540+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:48.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:48 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:48.540+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:48.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:48 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:48.540+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:49.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:48.924+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:49.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:48.924+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:49.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:48.924+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:49.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:48.924+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:22.855575+0000 front 2024-04-24T08:33:22.855556+0000 (oldest deadline 2024-04-24T08:33:48.155742+0000) 2024-04-24T08:33:49.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:48.909+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:49.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:48.909+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:49.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:48.909+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:49.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:48.909+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:49.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:48 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:48.909+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:49.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:49 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:49.170+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:49.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:49 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:49.170+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:49.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:49 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:49.170+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:49.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:49 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:49.182+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:49.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:49 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:49.182+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:49.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:49 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:49.182+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:49.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:49 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:49.538+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:49.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:49 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:49.538+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:49.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:49 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:49.538+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:49.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:49 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:49.538+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:50.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:49.906+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:50.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:49.906+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:50.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:49.906+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:50.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:49.906+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:22.855575+0000 front 2024-04-24T08:33:22.855556+0000 (oldest deadline 2024-04-24T08:33:48.155742+0000) 2024-04-24T08:33:50.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:49.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:50.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:49.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:50.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:49.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:50.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:49.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:50.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:49 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:49.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:50.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:50 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:50.172+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:50.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:50 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:50.172+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:50.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:50 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:50.172+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:50.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:50 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:50.198+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:50.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:50 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:50.198+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:50.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:50 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:50.198+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:50.994 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:50.582+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:50.994 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:50.582+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:50.994 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:50.582+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:50.994 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:50 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:50.582+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:51.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:50.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.551894+0000 front 2024-04-24T08:32:59.551920+0000 (oldest deadline 2024-04-24T08:33:24.851994+0000) 2024-04-24T08:33:51.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:50.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:09.353998+0000 front 2024-04-24T08:33:09.353919+0000 (oldest deadline 2024-04-24T08:33:32.254035+0000) 2024-04-24T08:33:51.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:50.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.955313+0000 front 2024-04-24T08:33:16.955452+0000 (oldest deadline 2024-04-24T08:33:42.855210+0000) 2024-04-24T08:33:51.211 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7[50831]: 2024-04-24T08:33:50.947+0000 7f5a2de3b700 -1 osd.7 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:22.855575+0000 front 2024-04-24T08:33:22.855556+0000 (oldest deadline 2024-04-24T08:33:48.155742+0000) 2024-04-24T08:33:51.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:50.818+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:51.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:50.818+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:51.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:50.818+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:51.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:50.818+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:51.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:50.818+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:51.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:51.151+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:51.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:51.151+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:51.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:51.151+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:51.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:51.151+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:51.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:51.220+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:51.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:51.220+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:51.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:51.220+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:51.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:51 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:51.220+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:51.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:51 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:51.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:51.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:51 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:51.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:51.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:51 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:51.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:51.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:51 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:51.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:52.079 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:51 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:51.781+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:52.079 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:51 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:51.781+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:52.080 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:51 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:51.781+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:52.080 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:51 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:51.781+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:52.080 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:51 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:51.781+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:52.330 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:52 smithi153 podman[68895]: 2024-04-24 08:33:52.078988799 +0000 UTC m=+5.580602376 container died 4fdb395010caab8ecff3d65f6f2054446a0272b3b411a23b8197e6145c90fca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True) 2024-04-24T08:33:52.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:52.245+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:52.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:52.245+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:52.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:52.245+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:52.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:52.245+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:52.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:52.125+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:52.567 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:52.125+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:52.567 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:52.125+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:52.567 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:52 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:52.125+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:52.630 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:52 smithi153 podman[68895]: 2024-04-24 08:33:52.411542116 +0000 UTC m=+5.913155692 container cleanup 4fdb395010caab8ecff3d65f6f2054446a0272b3b411a23b8197e6145c90fca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-24T08:33:52.631 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:52 smithi153 bash[68895]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7 2024-04-24T08:33:52.960 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:52 smithi153 podman[68912]: 2024-04-24 08:33:52.629892028 +0000 UTC m=+0.543506069 container remove 4fdb395010caab8ecff3d65f6f2054446a0272b3b411a23b8197e6145c90fca0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-24T08:33:52.961 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:52.802+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:52.961 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:52.802+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:52.961 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:52.802+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:52.961 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:52.802+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:52.962 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:52 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:52.802+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:52.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:52.602+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:52.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:52.602+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:52.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:52.602+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:52.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:52 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:52.602+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:53.301 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:53 smithi153 podman[69001]: 2024-04-24 08:33:52.938500507 +0000 UTC m=+0.018176019 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:33:53.301 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:53 smithi153 podman[69001]: 2024-04-24 08:33:53.073265728 +0000 UTC m=+0.152941215 container create 32451a8b9e27973cd1571bfceb3d665b0776de701304f56653d8c30a71a4655a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:33:53.301 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:53 smithi153 podman[69001]: 2024-04-24 08:33:53.300768444 +0000 UTC m=+0.380443929 container init 32451a8b9e27973cd1571bfceb3d665b0776de701304f56653d8c30a71a4655a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-24T08:33:53.466 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:53.139+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:53.467 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:53.139+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:53.467 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:53.139+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:53.467 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:53.139+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:53.467 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:53.279+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:53.467 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:53.279+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:53.468 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:53.279+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:53.468 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:53 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:53.279+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:53.596 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:53 smithi153 podman[69001]: 2024-04-24 08:33:53.304938641 +0000 UTC m=+0.384614134 container start 32451a8b9e27973cd1571bfceb3d665b0776de701304f56653d8c30a71a4655a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T08:33:53.596 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:53 smithi153 podman[69001]: 2024-04-24 08:33:53.384091537 +0000 UTC m=+0.463767030 container attach 32451a8b9e27973cd1571bfceb3d665b0776de701304f56653d8c30a71a4655a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-24T08:33:53.847 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 24 08:33:53 smithi153 podman[69001]: 2024-04-24 08:33:53.595849106 +0000 UTC m=+0.675524601 container died 32451a8b9e27973cd1571bfceb3d665b0776de701304f56653d8c30a71a4655a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-24T08:33:53.894 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:53.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:53.895 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:53.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:53.895 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:53.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:53.895 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:53 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:53.628+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:54.138 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:53 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:53.846+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:54.138 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:53 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:53.846+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:54.139 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:53 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:53.846+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:54.139 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:53 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:53.846+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:54.139 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:53 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:53.846+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:54.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.7.service' 2024-04-24T08:33:54.196 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:33:54.196 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-24T08:33:54.197 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-24T08:33:54.197 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.8 2024-04-24T08:33:54.506 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:54.154+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:54.506 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:54.154+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:54.506 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:54.154+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:54.506 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:54.154+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:54.507 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 systemd[1]: Stopping Ceph osd.8 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:33:54.507 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:54.312+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:54.507 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:54.312+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:54.507 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:54.312+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:54.507 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:54.312+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:54.815 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:54.504+0000 7fa576713700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:33:54.815 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:54.504+0000 7fa576713700 -1 osd.8 104 *** Got signal Terminated *** 2024-04-24T08:33:54.816 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:54 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:54.505+0000 7fa576713700 -1 osd.8 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:33:54.930 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:54.588+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:54.931 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:54.588+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:54.931 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:54.588+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:54.931 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:54 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:54.588+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:55.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:54 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:54.876+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:55.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:54 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:54.876+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:55.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:54 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:54.876+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:55.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:54 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:54.876+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:55.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:54 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:54.876+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:55.432 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:55.276+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:55.432 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:55.276+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:55.432 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:55.276+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:55.432 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:55.276+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:55.432 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:55.157+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:55.433 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:55.157+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:55.433 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:55.157+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:55.433 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:55 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:55.157+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:55.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:55 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:55.613+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:55.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:55 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:55.613+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:55.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:55 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:55.613+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:55.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:55 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:55.613+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:56.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:55 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:55.902+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:56.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:55 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:55.902+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:56.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:55 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:55.902+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:56.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:55 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:55.902+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:56.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:55 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:55.902+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:56.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:56.194+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:56.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:56.194+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:56.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:56.194+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:56.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:56.194+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:56.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:56.271+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:56.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:56.271+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:56.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:56.271+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:56.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:56 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:56.271+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:56.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:56 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:56.578+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:56.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:56 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:56.578+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:56.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:56 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:56.578+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:56.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:56 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:56.578+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:57.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:56.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:57.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:56.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:57.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:56.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:57.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:56.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:57.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:56 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:56.864+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:57.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:57.310+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:57.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:57.310+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:57.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:57.310+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:57.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:57.310+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:57.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:57.208+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:57.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:57.208+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:57.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:57.208+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:57.567 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:57 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:57.208+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:57.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:57 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:57.557+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:57.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:57 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:57.557+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:57.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:57 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:57.557+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:57.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:57 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:57.557+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:57.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:57 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:57.557+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:33:58.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:57 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:57.887+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:58.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:57 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:57.887+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:58.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:57 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:57.887+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:58.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:57 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:57.887+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:58.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:57 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:57.887+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:58.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:57 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:57.887+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:33:58.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:58.318+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:58.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:58.318+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:58.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:58.318+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:58.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:58.318+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:58.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:58.188+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:58.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:58.188+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:58.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:58.188+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:58.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:58 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:58.188+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:58.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:58.514+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:58.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:58.514+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:58.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:58.514+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:58.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:58.514+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:58.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:58 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:58.514+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:33:59.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:58 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:58.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:33:59.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:58 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:58.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:33:59.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:58 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:58.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:33:59.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:58 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:58.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:33:59.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:58 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:58.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:33:59.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:58 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:58.866+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:33:59.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:59.339+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:58.178074+0000 front 2024-04-24T08:32:58.177959+0000 (oldest deadline 2024-04-24T08:33:22.877884+0000) 2024-04-24T08:33:59.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:59.339+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.978659+0000 front 2024-04-24T08:33:06.978808+0000 (oldest deadline 2024-04-24T08:33:32.278847+0000) 2024-04-24T08:33:59.565 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:59.339+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.280266+0000 front 2024-04-24T08:33:16.280371+0000 (oldest deadline 2024-04-24T08:33:42.680573+0000) 2024-04-24T08:33:59.566 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8[51659]: 2024-04-24T08:33:59.339+0000 7fa56ecff700 -1 osd.8 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:26.181759+0000 front 2024-04-24T08:33:26.181845+0000 (oldest deadline 2024-04-24T08:33:50.281624+0000) 2024-04-24T08:33:59.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:59.195+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:33:59.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:59.195+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:33:59.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:59.195+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:33:59.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:59.195+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:33:59.567 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:33:59 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:33:59.195+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:33:59.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:59.515+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:33:59.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:59.515+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:33:59.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:59.515+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:33:59.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:59.515+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:33:59.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:33:59 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:33:59.515+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:00.208 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:33:59 smithi177 podman[70685]: 2024-04-24 08:33:59.935445379 +0000 UTC m=+5.548238984 container died 526e4b73ae858c5db7e9c5958123b096208832a6a19e1faeb48ce38ee1c8d9c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-24T08:34:00.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:59 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:59.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:00.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:59 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:59.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:00.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:59 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:59.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:00.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:59 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:59.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:00.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:59 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:59.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:00.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:33:59 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:33:59.894+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:00.465 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:00 smithi177 podman[70685]: 2024-04-24 08:34:00.212658746 +0000 UTC m=+5.825452358 container cleanup 526e4b73ae858c5db7e9c5958123b096208832a6a19e1faeb48ce38ee1c8d9c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:34:00.465 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:00 smithi177 bash[70685]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8 2024-04-24T08:34:00.465 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:00 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:00.207+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:00.466 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:00 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:00.207+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:00.466 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:00 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:00.207+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:00.466 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:00 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:00.207+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:00.466 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:00 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:00.207+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:00.774 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:00 smithi177 podman[70715]: 2024-04-24 08:34:00.464350714 +0000 UTC m=+0.525595129 container remove 526e4b73ae858c5db7e9c5958123b096208832a6a19e1faeb48ce38ee1c8d9c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-24T08:34:00.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:00.497+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:00.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:00.497+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:00.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:00.497+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:00.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:00.497+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:00.993 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:00 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:00.497+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:01.042 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:00 smithi177 podman[70807]: 2024-04-24 08:34:00.773923425 +0000 UTC m=+0.108727525 container create fa81eed5ac3e666e8590837d9063c41dd2ddb0112619099ee07fdc63ad119d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, ceph=True) 2024-04-24T08:34:01.042 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:00 smithi177 podman[70807]: 2024-04-24 08:34:00.683091929 +0000 UTC m=+0.017896031 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:34:01.042 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:00 smithi177 podman[70807]: 2024-04-24 08:34:00.959367681 +0000 UTC m=+0.294171791 container init fa81eed5ac3e666e8590837d9063c41dd2ddb0112619099ee07fdc63ad119d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T08:34:01.042 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:00 smithi177 podman[70807]: 2024-04-24 08:34:00.962621616 +0000 UTC m=+0.297425720 container start fa81eed5ac3e666e8590837d9063c41dd2ddb0112619099ee07fdc63ad119d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:34:01.043 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:01 smithi177 podman[70807]: 2024-04-24 08:34:01.042089464 +0000 UTC m=+0.376893574 container attach fa81eed5ac3e666e8590837d9063c41dd2ddb0112619099ee07fdc63ad119d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T08:34:01.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:00.892+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:01.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:00.892+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:01.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:00.892+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:01.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:00.892+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:01.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:00.892+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:01.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:00.892+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:01.315 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 24 08:34:01 smithi177 podman[70807]: 2024-04-24 08:34:01.258244133 +0000 UTC m=+0.593048235 container died fa81eed5ac3e666e8590837d9063c41dd2ddb0112619099ee07fdc63ad119d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-24T08:34:01.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:01.223+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:01.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:01.223+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:01.315 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:01.223+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:01.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:01.223+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:01.316 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:01 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:01.223+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:01.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:01.482+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:01.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:01.482+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:01.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:01.482+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:01.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:01.482+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:01.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:01 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:01.482+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:01.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.8.service' 2024-04-24T08:34:01.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:01.925 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-24T08:34:01.925 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-24T08:34:01.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.9 2024-04-24T08:34:02.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:01.877+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:02.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:01.877+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:02.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:01.877+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:02.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:01.877+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:02.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:01.877+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:02.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:01 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:01.877+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:02.483 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 systemd[1]: Stopping Ceph osd.9 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:02.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:02.258+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:02.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:02.258+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:02.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:02.258+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:02.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:02.258+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:02.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:02 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:02.258+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:02.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.481+0000 7f5bb1c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:02.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.481+0000 7f5bb1c00700 -1 osd.9 104 *** Got signal Terminated *** 2024-04-24T08:34:02.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.481+0000 7f5bb1c00700 -1 osd.9 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:34:02.744 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.507+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:02.744 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.507+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:02.744 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.507+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:02.744 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.507+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:02.744 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:02 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:02.507+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:03.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:02.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:03.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:02.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:03.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:02.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:03.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:02.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:03.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:02.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:03.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:02 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:02.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:03.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:03 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:03.268+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:03.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:03 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:03.268+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:03.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:03 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:03.268+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:03.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:03 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:03.268+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:03.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:03 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:03.268+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:03.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:03 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:03.464+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:03.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:03 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:03.464+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:03.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:03 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:03.464+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:03.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:03 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:03.464+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:03.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:03 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:03.464+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:04.207 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:03.891+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:04.207 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:03.891+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:04.208 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:03.891+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:04.208 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:03.891+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:04.208 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:03.891+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:04.208 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:03 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:03.891+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:04.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:04.270+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:04.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:04.270+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:04.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:04.270+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:04.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:04.270+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:04.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:04 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:04.270+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:04.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:04.416+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:04.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:04.416+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:04.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:04.416+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:04.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:04.416+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:04.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:04 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:04.416+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:05.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:04.855+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:05.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:04.855+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:05.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:04.855+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:05.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:04.855+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:05.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:04.855+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:05.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:04.855+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:05.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:04.855+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:05.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:05.230+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:05.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:05.230+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:05.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:05.230+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:05.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:05.230+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:05.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:05.230+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:05.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:05 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:05.230+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:05.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:05.410+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:05.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:05.410+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:05.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:05.410+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:05.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:05.410+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:05.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:05 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:05.410+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:06.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:05.900+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:06.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:05.900+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:06.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:05.900+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:06.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:05.900+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:06.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:05.900+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:06.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:05.900+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:06.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:05 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:05.900+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:06.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:06.278+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:06.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:06.278+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:06.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:06.278+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:06.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:06.278+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:06.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:06.278+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:06.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:06.278+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:06.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:06.378+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:06.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:06.378+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:06.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:06.378+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:06.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:06.378+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:06.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:06 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:06.378+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:07.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:06.936+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:07.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:06.936+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:07.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:06.936+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:07.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:06.936+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:07.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:06.936+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:07.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:06.936+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:07.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:06 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:06.936+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:07.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:07.253+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:07.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:07.253+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:07.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:07.253+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:07.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:07.253+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:07.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:07.253+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:07.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:07 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:07.253+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:07.742 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:07.335+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:51.428697+0000 front 2024-04-24T08:32:51.428630+0000 (oldest deadline 2024-04-24T08:33:13.728580+0000) 2024-04-24T08:34:07.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:07.335+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.331549+0000 front 2024-04-24T08:33:01.331665+0000 (oldest deadline 2024-04-24T08:33:26.031255+0000) 2024-04-24T08:34:07.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:07.335+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.333164+0000 front 2024-04-24T08:33:08.333267+0000 (oldest deadline 2024-04-24T08:33:30.033360+0000) 2024-04-24T08:34:07.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:07.335+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:17.636784+0000 front 2024-04-24T08:33:17.636827+0000 (oldest deadline 2024-04-24T08:33:43.536647+0000) 2024-04-24T08:34:07.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:07.335+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.341328+0000 front 2024-04-24T08:33:33.341511+0000 (oldest deadline 2024-04-24T08:33:57.441767+0000) 2024-04-24T08:34:07.743 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:07 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9[62823]: 2024-04-24T08:34:07.335+0000 7f5ba8400700 -1 osd.9 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:39.143880+0000 front 2024-04-24T08:33:39.143730+0000 (oldest deadline 2024-04-24T08:34:06.744944+0000) 2024-04-24T08:34:08.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:07.949+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:08.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:07.949+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:08.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:07.949+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:08.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:07.949+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:08.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:07.949+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:08.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:07.949+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:08.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:07 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:07.949+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:08.289 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:08 smithi146 podman[101516]: 2024-04-24 08:34:08.007852759 +0000 UTC m=+5.655621745 container died 5b753f0fa08c60dcd01e7a491f2533e439f1deb09e6d67af1c708d0af98ed251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424) 2024-04-24T08:34:08.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:08.289+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:08.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:08.289+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:08.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:08.289+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:08.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:08.289+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:08.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:08.289+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:08.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:08 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:08.289+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:08.602 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:08 smithi146 podman[101516]: 2024-04-24 08:34:08.409108558 +0000 UTC m=+6.056877603 container cleanup 5b753f0fa08c60dcd01e7a491f2533e439f1deb09e6d67af1c708d0af98ed251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T08:34:08.603 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:08 smithi146 bash[101516]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9 2024-04-24T08:34:08.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:08 smithi146 podman[101533]: 2024-04-24 08:34:08.601874817 +0000 UTC m=+0.582159065 container remove 5b753f0fa08c60dcd01e7a491f2533e439f1deb09e6d67af1c708d0af98ed251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T08:34:09.460 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:08.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:09.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:08.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:09.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:08.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:09.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:08.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:09.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:08.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:09.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:08.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:09.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:08 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:08.964+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:09.707 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:09 smithi146 podman[101627]: 2024-04-24 08:34:09.339377933 +0000 UTC m=+0.033352797 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:34:09.707 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:09 smithi146 podman[101627]: 2024-04-24 08:34:09.435933277 +0000 UTC m=+0.129908178 container create d50fedda4df566e1c40ed66a2fd497eb384cee54dfa820e287c0143f36da909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:34:09.708 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:09 smithi146 podman[101627]: 2024-04-24 08:34:09.706292292 +0000 UTC m=+0.400267199 container init d50fedda4df566e1c40ed66a2fd497eb384cee54dfa820e287c0143f36da909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:34:09.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:09 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:09.323+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:09.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:09 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:09.323+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:09.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:09 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:09.323+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:09.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:09 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:09.323+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:09.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:09 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:09.323+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:09.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:09 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:09.323+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:09.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:09 smithi146 podman[101627]: 2024-04-24 08:34:09.717936841 +0000 UTC m=+0.411911713 container start d50fedda4df566e1c40ed66a2fd497eb384cee54dfa820e287c0143f36da909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-24T08:34:09.992 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:09 smithi146 podman[101627]: 2024-04-24 08:34:09.796661075 +0000 UTC m=+0.490636016 container attach d50fedda4df566e1c40ed66a2fd497eb384cee54dfa820e287c0143f36da909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:34:10.460 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:09.973+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:10.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:09.973+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:10.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:09.973+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:10.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:09.973+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:10.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:09.973+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:10.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:09.973+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:10.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:09 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:09.973+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:10.492 INFO:journalctl@ceph.osd.9.smithi146.stdout:Apr 24 08:34:10 smithi146 podman[101653]: 2024-04-24 08:34:10.239828516 +0000 UTC m=+0.041628449 container died d50fedda4df566e1c40ed66a2fd497eb384cee54dfa820e287c0143f36da909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-24T08:34:10.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:10 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:10.334+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:10.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:10 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:10.334+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:10.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:10 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:10.334+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:10.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:10 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:10.334+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:10.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:10 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:10.334+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:10.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:10 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:10.334+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:10.875 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.9.service' 2024-04-24T08:34:10.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:10.961 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-24T08:34:10.961 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-24T08:34:10.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.10 2024-04-24T08:34:11.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:10 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:10.947+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:11.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:10 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:10.947+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:11.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:10 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:10.947+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:11.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:10 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:10.947+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:11.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:10 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:10.947+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:11.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:10 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:10.947+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:11.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:10 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:10.947+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:11.212 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 systemd[1]: Stopping Ceph osd.10 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:11.710 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.298+0000 7ff64a3af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:11.710 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.298+0000 7ff64a3af700 -1 osd.10 104 *** Got signal Terminated *** 2024-04-24T08:34:11.711 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.298+0000 7ff64a3af700 -1 osd.10 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:34:11.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:11.340+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:11.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:11.340+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:11.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:11.340+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:11.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:11.340+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:11.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:11.340+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:11.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:11.340+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:12.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:12.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:12.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:12.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:12.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:12.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:12.462 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:11 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:11.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:12.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:12 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:12.311+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:12.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:12 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:12.311+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:12.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:12 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:12.311+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:12.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:12 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:12.311+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:12.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:12 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:12.311+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:12.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:12 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:12.311+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:13.460 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.026+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:13.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.026+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:13.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.026+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:13.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.026+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:13.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.026+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:13.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.026+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:13.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.026+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:13.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:13.345+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:13.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:13.345+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:13.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:13.345+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:13.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:13.345+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:13.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:13.345+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:13.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:13.345+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:13.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:13 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:13.345+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:14.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:14.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:14.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:14.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:14.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:14.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:14.461 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:13 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:13.996+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:14.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:14.389+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:14.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:14.389+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:14.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:14.389+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:14.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:14.389+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:14.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:14.389+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:14.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:14.389+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:14.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:14 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:14.389+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:15.362 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:14.962+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:15.362 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:14.962+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:15.362 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:14.962+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:15.363 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:14.962+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:15.363 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:14.962+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:15.363 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:14.962+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:15.363 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:14 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:14.962+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:15.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:15.347+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:15.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:15.347+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:15.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:15.347+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:15.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:15.347+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:15.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:15.347+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:15.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:15.347+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:15.816 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:15 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:15.347+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:16.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:49.246791+0000 front 2024-04-24T08:32:49.246834+0000 (oldest deadline 2024-04-24T08:33:14.546628+0000) 2024-04-24T08:34:16.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:32:59.847960+0000 front 2024-04-24T08:32:59.847929+0000 (oldest deadline 2024-04-24T08:33:22.648731+0000) 2024-04-24T08:34:16.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:08.251220+0000 front 2024-04-24T08:33:08.251172+0000 (oldest deadline 2024-04-24T08:33:31.151269+0000) 2024-04-24T08:34:16.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6814 osd.3 since back 2024-04-24T08:33:16.253307+0000 front 2024-04-24T08:33:16.253208+0000 (oldest deadline 2024-04-24T08:33:40.352942+0000) 2024-04-24T08:34:16.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:23.253885+0000 front 2024-04-24T08:33:23.253931+0000 (oldest deadline 2024-04-24T08:33:47.353961+0000) 2024-04-24T08:34:16.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.656065+0000 front 2024-04-24T08:33:33.656076+0000 (oldest deadline 2024-04-24T08:33:57.755750+0000) 2024-04-24T08:34:16.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:15 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10[57429]: 2024-04-24T08:34:15.918+0000 7ff6469a3700 -1 osd.10 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:41.856878+0000 front 2024-04-24T08:33:41.857025+0000 (oldest deadline 2024-04-24T08:34:04.156489+0000) 2024-04-24T08:34:16.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:16.307+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:16.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:16.307+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:16.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:16.307+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:16.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:16.307+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:16.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:16.307+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:16.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:16.307+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:16.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:16.307+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:16.927 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:16 smithi153 podman[69107]: 2024-04-24 08:34:16.657182096 +0000 UTC m=+5.499726736 container died 8ee83a9a3b760f991c3a3069f0b424005b4dedce06fff7bd8e05c955f6b6c64f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-24T08:34:17.210 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 podman[69107]: 2024-04-24 08:34:17.026348165 +0000 UTC m=+5.868892801 container cleanup 8ee83a9a3b760f991c3a3069f0b424005b4dedce06fff7bd8e05c955f6b6c64f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-24T08:34:17.211 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 bash[69107]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10 2024-04-24T08:34:17.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:17.272+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:17.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:17.272+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:17.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:17.272+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:17.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:17.272+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:17.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:17.272+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:17.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:17.272+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:17.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:17 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:17.272+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:17.619 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 podman[69126]: 2024-04-24 08:34:17.302444055 +0000 UTC m=+0.645589661 container remove 8ee83a9a3b760f991c3a3069f0b424005b4dedce06fff7bd8e05c955f6b6c64f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-24T08:34:17.897 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 podman[69221]: 2024-04-24 08:34:17.537579225 +0000 UTC m=+0.018429397 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:34:17.898 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 podman[69221]: 2024-04-24 08:34:17.653314682 +0000 UTC m=+0.134164827 container create c402367ed6e97f9d066799759e131687d7674c5d1b4c56e041f4edcbd9b94077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T08:34:17.898 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 podman[69221]: 2024-04-24 08:34:17.897556267 +0000 UTC m=+0.378406412 container init c402367ed6e97f9d066799759e131687d7674c5d1b4c56e041f4edcbd9b94077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-24T08:34:18.192 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 podman[69221]: 2024-04-24 08:34:17.900679603 +0000 UTC m=+0.381529749 container start c402367ed6e97f9d066799759e131687d7674c5d1b4c56e041f4edcbd9b94077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T08:34:18.192 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:17 smithi153 podman[69221]: 2024-04-24 08:34:17.972592621 +0000 UTC m=+0.453442783 container attach c402367ed6e97f9d066799759e131687d7674c5d1b4c56e041f4edcbd9b94077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-24T08:34:18.442 INFO:journalctl@ceph.osd.10.smithi153.stdout:Apr 24 08:34:18 smithi153 podman[69221]: 2024-04-24 08:34:18.191890433 +0000 UTC m=+0.672740596 container died c402367ed6e97f9d066799759e131687d7674c5d1b4c56e041f4edcbd9b94077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-24T08:34:18.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:18.283+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:18.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:18.283+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:18.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:18.283+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:18.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:18.283+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:18.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:18.283+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:18.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:18.283+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:18.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:18.283+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:18.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.10.service' 2024-04-24T08:34:18.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:18.784 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-24T08:34:18.784 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-24T08:34:18.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.11 2024-04-24T08:34:19.065 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:18 smithi177 systemd[1]: Stopping Ceph osd.11 for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:19.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.070+0000 7f91495ae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:19.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.070+0000 7f91495ae700 -1 osd.11 104 *** Got signal Terminated *** 2024-04-24T08:34:19.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.070+0000 7f91495ae700 -1 osd.11 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:34:19.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.299+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:19.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.299+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:19.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.299+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:19.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.299+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:19.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.299+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:19.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.299+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:19.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:19 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:19.299+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:20.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:20.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:20.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:20.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:20.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:20.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:20.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:20.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:20.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:20.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:20.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:20.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:20.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:20.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:21.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:21.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:21.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:21.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:21.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:21.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:21.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:21.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:21.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:21.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:21.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:21.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:21.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:21 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:21.259+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:22.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:22.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:22.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:22.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:22.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:22.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:22.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:22.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:22 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:22.245+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6820 osd.8 since back 2024-04-24T08:33:55.398218+0000 front 2024-04-24T08:33:55.398258+0000 (oldest deadline 2024-04-24T08:34:21.298336+0000) 2024-04-24T08:34:23.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6806 osd.0 since back 2024-04-24T08:32:50.890945+0000 front 2024-04-24T08:32:50.890934+0000 (oldest deadline 2024-04-24T08:33:16.190714+0000) 2024-04-24T08:34:23.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6804 osd.1 since back 2024-04-24T08:33:01.492075+0000 front 2024-04-24T08:33:01.491928+0000 (oldest deadline 2024-04-24T08:33:25.591916+0000) 2024-04-24T08:34:23.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-24T08:33:06.692787+0000 front 2024-04-24T08:33:06.692724+0000 (oldest deadline 2024-04-24T08:33:31.992893+0000) 2024-04-24T08:34:23.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6812 osd.4 since back 2024-04-24T08:33:25.495219+0000 front 2024-04-24T08:33:25.495288+0000 (oldest deadline 2024-04-24T08:33:50.195111+0000) 2024-04-24T08:34:23.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-24T08:33:33.695912+0000 front 2024-04-24T08:33:33.695961+0000 (oldest deadline 2024-04-24T08:33:58.996069+0000) 2024-04-24T08:34:23.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.146:6822 osd.6 since back 2024-04-24T08:33:38.996784+0000 front 2024-04-24T08:33:38.996901+0000 (oldest deadline 2024-04-24T08:34:04.896658+0000) 2024-04-24T08:34:23.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.153:6820 osd.7 since back 2024-04-24T08:33:46.597666+0000 front 2024-04-24T08:33:46.597641+0000 (oldest deadline 2024-04-24T08:34:12.497495+0000) 2024-04-24T08:34:23.566 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:23 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11[58711]: 2024-04-24T08:34:23.290+0000 7f9145ba2700 -1 osd.11 104 heartbeat_check: no reply from 172.21.15.177:6820 osd.8 since back 2024-04-24T08:33:55.398218+0000 front 2024-04-24T08:33:55.398258+0000 (oldest deadline 2024-04-24T08:34:21.298336+0000) 2024-04-24T08:34:24.815 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:24 smithi177 podman[70914]: 2024-04-24 08:34:24.5556268 +0000 UTC m=+5.578531332 container died 2c387404e5da68f30504d74640d95b848c31fd4f94d76ca26099e651bef8c69d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T08:34:25.189 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:24 smithi177 podman[70914]: 2024-04-24 08:34:24.913264291 +0000 UTC m=+5.936168813 container cleanup 2c387404e5da68f30504d74640d95b848c31fd4f94d76ca26099e651bef8c69d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-24T08:34:25.190 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:24 smithi177 bash[70914]: ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11 2024-04-24T08:34:25.506 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:25 smithi177 podman[70942]: 2024-04-24 08:34:25.189268229 +0000 UTC m=+0.630741060 container remove 2c387404e5da68f30504d74640d95b848c31fd4f94d76ca26099e651bef8c69d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-24T08:34:25.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:25 smithi177 podman[71030]: 2024-04-24 08:34:25.506676456 +0000 UTC m=+0.105065793 container create fae19fcb175585fdd9956ecfd09ee14402659b7c54fff84956a540dbe21d7fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:34:25.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:25 smithi177 podman[71030]: 2024-04-24 08:34:25.419322586 +0000 UTC m=+0.017711941 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:34:25.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:25 smithi177 podman[71030]: 2024-04-24 08:34:25.692201552 +0000 UTC m=+0.290590889 container init fae19fcb175585fdd9956ecfd09ee14402659b7c54fff84956a540dbe21d7fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-24T08:34:25.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:25 smithi177 podman[71030]: 2024-04-24 08:34:25.695240075 +0000 UTC m=+0.293629415 container start fae19fcb175585fdd9956ecfd09ee14402659b7c54fff84956a540dbe21d7fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-24T08:34:25.776 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:25 smithi177 podman[71030]: 2024-04-24 08:34:25.775432347 +0000 UTC m=+0.373821712 container attach fae19fcb175585fdd9956ecfd09ee14402659b7c54fff84956a540dbe21d7fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-24T08:34:26.065 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 24 08:34:25 smithi177 podman[71030]: 2024-04-24 08:34:25.99035907 +0000 UTC m=+0.588748407 container died fae19fcb175585fdd9956ecfd09ee14402659b7c54fff84956a540dbe21d7fa7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T08:34:26.610 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@osd.11.service' 2024-04-24T08:34:26.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:26.649 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-24T08:34:26.649 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-24T08:34:26.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.a 2024-04-24T08:34:26.992 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:34:26 smithi146 systemd[1]: Stopping Ceph mds.a for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:27.483 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:34:27 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a[69454]: 2024-04-24T08:34:27.216+0000 7fbb60400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:27.484 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:34:27 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a[69454]: 2024-04-24T08:34:27.216+0000 7fbb60400700 -1 mds.a *** got signal Terminated *** 2024-04-24T08:34:31.431 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:34:31 smithi146 podman[101745]: 2024-04-24 08:34:31.136747275 +0000 UTC m=+4.055883625 container died 16a5089a1089ab04ba94be053848bae524222cd0fc47201a65e0bb9fc27e5cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:34:31.742 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:34:31 smithi146 podman[101745]: 2024-04-24 08:34:31.542278452 +0000 UTC m=+4.461414769 container cleanup 16a5089a1089ab04ba94be053848bae524222cd0fc47201a65e0bb9fc27e5cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-24T08:34:31.743 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:34:31 smithi146 bash[101745]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a 2024-04-24T08:34:32.205 INFO:journalctl@ceph.mds.a.smithi146.stdout:Apr 24 08:34:31 smithi146 podman[101770]: 2024-04-24 08:34:31.776826276 +0000 UTC m=+0.625531981 container remove 16a5089a1089ab04ba94be053848bae524222cd0fc47201a65e0bb9fc27e5cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-a, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-24T08:34:32.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.a.service' 2024-04-24T08:34:32.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:32.348 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-24T08:34:32.348 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-24T08:34:32.348 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.d 2024-04-24T08:34:32.929 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:32 smithi146 systemd[1]: Stopping Ceph mds.d for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:33.242 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:32 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d[68466]: 2024-04-24T08:34:32.927+0000 7f7b98e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:33.242 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:32 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d[68466]: 2024-04-24T08:34:32.927+0000 7f7b98e00700 -1 mds.d *** got signal Terminated *** 2024-04-24T08:34:37.226 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:36 smithi146 podman[101877]: 2024-04-24 08:34:36.94929242 +0000 UTC m=+4.143628805 container died 6d99d3073189ad03a5d3d02daed44eb48f2bf547a7eaa3a8abaa54d4d48e5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-24T08:34:37.492 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:37 smithi146 podman[101877]: 2024-04-24 08:34:37.395613753 +0000 UTC m=+4.589950190 container cleanup 6d99d3073189ad03a5d3d02daed44eb48f2bf547a7eaa3a8abaa54d4d48e5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:34:37.492 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:37 smithi146 bash[101877]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d 2024-04-24T08:34:37.862 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:37 smithi146 podman[101891]: 2024-04-24 08:34:37.596636975 +0000 UTC m=+0.632482192 container remove 6d99d3073189ad03a5d3d02daed44eb48f2bf547a7eaa3a8abaa54d4d48e5388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-24T08:34:38.069 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.d.service' 2024-04-24T08:34:38.142 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:38 smithi146 systemd[1]: ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.d.service: Deactivated successfully. 2024-04-24T08:34:38.142 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:38 smithi146 systemd[1]: Stopped Ceph mds.d for cca14746-0211-11ef-bc93-c7b262605968. 2024-04-24T08:34:38.143 INFO:journalctl@ceph.mds.d.smithi146.stdout:Apr 24 08:34:38 smithi146 systemd[1]: ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.d.service: Consumed 2.494s CPU time. 2024-04-24T08:34:38.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:38.174 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-24T08:34:38.174 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-24T08:34:38.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.g 2024-04-24T08:34:38.492 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:34:38 smithi146 systemd[1]: Stopping Ceph mds.g for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:38.992 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:34:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g[69067]: 2024-04-24T08:34:38.755+0000 7f4e60c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:38.992 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:34:38 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g[69067]: 2024-04-24T08:34:38.755+0000 7f4e60c00700 -1 mds.g *** got signal Terminated *** 2024-04-24T08:34:43.093 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:34:42 smithi146 podman[101997]: 2024-04-24 08:34:42.771504251 +0000 UTC m=+4.139197418 container died ad5bb208c020df655189a9aaa300c247c388ce7931b7b129e67d646d0dbc074b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:34:43.459 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:34:43 smithi146 podman[101997]: 2024-04-24 08:34:43.190886302 +0000 UTC m=+4.558579486 container cleanup ad5bb208c020df655189a9aaa300c247c388ce7931b7b129e67d646d0dbc074b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-24T08:34:43.459 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:34:43 smithi146 bash[101997]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g 2024-04-24T08:34:43.742 INFO:journalctl@ceph.mds.g.smithi146.stdout:Apr 24 08:34:43 smithi146 podman[102011]: 2024-04-24 08:34:43.458441745 +0000 UTC m=+0.669942312 container remove ad5bb208c020df655189a9aaa300c247c388ce7931b7b129e67d646d0dbc074b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-24T08:34:43.941 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.g.service' 2024-04-24T08:34:44.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:44.047 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-24T08:34:44.047 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-24T08:34:44.047 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.j 2024-04-24T08:34:44.492 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:34:44 smithi146 systemd[1]: Stopping Ceph mds.j for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:44.992 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:34:44 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j[68112]: 2024-04-24T08:34:44.582+0000 7f3006000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:44.992 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:34:44 smithi146 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j[68112]: 2024-04-24T08:34:44.582+0000 7f3006000700 -1 mds.j *** got signal Terminated *** 2024-04-24T08:34:48.891 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:34:48 smithi146 podman[102117]: 2024-04-24 08:34:48.609948128 +0000 UTC m=+4.134726084 container died 2e91614dea602b93596bc2ef1d4cdb67367a59368b1c75266765f703883c1563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:34:49.242 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:34:49 smithi146 podman[102117]: 2024-04-24 08:34:49.077091339 +0000 UTC m=+4.601869272 container cleanup 2e91614dea602b93596bc2ef1d4cdb67367a59368b1c75266765f703883c1563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:34:49.243 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:34:49 smithi146 bash[102117]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j 2024-04-24T08:34:49.723 INFO:journalctl@ceph.mds.j.smithi146.stdout:Apr 24 08:34:49 smithi146 podman[102142]: 2024-04-24 08:34:49.266888856 +0000 UTC m=+0.644391476 container remove 2e91614dea602b93596bc2ef1d4cdb67367a59368b1c75266765f703883c1563 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-j, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:34:49.752 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.j.service' 2024-04-24T08:34:49.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:49.859 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-24T08:34:49.859 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-24T08:34:49.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.b 2024-04-24T08:34:50.210 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:34:49 smithi153 systemd[1]: Stopping Ceph mds.b for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:50.475 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:34:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: 2024-04-24T08:34:50.291+0000 7f9d664c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:50.475 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:34:50 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b[62792]: 2024-04-24T08:34:50.291+0000 7f9d664c7700 -1 mds.b *** got signal Terminated *** 2024-04-24T08:34:54.617 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:34:54 smithi153 podman[69330]: 2024-04-24 08:34:54.300500686 +0000 UTC m=+4.253068186 container died e1abbd03208fb96e7f27c06ff5e6aa87391fc59e298e4004373028fd8274cfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T08:34:54.934 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:34:54 smithi153 podman[69330]: 2024-04-24 08:34:54.691860777 +0000 UTC m=+4.644428275 container cleanup e1abbd03208fb96e7f27c06ff5e6aa87391fc59e298e4004373028fd8274cfe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-24T08:34:54.935 INFO:journalctl@ceph.mds.b.smithi153.stdout:Apr 24 08:34:54 smithi153 bash[69330]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-b 2024-04-24T08:34:55.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.b.service' 2024-04-24T08:34:55.100 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:34:55.101 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-24T08:34:55.101 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-24T08:34:55.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.e 2024-04-24T08:34:55.210 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:34:55 smithi153 systemd[1]: Stopping Ceph mds.e for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:34:55.710 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:34:55 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e[62164]: 2024-04-24T08:34:55.420+0000 7f50493cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:34:55.710 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:34:55 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e[62164]: 2024-04-24T08:34:55.420+0000 7f50493cd700 -1 mds.e *** got signal Terminated *** 2024-04-24T08:34:59.710 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:34:59 smithi153 podman[69467]: 2024-04-24 08:34:59.427055568 +0000 UTC m=+4.107314206 container died aee38ccaeee3f5b18b8aba7d7630ba2026487ca9c13204030902ee4f3c5deaec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T08:35:00.063 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:34:59 smithi153 podman[69467]: 2024-04-24 08:34:59.79515847 +0000 UTC m=+4.475417096 container cleanup aee38ccaeee3f5b18b8aba7d7630ba2026487ca9c13204030902ee4f3c5deaec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-24T08:35:00.063 INFO:journalctl@ceph.mds.e.smithi153.stdout:Apr 24 08:34:59 smithi153 bash[69467]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-e 2024-04-24T08:35:00.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.e.service' 2024-04-24T08:35:00.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:35:00.248 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-24T08:35:00.248 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-24T08:35:00.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.h 2024-04-24T08:35:00.710 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:35:00 smithi153 systemd[1]: Stopping Ceph mds.h for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:35:00.710 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:35:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h[62476]: 2024-04-24T08:35:00.565+0000 7f20c544e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:35:00.710 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:35:00 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h[62476]: 2024-04-24T08:35:00.565+0000 7f20c544e700 -1 mds.h *** got signal Terminated *** 2024-04-24T08:35:03.383 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:35:03 smithi153 podman[69587]: 2024-04-24 08:35:03.094121822 +0000 UTC m=+2.635813772 container died 8bacd1b36dfece32ec1194df24dee7e03ded91a0885aedc62fe35b99e6e32c84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-24T08:35:03.710 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:35:03 smithi153 podman[69587]: 2024-04-24 08:35:03.465704333 +0000 UTC m=+3.007396271 container cleanup 8bacd1b36dfece32ec1194df24dee7e03ded91a0885aedc62fe35b99e6e32c84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0) 2024-04-24T08:35:03.711 INFO:journalctl@ceph.mds.h.smithi153.stdout:Apr 24 08:35:03 smithi153 bash[69587]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-h 2024-04-24T08:35:03.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.h.service' 2024-04-24T08:35:03.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:35:03.894 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-24T08:35:03.895 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-24T08:35:03.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.k 2024-04-24T08:35:04.210 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:35:03 smithi153 systemd[1]: Stopping Ceph mds.k for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:35:04.710 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:35:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k[61845]: 2024-04-24T08:35:04.210+0000 7fa7555af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:35:04.710 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:35:04 smithi153 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k[61845]: 2024-04-24T08:35:04.210+0000 7fa7555af700 -1 mds.k *** got signal Terminated *** 2024-04-24T08:35:05.904 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:35:05 smithi153 podman[69706]: 2024-04-24 08:35:05.624429908 +0000 UTC m=+1.516608642 container died 5015291c49b3be6f6d3c7992293538d22b7881e223f38786c3952f83b9dc4b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T08:35:06.210 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:35:05 smithi153 podman[69706]: 2024-04-24 08:35:05.969607981 +0000 UTC m=+1.861786721 container cleanup 5015291c49b3be6f6d3c7992293538d22b7881e223f38786c3952f83b9dc4b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:35:06.210 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:35:05 smithi153 bash[69706]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k 2024-04-24T08:35:06.211 INFO:journalctl@ceph.mds.k.smithi153.stdout:Apr 24 08:35:06 smithi153 podman[69720]: 2024-04-24 08:35:06.137248926 +0000 UTC m=+0.507609098 container remove 5015291c49b3be6f6d3c7992293538d22b7881e223f38786c3952f83b9dc4b53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-24T08:35:06.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.k.service' 2024-04-24T08:35:06.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:35:06.422 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-24T08:35:06.422 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-24T08:35:06.423 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.c 2024-04-24T08:35:06.703 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:35:06 smithi177 systemd[1]: Stopping Ceph mds.c for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:35:07.022 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:35:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c[64134]: 2024-04-24T08:35:06.702+0000 7f786dd09700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:35:07.022 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:35:06 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c[64134]: 2024-04-24T08:35:06.702+0000 7f786dd09700 -1 mds.c *** got signal Terminated *** 2024-04-24T08:35:10.995 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:35:10 smithi177 podman[71135]: 2024-04-24 08:35:10.712124783 +0000 UTC m=+4.098526130 container died 6f268dd544ed84372ded2e8d685f1ba2795d507af43af90785ab09bfb388a051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-24T08:35:11.315 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:35:11 smithi177 podman[71135]: 2024-04-24 08:35:11.077282754 +0000 UTC m=+4.463684100 container cleanup 6f268dd544ed84372ded2e8d685f1ba2795d507af43af90785ab09bfb388a051 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:35:11.315 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 24 08:35:11 smithi177 bash[71135]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-c 2024-04-24T08:35:11.518 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.c.service' 2024-04-24T08:35:11.553 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:35:11.553 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-24T08:35:11.553 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-24T08:35:11.553 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.f 2024-04-24T08:35:11.873 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:35:11 smithi177 systemd[1]: Stopping Ceph mds.f for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:35:12.314 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:35:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f[63829]: 2024-04-24T08:35:11.872+0000 7fdfdb272700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:35:12.315 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:35:11 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f[63829]: 2024-04-24T08:35:11.872+0000 7fdfdb272700 -1 mds.f *** got signal Terminated *** 2024-04-24T08:35:15.314 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:35:15 smithi177 podman[71267]: 2024-04-24 08:35:15.058124358 +0000 UTC m=+3.291112098 container died 5c73c36cb987e2f6f56a477f577c139c3eff894c39ca727d0fd0232d89a77fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-24T08:35:15.765 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:35:15 smithi177 podman[71267]: 2024-04-24 08:35:15.506245198 +0000 UTC m=+3.739232946 container cleanup 5c73c36cb987e2f6f56a477f577c139c3eff894c39ca727d0fd0232d89a77fc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:35:15.766 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 24 08:35:15 smithi177 bash[71267]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-f 2024-04-24T08:35:15.932 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.f.service' 2024-04-24T08:35:15.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:35:15.963 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-24T08:35:15.963 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-24T08:35:15.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.i 2024-04-24T08:35:16.314 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:35:16 smithi177 systemd[1]: Stopping Ceph mds.i for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:35:16.315 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:35:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i[64442]: 2024-04-24T08:35:16.267+0000 7fe219254700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:35:16.315 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:35:16 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i[64442]: 2024-04-24T08:35:16.267+0000 7fe219254700 -1 mds.i *** got signal Terminated *** 2024-04-24T08:35:19.495 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:35:19 smithi177 podman[71387]: 2024-04-24 08:35:19.090235627 +0000 UTC m=+2.914146055 container died bdc88fcb6b872863c250b67ed4b02ef387ee0b230a8b565146f987c1ca45f3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-24T08:35:19.815 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:35:19 smithi177 podman[71387]: 2024-04-24 08:35:19.576403035 +0000 UTC m=+3.400313468 container cleanup bdc88fcb6b872863c250b67ed4b02ef387ee0b230a8b565146f987c1ca45f3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-24T08:35:19.815 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 24 08:35:19 smithi177 bash[71387]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-i 2024-04-24T08:35:20.008 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.i.service' 2024-04-24T08:35:20.040 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:35:20.040 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-24T08:35:20.041 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-24T08:35:20.041 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.l 2024-04-24T08:35:20.372 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:35:20 smithi177 systemd[1]: Stopping Ceph mds.l for cca14746-0211-11ef-bc93-c7b262605968... 2024-04-24T08:35:20.815 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:35:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l[63523]: 2024-04-24T08:35:20.371+0000 7fcd096ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:35:20.815 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:35:20 smithi177 ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l[63523]: 2024-04-24T08:35:20.371+0000 7fcd096ef700 -1 mds.l *** got signal Terminated *** 2024-04-24T08:35:23.031 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:35:22 smithi177 podman[71507]: 2024-04-24 08:35:22.767611231 +0000 UTC m=+2.506743741 container died 8f4f881ab3472989e283d9df472de3d9130aa320666aec59e0a52d196c8d1107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-24T08:35:23.301 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:35:23 smithi177 podman[71507]: 2024-04-24 08:35:23.138384071 +0000 UTC m=+2.877516569 container cleanup 8f4f881ab3472989e283d9df472de3d9130aa320666aec59e0a52d196c8d1107 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-04-24T08:35:23.301 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 24 08:35:23 smithi177 bash[71507]: ceph-cca14746-0211-11ef-bc93-c7b262605968-mds-l 2024-04-24T08:35:23.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cca14746-0211-11ef-bc93-c7b262605968@mds.l.service' 2024-04-24T08:35:23.524 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:35:23.524 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-24T08:35:23.524 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cca14746-0211-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T08:35:23.851 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:35:28.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cca14746-0211-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T08:35:29.109 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:35:31.527 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cca14746-0211-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T08:35:31.715 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:35:34.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:35:34.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:35:34.179 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:35:34.205 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-24T08:35:34.208 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi146/crash 2024-04-24T08:35:34.209 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash -- . 2024-04-24T08:35:34.306 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T08:35:34.307 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now 2024-04-24T08:35:34.311 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi153/crash 2024-04-24T08:35:34.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash -- . 2024-04-24T08:35:34.356 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T08:35:34.357 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-04-24T08:35:34.359 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi177/crash 2024-04-24T08:35:34.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash -- . 2024-04-24T08:35:34.429 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/cca14746-0211-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T08:35:34.430 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-04-24T08:35:34.431 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-24T08:35:34.431 DEBUG:teuthology.orchestra.run.smithi146:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-24T08:35:34.532 INFO:tasks.cephadm:Compressing logs... 2024-04-24T08:35:34.533 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:35:34.537 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:35:34.540 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:35:34.563 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T08:35:34.563 INFO:teuthology.orchestra.run.smithi177.stderr:: No such file or directory 2024-04-24T08:35:34.563 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log 2024-04-24T08:35:34.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-24T08:35:34.564 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-24T08:35:34.565 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T08:35:34.565 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-24T08:35:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-04-24T08:35:34.573 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-24T08:35:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log 2024-04-24T08:35:34.573 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T08:35:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-24T08:35:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-24T08:35:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-24T08:35:34.574 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T08:35:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-24T08:35:34.574 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-24T08:35:34.575 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-24T08:35:34.575 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-24T08:35:34.575 INFO:teuthology.orchestra.run.smithi153.stderr: 87.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T08:35:34.575 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.4% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T08:35:34.576 INFO:teuthology.orchestra.run.smithi153.stderr: 91.9% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-24T08:35:34.578 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose 92.4% -- -- replaced with /var/log/ceph/cephadm.log.gz /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.4% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T08:35:34.579 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-24T08:35:34.580 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T08:35:34.580 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-24T08:35:34.580 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.11.log: 94.5% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-24T08:35:34.580 INFO:teuthology.orchestra.run.smithi177.stderr: 92.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-24T08:35:34.580 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T08:35:34.580 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-24T08:35:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T08:35:34.584 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-24T08:35:34.584 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T08:35:34.584 INFO:teuthology.orchestra.run.smithi153.stderr: 83.7% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T08:35:34.588 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.e.log: 94.3% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-24T08:35:34.589 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-24T08:35:34.589 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log: --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-24T08:35:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-24T08:35:34.593 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T08:35:34.594 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log 2024-04-24T08:35:34.595 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T08:35:34.596 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-24T08:35:34.597 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T08:35:34.597 INFO:teuthology.orchestra.run.smithi177.stderr: 94.5% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-24T08:35:34.598 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-24T08:35:34.599 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.g.log: 87.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T08:35:34.599 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-24T08:35:34.600 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-24T08:35:34.601 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.3.log: 90.3% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T08:35:34.601 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-24T08:35:34.602 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-24T08:35:34.603 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T08:35:34.604 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.0.log: /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.x.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T08:35:34.604 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-24T08:35:34.606 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T08:35:34.606 INFO:teuthology.orchestra.run.smithi146.stderr: 94.3% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-24T08:35:34.607 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-24T08:35:34.608 INFO:teuthology.orchestra.run.smithi153.stderr: 94.4% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-24T08:35:34.609 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T08:35:34.609 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T08:35:34.612 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-24T08:35:34.618 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-24T08:35:34.631 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.3% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-24T08:35:34.639 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T08:35:34.640 INFO:teuthology.orchestra.run.smithi153.stderr: 94.5% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-24T08:35:34.688 INFO:teuthology.orchestra.run.smithi146.stderr: 92.7% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-24T08:35:35.014 INFO:teuthology.orchestra.run.smithi146.stderr: 90.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-24T08:35:35.071 INFO:teuthology.orchestra.run.smithi177.stderr: 95.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-24T08:35:35.111 INFO:teuthology.orchestra.run.smithi153.stderr: 91.8% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-24T08:35:35.147 INFO:teuthology.orchestra.run.smithi177.stderr: 91.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-24T08:35:35.393 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-24T08:35:35.588 INFO:teuthology.orchestra.run.smithi177.stderr: 95.4% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-24T08:35:35.596 INFO:teuthology.orchestra.run.smithi177.stderr: 93.8% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-24T08:35:35.648 INFO:teuthology.orchestra.run.smithi146.stderr: 93.5% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-24T08:35:35.819 INFO:teuthology.orchestra.run.smithi146.stderr: 93.7% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-24T08:35:35.894 INFO:teuthology.orchestra.run.smithi146.stderr: 90.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-24T08:35:35.913 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-24T08:35:35.969 INFO:teuthology.orchestra.run.smithi177.stderr: 93.7% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-24T08:35:36.013 INFO:teuthology.orchestra.run.smithi177.stderr: 94.1% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-24T08:35:36.056 INFO:teuthology.orchestra.run.smithi153.stderr: 94.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-24T08:35:36.058 INFO:teuthology.orchestra.run.smithi177.stderr: 93.7% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-24T08:35:36.060 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-24T08:35:36.060 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m1.506s 2024-04-24T08:35:36.060 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m7.277s 2024-04-24T08:35:36.060 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.250s 2024-04-24T08:35:36.246 INFO:teuthology.orchestra.run.smithi146.stderr: 93.9% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-24T08:35:36.329 INFO:teuthology.orchestra.run.smithi153.stderr: 94.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-24T08:35:36.432 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-24T08:35:40.124 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-24T08:35:40.128 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-04-24T08:35:40.128 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m5.558s 2024-04-24T08:35:40.128 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m12.063s 2024-04-24T08:35:40.128 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m1.061s 2024-04-24T08:35:46.958 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/cca14746-0211-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-24T08:35:46.960 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-24T08:35:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m12.406s 2024-04-24T08:35:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m17.811s 2024-04-24T08:35:46.961 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.613s 2024-04-24T08:35:46.961 INFO:tasks.cephadm:Archiving logs... 2024-04-24T08:35:46.961 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi146/log 2024-04-24T08:35:46.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:35:48.606 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi153/log 2024-04-24T08:35:48.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:35:51.181 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi177/log 2024-04-24T08:35:51.182 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:35:52.116 INFO:tasks.cephadm:Removing cluster... 2024-04-24T08:35:52.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cca14746-0211-11ef-bc93-c7b262605968 --force 2024-04-24T08:35:52.438 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:35:53.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cca14746-0211-11ef-bc93-c7b262605968 --force 2024-04-24T08:35:53.278 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:35:53.795 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cca14746-0211-11ef-bc93-c7b262605968 --force 2024-04-24T08:35:53.986 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: cca14746-0211-11ef-bc93-c7b262605968 2024-04-24T08:35:54.248 INFO:tasks.cephadm:Removing cephadm ... 2024-04-24T08:35:54.249 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T08:35:54.305 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T08:35:54.322 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T08:35:54.338 INFO:tasks.cephadm:Teardown complete 2024-04-24T08:35:54.338 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T08:35:54.352 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T08:35:54.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:35:54.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:35:54.365 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:35:54.520 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T08:35:54.532 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T08:35:54.533 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:35:54.541 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:35:54.543 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:35:54.555 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:35:54.557 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:35:54.560 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:35:54.560 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-24T08:35:54.560 INFO:teuthology.orchestra.run.smithi153.stdout:^- hv01.front.sepia.ceph.com 2 7 377 104 +1253us[+1240us] +/- 588ms 2024-04-24T08:35:54.560 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -312us[ -328us] +/- 56ms 2024-04-24T08:35:54.560 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +361us[ +345us] +/- 68ms 2024-04-24T08:35:54.560 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:35:54.562 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:35:54.562 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-24T08:35:54.563 INFO:teuthology.orchestra.run.smithi177.stdout:^- hv01.front.sepia.ceph.com 2 6 377 40 +1180us[+1180us] +/- 589ms 2024-04-24T08:35:54.563 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -300us[ -311us] +/- 56ms 2024-04-24T08:35:54.563 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 43 +357us[ +345us] +/- 68ms 2024-04-24T08:35:54.563 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:35:54.587 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:35:54.597 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:35:54.597 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-04-24T08:35:54.597 INFO:teuthology.orchestra.run.smithi146.stdout:^- hv01.front.sepia.ceph.com 2 6 377 40 +1225us[+1225us] +/- 589ms 2024-04-24T08:35:54.597 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -301us[ -309us] +/- 56ms 2024-04-24T08:35:54.597 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +362us[ +362us] +/- 68ms 2024-04-24T08:35:54.598 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:35:54.599 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T08:35:54.612 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T08:35:54.613 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T08:35:54.668 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:35:54.859 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:35:54.926 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:35:54.993 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:35:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:35:55.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:35:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:35:55.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:35:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:35:55.140 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T08:35:55.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T08:35:55.165 INFO:teuthology.task.internal:Duration was 2099.828240 seconds 2024-04-24T08:35:55.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T08:35:55.177 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T08:35:55.177 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:35:55.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:35:55.187 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:35:55.222 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:35:55.223 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:35:55.324 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:35:55.603 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T08:35:55.603 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-04-24T08:35:55.604 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:35:55.678 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-04-24T08:35:55.678 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:35:55.700 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-04-24T08:35:55.700 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:35:55.726 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T08:35:55.727 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:35:55.734 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:35:55.742 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:35:55.818 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T08:35:55.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:35:55.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:35:55.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:35:56.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T08:35:56.349 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T08:35:56.349 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:35:56.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:35:56.452 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:35:56.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T08:35:56.488 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:35:56.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:35:56.498 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:35:56.517 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-04-24T08:35:56.539 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-04-24T08:35:56.586 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-04-24T08:35:56.627 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:35:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:35:56.675 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:35:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:35:56.689 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:35:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:35:56.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T08:35:56.717 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T08:35:56.718 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi146 2024-04-24T08:35:56.719 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:35:56.825 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi153 2024-04-24T08:35:56.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:35:56.861 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211/remote/smithi177 2024-04-24T08:35:56.861 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:35:56.897 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T08:35:56.897 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:35:56.905 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:35:56.907 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:35:56.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T08:35:56.971 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T08:35:56.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T08:35:56.983 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T08:35:56.984 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:35:56.989 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:35:56.992 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:35:57.003 INFO:teuthology.orchestra.run.smithi153.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:35 /home/ubuntu/cephtest 2024-04-24T08:35:57.006 INFO:teuthology.orchestra.run.smithi177.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:35 /home/ubuntu/cephtest 2024-04-24T08:35:57.039 INFO:teuthology.orchestra.run.smithi146.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:35 /home/ubuntu/cephtest 2024-04-24T08:35:57.042 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T08:35:57.055 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T08:35:57.081 INFO:teuthology.nuke:Checking targets against current locks 2024-04-24T08:35:57.108 DEBUG:teuthology.nuke:shortname: smithi146 2024-04-24T08:35:57.108 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:35:57.131 DEBUG:teuthology.nuke:shortname: smithi153 2024-04-24T08:35:57.132 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:35:57.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:18.284071', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH4UL0UGN77fOzq4DniUeKxxJ73gvr/+PwPkWcELJH4dBsnvsDVEuwhq201b3ENKd0KO+2H208H1050bayk7euM='} 2024-04-24T08:35:57.152 DEBUG:teuthology.nuke:shortname: smithi177 2024-04-24T08:35:57.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:35:57.159 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:18.286677', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZzf8luyseeVcaZiGowMJccyPvqtJlHQb3q3ff/S05DEjC/HQHGz1uWFhCxDIrsszFBCNYdVxNPALAQjqXK4rg='} 2024-04-24T08:35:57.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671211', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:18.285492', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKl74kDCk+AtyDYOUxaHR1cEiDpsNpuAESOVvS/cWRalGBHLjMWI3kUIAgGfchrR4oozK3ox6O683eTo0RnqhY='} 2024-04-24T08:35:57.190 INFO:teuthology.orchestra.console.smithi146:Power off 2024-04-24T08:35:57.190 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:35:57.213 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-04-24T08:35:57.213 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:35:57.233 INFO:teuthology.orchestra.console.smithi177:Power off 2024-04-24T08:35:57.234 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:35:57.251 INFO:teuthology.orchestra.console.smithi153:Power off 2024-04-24T08:35:57.252 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:35:57.263 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-04-24T08:35:57.263 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-04-24T08:35:57.264 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:35:57.275 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-04-24T08:35:57.276 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:35:57.286 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-24T08:35:57.300 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-24T08:36:01.264 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:36:01.287 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:36:01.301 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:36:01.376 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-04-24T08:36:01.398 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-24T08:36:01.411 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-24T08:36:05.381 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:36:05.400 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:36:05.412 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:36:05.493 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-04-24T08:36:05.509 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-04-24T08:36:05.523 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-24T08:36:05.595 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-04-24T08:36:05.610 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-04-24T08:36:09.528 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:36:09.638 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-04-24T08:36:09.738 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-04-24T08:36:09.846 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 2099.8282401561737 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi146 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1b14c3b5a7cc45bbb1d079bdc51b16bf status: fail success: false 2024-04-24T08:36:09.846 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T08:36:09.937 INFO:teuthology.run:FAIL